See <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/4766/display/redirect?page=changes>
Changes: [stefan.istrate] Fix "too many pings" errors. [stefan.istrate] Increase keepalive timeout to 5 minutes. [Robert Bradshaw] Dead letter option. [Robert Bradshaw] Guard setup.py logic with __main__ condition. [stefan.istrate] Fix yapf complaints. [aydar.zaynutdinov] [BEAM-12969] [Playground] [Robert Bradshaw] Add the ability to use subprocesses with the dead letter queue. [Robert Bradshaw] Support multi-output DoFns. [noreply] Merge pull request #15510 from [BEAM-12883] Add coder for [Robert Bradshaw] multi-output fix [Robert Bradshaw] Add thresholding to dead letter pattern. [Robert Bradshaw] treshold test fixes [Robert Bradshaw] Better naming, documentation. [noreply] [BEAM-12482] Ensure that we ignore schema update options when loading [Kyle Weaver] Moving to 2.35.0-SNAPSHOT on master branch. [Kyle Weaver] Add 2.35.0 section to changelog. [noreply] Fix email links in the contact page ------------------------------------------ [...truncated 1.01 MB...] timeout method: signal timeout func_only: False collected 4979 items / 4950 deselected / 3 skipped / 26 selected apache_beam/dataframe/io_test.py . [ 3%] apache_beam/io/parquetio_test.py ............................ [100%] =============================== warnings summary =============================== target/.tox-py38-pyarrow-4/py38-pyarrow-4/lib/python3.8/site-packages/tenacity/_asyncio.py:42 <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/target/.tox-py38-pyarrow-4/py38-pyarrow-4/lib/python3.8/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead def call(self, fn, *args, **kwargs): apache_beam/dataframe/io_test.py::IOTest::test_read_write_parquet <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/apache_beam/dataframe/io.py>:569: FutureWarning: WriteToFiles is experimental. return pcoll | fileio.WriteToFiles( apache_beam/dataframe/io_test.py::IOTest::test_read_write_parquet apache_beam/dataframe/io_test.py::IOTest::test_read_write_parquet <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/apache_beam/io/fileio.py>:550: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported p.options.view_as(GoogleCloudOptions).temp_location or apache_beam/io/parquetio_test.py::TestParquet::test_int96_type_conversion <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/apache_beam/io/parquetio_test.py>:419: FutureWarning: The parameter chunksize is deprecated for pyarrow.Table.to_batches as of 0.15, please use the parameter max_chunksize instead for batch in orig.to_batches(chunksize=20) apache_beam/io/parquetio_test.py::TestParquet::test_read_with_splitting_multiple_row_group <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/apache_beam/io/parquetio_test.py>:380: FutureWarning: The parameter chunksize is deprecated for pyarrow.Table.to_batches as of 0.15, please use the parameter max_chunksize instead pa.Table.from_batches([batch]) for batch in self._records_as_arrow( apache_beam/io/parquetio_test.py::TestParquet::test_read_without_splitting_multiple_row_group <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/apache_beam/io/parquetio_test.py>:370: FutureWarning: The parameter chunksize is deprecated for pyarrow.Table.to_batches as of 0.15, please use the parameter max_chunksize instead pa.Table.from_batches([batch]) for batch in self._records_as_arrow( -- Docs: https://docs.pytest.org/en/latest/warnings.html - generated xml file: <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/pytest_py38-pyarrow-4_no_xdist.xml> - ====== 29 passed, 3 skipped, 4950 deselected, 7 warnings in 50.75 seconds ====== [1mpy38-pyarrow-4 run-test-post: commands[0] | bash <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/scripts/run_tox_cleanup.sh> [0m___________________________________ summary ____________________________________ [32m py38-pyarrow-4: commands succeeded [0m[32m congratulations :) [0m > Task :sdks:python:test-suites:tox:py38:testPy38pyarrow-5 [1mGLOB sdist-make: <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/setup.py> [0m[1mpy38-pyarrow-5 create: <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/target/.tox-py38-pyarrow-5/py38-pyarrow-5> [0m[1mpy38-pyarrow-5 installdeps: pyarrow>=5,<6 [0m[1mpy38-pyarrow-5 inst: <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/target/.tox-py38-pyarrow-5/.tmp/package/1/apache-beam-2.35.0.dev0.zip> [0m[1mpy38-pyarrow-5 installed: apache-beam @ file://<https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/target/.tox-py38-pyarrow-5/.tmp/package/1/apache-beam-2.35.0.dev0.zip,atomicwrites==1.4.0,attrs==21.2.0,avro-python3==1.9.2.1,certifi==2021.5.30,charset-normalizer==2.0.6,crcmod==1.7,deprecation==2.1.0,dill==0.3.1.1,docker==5.0.2,docopt==0.6.2,execnet==1.9.0,fastavro==1.4.5,freezegun==1.1.0,future==0.18.2,greenlet==1.1.2,grpcio==1.41.0,hdfs==2.6.0,httplib2==0.19.1,idna==3.2,mock==2.0.0,more-itertools==8.10.0,numpy==1.20.3,oauth2client==4.1.3,orjson==3.6.4,packaging==21.0,pandas==1.3.3,parameterized==0.7.5,pbr==5.6.0,pluggy==0.13.1,protobuf==3.18.1,psycopg2-binary==2.9.1,py==1.10.0,pyarrow==5.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pydot==1.4.2,PyHamcrest==1.10.1,pymongo==3.12.0,pyparsing==2.4.7,pytest==4.6.11,pytest-forked==1.3.0,pytest-timeout==1.4.2,pytest-xdist==1.34.0,python-dateutil==2.8.2,pytz==2021.3,PyYAML==5.4.1,requests==2.26.0,requests-mock==1.9.3,rsa==4.7.2,six==1.16.0,SQLAlchemy==1.4.25,tenacity==5.1.5,testcontainers==3.4.2,typing-extensions==3.10.0.2,urllib3==1.26.7,wcwidth==0.2.5,websocket-client==1.2.1,wrapt==1.13.1> [0m[1mpy38-pyarrow-5 run-test-pre: PYTHONHASHSEED='3403294354' [0m[1mpy38-pyarrow-5 run-test-pre: commands[0] | python --version [0mPython 3.8.5 [1mpy38-pyarrow-5 run-test-pre: commands[1] | pip --version [0mpip 21.2.4 from <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/target/.tox-py38-pyarrow-5/py38-pyarrow-5/lib/python3.8/site-packages/pip> (python 3.8) [1mpy38-pyarrow-5 run-test-pre: commands[2] | pip check [0mNo broken requirements found. [1mpy38-pyarrow-5 run-test-pre: commands[3] | bash <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/scripts/run_tox_cleanup.sh> [0m[1mpy38-pyarrow-5 run-test: commands[0] | /bin/sh -c 'pip freeze | grep -E '"'"'(pyarrow|numpy)'"'"'' [0mapache-beam @ file://<https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/target/.tox-py38-pyarrow-5/.tmp/package/1/apache-beam-2.35.0.dev0.zip> numpy==1.20.3 pyarrow==5.0.0 [1mpy38-pyarrow-5 run-test: commands[1] | <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/scripts/run_pytest.sh> py38-pyarrow-5 '' '-m uses_pyarrow' [0m============================= test session starts ============================== platform linux -- Python 3.8.5, pytest-4.6.11, py-1.10.0, pluggy-0.13.1 cachedir: target/.tox-py38-pyarrow-5/py38-pyarrow-5/.pytest_cache rootdir: <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python,> inifile: pytest.ini plugins: xdist-1.34.0, timeout-1.4.2, forked-1.3.0, requests-mock-1.9.3 timeout: 600.0s timeout method: signal timeout func_only: False gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I gw0 [29] / gw1 [29] / gw2 [29] / gw3 [29] / gw4 [29] / gw5 [29] ............................. [100%] =============================== warnings summary =============================== apache_beam/io/filesystems_test.py:54 apache_beam/io/filesystems_test.py:54 apache_beam/io/filesystems_test.py:54 apache_beam/io/filesystems_test.py:54 <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/apache_beam/io/filesystems_test.py>:54: DeprecationWarning: invalid escape sequence \c self.assertIsNone(FileSystems.get_scheme('c:\\abc\cdf')) # pylint: disable=anomalous-backslash-in-string apache_beam/io/filesystems_test.py:62 apache_beam/io/filesystems_test.py:62 apache_beam/io/filesystems_test.py:62 apache_beam/io/filesystems_test.py:62 <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/apache_beam/io/filesystems_test.py>:62: DeprecationWarning: invalid escape sequence \d self.assertTrue(isinstance(FileSystems.get_filesystem('c:\\abc\def'), # pylint: disable=anomalous-backslash-in-string target/.tox-py38-pyarrow-5/py38-pyarrow-5/lib/python3.8/site-packages/tenacity/_asyncio.py:42 target/.tox-py38-pyarrow-5/py38-pyarrow-5/lib/python3.8/site-packages/tenacity/_asyncio.py:42 target/.tox-py38-pyarrow-5/py38-pyarrow-5/lib/python3.8/site-packages/tenacity/_asyncio.py:42 target/.tox-py38-pyarrow-5/py38-pyarrow-5/lib/python3.8/site-packages/tenacity/_asyncio.py:42 target/.tox-py38-pyarrow-5/py38-pyarrow-5/lib/python3.8/site-packages/tenacity/_asyncio.py:42 target/.tox-py38-pyarrow-5/py38-pyarrow-5/lib/python3.8/site-packages/tenacity/_asyncio.py:42 <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/target/.tox-py38-pyarrow-5/py38-pyarrow-5/lib/python3.8/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead def call(self, fn, *args, **kwargs): apache_beam/io/parquetio_test.py:419 <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/apache_beam/io/parquetio_test.py>:419: FutureWarning: The parameter chunksize is deprecated for pyarrow.Table.to_batches as of 0.15, please use the parameter max_chunksize instead for batch in orig.to_batches(chunksize=20) apache_beam/io/parquetio_test.py:380 <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/apache_beam/io/parquetio_test.py>:380: FutureWarning: The parameter chunksize is deprecated for pyarrow.Table.to_batches as of 0.15, please use the parameter max_chunksize instead pa.Table.from_batches([batch]) for batch in self._records_as_arrow( apache_beam/io/parquetio_test.py:370 <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/apache_beam/io/parquetio_test.py>:370: FutureWarning: The parameter chunksize is deprecated for pyarrow.Table.to_batches as of 0.15, please use the parameter max_chunksize instead pa.Table.from_batches([batch]) for batch in self._records_as_arrow( apache_beam/dataframe/io.py:569 <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/apache_beam/dataframe/io.py>:569: FutureWarning: WriteToFiles is experimental. return pcoll | fileio.WriteToFiles( apache_beam/io/fileio.py:550 apache_beam/io/fileio.py:550 <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/apache_beam/io/fileio.py>:550: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported p.options.view_as(GoogleCloudOptions).temp_location or -- Docs: https://docs.pytest.org/en/latest/warnings.html - generated xml file: <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/pytest_py38-pyarrow-5.xml> - ============== 29 passed, 1 skipped, 20 warnings in 36.60 seconds ============== ERROR: Traceback (most recent call last): File "<https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/target/.tox-py38-pyarrow-5/py38-pyarrow-5/lib/python3.8/site-packages/_pytest/config/__init__.py";,> line 447, in _importconftest return self._conftestpath2mod[conftestpath] KeyError: local('<https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/conftest.py')> During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/apache_beam/io/gcp/bigquery.py";,> line 341, in <module> import google.cloud.bigquery_storage_v1 as bq_storage ModuleNotFoundError: No module named 'google.cloud' ============================= test session starts ============================== platform linux -- Python 3.8.5, pytest-4.6.11, py-1.10.0, pluggy-0.13.1 cachedir: target/.tox-py38-pyarrow-5/py38-pyarrow-5/.pytest_cache rootdir: <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python,> inifile: pytest.ini plugins: xdist-1.34.0, timeout-1.4.2, forked-1.3.0, requests-mock-1.9.3 timeout: 600.0s timeout method: signal timeout func_only: False collected 4979 items / 4950 deselected / 3 skipped / 26 selected apache_beam/dataframe/io_test.py . [ 3%] apache_beam/io/parquetio_test.py ............................ [100%] =============================== warnings summary =============================== target/.tox-py38-pyarrow-5/py38-pyarrow-5/lib/python3.8/site-packages/tenacity/_asyncio.py:42 <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/target/.tox-py38-pyarrow-5/py38-pyarrow-5/lib/python3.8/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead def call(self, fn, *args, **kwargs): apache_beam/dataframe/io_test.py::IOTest::test_read_write_parquet <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/apache_beam/dataframe/io.py>:569: FutureWarning: WriteToFiles is experimental. return pcoll | fileio.WriteToFiles( apache_beam/dataframe/io_test.py::IOTest::test_read_write_parquet apache_beam/dataframe/io_test.py::IOTest::test_read_write_parquet <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/apache_beam/io/fileio.py>:550: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported p.options.view_as(GoogleCloudOptions).temp_location or apache_beam/io/parquetio_test.py::TestParquet::test_int96_type_conversion <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/apache_beam/io/parquetio_test.py>:419: FutureWarning: The parameter chunksize is deprecated for pyarrow.Table.to_batches as of 0.15, please use the parameter max_chunksize instead for batch in orig.to_batches(chunksize=20) apache_beam/io/parquetio_test.py::TestParquet::test_read_with_splitting_multiple_row_group <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/apache_beam/io/parquetio_test.py>:380: FutureWarning: The parameter chunksize is deprecated for pyarrow.Table.to_batches as of 0.15, please use the parameter max_chunksize instead pa.Table.from_batches([batch]) for batch in self._records_as_arrow( apache_beam/io/parquetio_test.py::TestParquet::test_read_without_splitting_multiple_row_group <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/apache_beam/io/parquetio_test.py>:370: FutureWarning: The parameter chunksize is deprecated for pyarrow.Table.to_batches as of 0.15, please use the parameter max_chunksize instead pa.Table.from_batches([batch]) for batch in self._records_as_arrow( -- Docs: https://docs.pytest.org/en/latest/warnings.html - generated xml file: <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/pytest_py38-pyarrow-5_no_xdist.xml> - ====== 29 passed, 3 skipped, 4950 deselected, 7 warnings in 49.44 seconds ====== [1mpy38-pyarrow-5 run-test-post: commands[0] | bash <https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/tox/py38/build/srcs/sdks/python/scripts/run_tox_cleanup.sh> [0m___________________________________ summary ____________________________________ [32m py38-pyarrow-5: commands succeeded [0m[32m congratulations :) [0m > Task :sdks:python:test-suites:tox:py38:preCommitPy38 FAILURE: Build completed with 4 failures. 1: Task failed with an exception. ----------- * Where: Script '<https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 80 * What went wrong: Execution failed for task ':sdks:python:test-suites:dataflow:py37:preCommitIT_batch'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== 2: Task failed with an exception. ----------- * Where: Script '<https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 80 * What went wrong: Execution failed for task ':sdks:python:test-suites:dataflow:py36:preCommitIT_batch'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== 3: Task failed with an exception. ----------- * Where: Script '<https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 80 * What went wrong: Execution failed for task ':sdks:python:test-suites:dataflow:py37:preCommitIT_streaming_V2'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== 4: Task failed with an exception. ----------- * Where: Script '<https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 80 * What went wrong: Execution failed for task ':sdks:python:test-suites:dataflow:py36:preCommitIT_streaming_V2'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.9.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 47m 43s 107 actionable tasks: 79 executed, 28 from cache Publishing build scan... https://gradle.com/s/7q3qcqgnolqn2 Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
