See <https://ci-beam.apache.org/job/beam_LoadTests_Python_Combine_Dataflow_Batch/1205/display/redirect?page=changes>
Changes: [Andrew Pilloud] Publish Python nexmark metrics to influxdb [Andrew Pilloud] Exclude nexmark from codecov, it has no tests [noreply] Updated ipywidgets [yathu] Bump dataflow java fn container version to beam-master-20221022 [noreply] Ignoring BigQuery partitions with empty files (#23710) [noreply] Benchmarking RunInference Example (#23554) [Kiley Sok] Increate timeout for test pipelines [noreply] Bump Dataflow python containers to 20221021 (#23807) [noreply] Allow MoreFutures.allAsList/allAsListWithExceptions to have the passed ------------------------------------------ [...truncated 100.92 KB...] INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-25T15:49:53.269Z: JOB_MESSAGE_BASIC: Finished operation Combine with Top 0/CombinePerKey/GroupByKey/Close INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-25T15:49:53.279Z: JOB_MESSAGE_BASIC: Executing operation Combine with Top 2/CombinePerKey/GroupByKey/Read+Combine with Top 2/CombinePerKey/Combine+Combine with Top 2/CombinePerKey/Combine/Extract+Combine with Top 2/UnKey+Consume 2+Measure time: End 2 INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-25T15:49:53.313Z: JOB_MESSAGE_BASIC: Executing operation Combine with Top 3/CombinePerKey/GroupByKey/Read+Combine with Top 3/CombinePerKey/Combine+Combine with Top 3/CombinePerKey/Combine/Extract+Combine with Top 3/UnKey+Consume 3+Measure time: End 3 INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-25T15:49:53.347Z: JOB_MESSAGE_BASIC: Executing operation Combine with Top 0/CombinePerKey/GroupByKey/Read+Combine with Top 0/CombinePerKey/Combine+Combine with Top 0/CombinePerKey/Combine/Extract+Combine with Top 0/UnKey+Consume 0+Measure time: End 0 INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-25T15:49:57.634Z: JOB_MESSAGE_BASIC: Finished operation Combine with Top 1/CombinePerKey/GroupByKey/Read+Combine with Top 1/CombinePerKey/Combine+Combine with Top 1/CombinePerKey/Combine/Extract+Combine with Top 1/UnKey+Consume 1+Measure time: End 1 INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-25T15:49:59.544Z: JOB_MESSAGE_BASIC: Finished operation Combine with Top 0/CombinePerKey/GroupByKey/Read+Combine with Top 0/CombinePerKey/Combine+Combine with Top 0/CombinePerKey/Combine/Extract+Combine with Top 0/UnKey+Consume 0+Measure time: End 0 INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-25T15:49:59.719Z: JOB_MESSAGE_BASIC: Finished operation Combine with Top 2/CombinePerKey/GroupByKey/Read+Combine with Top 2/CombinePerKey/Combine+Combine with Top 2/CombinePerKey/Combine/Extract+Combine with Top 2/UnKey+Consume 2+Measure time: End 2 INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-25T15:50:00.034Z: JOB_MESSAGE_BASIC: Finished operation Combine with Top 3/CombinePerKey/GroupByKey/Read+Combine with Top 3/CombinePerKey/Combine+Combine with Top 3/CombinePerKey/Combine/Extract+Combine with Top 3/UnKey+Consume 3+Measure time: End 3 INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-25T15:50:00.100Z: JOB_MESSAGE_DEBUG: Executing success step success68 INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-25T15:50:00.170Z: JOB_MESSAGE_DETAILED: Cleaning up. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-25T15:50:00.219Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-25T15:50:00.257Z: JOB_MESSAGE_BASIC: Stopping **** pool... INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-25T15:52:11.243Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 16 to 0. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-25T15:52:11.292Z: JOB_MESSAGE_BASIC: Worker pool stopped. INFO:apache_beam.runners.dataflow.dataflow_runner:2022-10-25T15:52:11.333Z: JOB_MESSAGE_DEBUG: Tearing down pending resources... INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2022-10-25_08_43_38-9178062866469767656 is in state JOB_STATE_DONE INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 43f63d3306e443478ca10a3bf95def76 and timestamp: 1666713139.5840316: INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_combine_4_runtime Value: 80 Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0. You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins. See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 10m 13s 15 actionable tasks: 5 executed, 10 up-to-date Publishing build scan... https://gradle.com/s/ru6z4i4mukwz2 [beam_LoadTests_Python_Combine_Dataflow_Batch] $ /bin/bash -xe /tmp/jenkins5060306769058418720.sh + echo '*** Combine Python Load test: 2GB Fanout 8 ***' *** Combine Python Load test: 2GB Fanout 8 *** [Gradle] - Launching build. [src] $ <https://ci-beam.apache.org/job/beam_LoadTests_Python_Combine_Dataflow_Batch/ws/src/gradlew> -PloadTest.mainClass=apache_beam.testing.load_tests.combine_test -Prunner=DataflowRunner '-PloadTest.args=--job_name=load-tests-python-dataflow-batch-combine-5-1025150154 --project=apache-beam-testing --region=us-central1 --temp_location=gs://temp-storage-for-perf-tests/smoketests --publish_to_big_query=true --metrics_dataset=load_test --metrics_table=python_dataflow_batch_combine_5 --influx_measurement=python_batch_combine_5 --input_options='{"num_records": 2500000,"key_size": 10,"value_size": 90}' --num_****s=16 --autoscaling_algorithm=NONE --fanout=8 --top_count=20 --influx_db_name=beam_test_metrics --influx_hostname=http://10.128.0.96:8086 --runner=DataflowRunner' -PpythonVersion=3.7 --continue --max-****s=8 -Dorg.gradle.jvmargs=-Xms2g -Dorg.gradle.jvmargs=-Xmx6g -Dorg.gradle.vfs.watch=false -Pdocker-pull-licenses :sdks:python:apache_beam:testing:load_tests:run Configuration on demand is an incubating feature. > Task :buildSrc:compileJava NO-SOURCE > Task :buildSrc:compileGroovy UP-TO-DATE > Task :buildSrc:pluginDescriptors UP-TO-DATE > Task :buildSrc:processResources UP-TO-DATE > Task :buildSrc:classes UP-TO-DATE > Task :buildSrc:jar UP-TO-DATE > Task :buildSrc:assemble UP-TO-DATE > Task :buildSrc:spotlessGroovy UP-TO-DATE > Task :buildSrc:spotlessGroovyCheck UP-TO-DATE > Task :buildSrc:spotlessGroovyGradle UP-TO-DATE > Task :buildSrc:spotlessGroovyGradleCheck UP-TO-DATE > Task :buildSrc:spotlessCheck UP-TO-DATE > Task :buildSrc:pluginUnderTestMetadata UP-TO-DATE > Task :buildSrc:compileTestJava NO-SOURCE > Task :buildSrc:compileTestGroovy NO-SOURCE > Task :buildSrc:processTestResources NO-SOURCE > Task :buildSrc:testClasses UP-TO-DATE > Task :buildSrc:test NO-SOURCE > Task :buildSrc:validatePlugins UP-TO-DATE > Task :buildSrc:check UP-TO-DATE > Task :buildSrc:build UP-TO-DATE > Task :sdks:python:setupVirtualenv Collecting pip Using cached pip-22.3-py3-none-any.whl (2.1 MB) Installing collected packages: pip Attempting uninstall: pip Found existing installation: pip 20.1.1 Uninstalling pip-20.1.1: Successfully uninstalled pip-20.1.1 Successfully installed pip-22.3 Ignoring grpcio: markers 'sys_platform == "darwin"' don't match your environment Ignoring protobuf: markers 'python_version == "3.10" and sys_platform == "darwin"' don't match your environment Collecting tox==3.20.1 Using cached tox-3.20.1-py2.py3-none-any.whl (83 kB) Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_Combine_Dataflow_Batch/ws/src/build/gradleenv/1922375555/lib/python3.7/site-packages> (from -r <https://ci-beam.apache.org/job/beam_LoadTests_Python_Combine_Dataflow_Batch/ws/src/sdks/python/build-requirements.txt> (line 20)) (47.1.0) Collecting setuptools Using cached setuptools-65.5.0-py3-none-any.whl (1.2 MB) Collecting wheel>=0.36.0 Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB) Collecting grpcio-tools==1.37.0 Using cached grpcio_tools-1.37.0-cp37-cp37m-manylinux2014_x86_64.whl (2.5 MB) Collecting mypy-protobuf==1.18 Using cached mypy_protobuf-1.18-py3-none-any.whl (7.3 kB) Collecting distlib==0.3.1 Using cached distlib-0.3.1-py2.py3-none-any.whl (335 kB) Collecting numpy<1.23.0,>=1.14.3 Using cached numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB) Collecting virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0 Using cached virtualenv-20.16.5-py3-none-any.whl (8.8 MB) Collecting pluggy>=0.12.0 Using cached pluggy-1.0.0-py2.py3-none-any.whl (13 kB) Collecting importlib-metadata<3,>=0.12 Using cached importlib_metadata-2.1.3-py2.py3-none-any.whl (10 kB) Collecting filelock>=3.0.0 Using cached filelock-3.8.0-py3-none-any.whl (10 kB) Collecting py>=1.4.17 Using cached py-1.11.0-py2.py3-none-any.whl (98 kB) Collecting packaging>=14 Using cached packaging-21.3-py3-none-any.whl (40 kB) Collecting six>=1.14.0 Using cached six-1.16.0-py2.py3-none-any.whl (11 kB) Collecting toml>=0.9.4 Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB) Collecting grpcio>=1.37.0 Using cached grpcio-1.50.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.7 MB) Collecting protobuf<4.0dev,>=3.5.0.post1 Using cached protobuf-3.20.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB) Collecting zipp>=0.5 Using cached zipp-3.10.0-py3-none-any.whl (6.2 kB) Collecting pyparsing!=3.0.5,>=2.0.2 Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB) Collecting platformdirs<3,>=2.4 Using cached platformdirs-2.5.2-py3-none-any.whl (14 kB) Collecting virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0 Using cached virtualenv-20.16.4-py3-none-any.whl (8.8 MB) Using cached virtualenv-20.16.3-py2.py3-none-any.whl (8.8 MB) Using cached virtualenv-20.16.2-py2.py3-none-any.whl (8.8 MB) Installing collected packages: distlib, zipp, wheel, toml, six, setuptools, pyparsing, py, protobuf, platformdirs, numpy, filelock, packaging, mypy-protobuf, importlib-metadata, grpcio, virtualenv, pluggy, grpcio-tools, tox Attempting uninstall: setuptools Found existing installation: setuptools 47.1.0 Uninstalling setuptools-47.1.0: Successfully uninstalled setuptools-47.1.0 Successfully installed distlib-0.3.1 filelock-3.8.0 grpcio-1.50.0 grpcio-tools-1.37.0 importlib-metadata-2.1.3 mypy-protobuf-1.18 numpy-1.21.6 packaging-21.3 platformdirs-2.5.2 pluggy-1.0.0 protobuf-3.20.3 py-1.11.0 pyparsing-3.0.9 setuptools-65.5.0 six-1.16.0 toml-0.10.2 tox-3.20.1 virtualenv-20.16.2 wheel-0.37.1 zipp-3.10.0 > Task :sdks:python:apache_beam:testing:load_tests:setupVirtualenv Collecting pip Using cached pip-22.3-py3-none-any.whl (2.1 MB) Installing collected packages: pip Attempting uninstall: pip Found existing installation: pip 20.1.1 Uninstalling pip-20.1.1: Successfully uninstalled pip-20.1.1 Successfully installed pip-22.3 Ignoring grpcio: markers 'sys_platform == "darwin"' don't match your environment Ignoring protobuf: markers 'python_version == "3.10" and sys_platform == "darwin"' don't match your environment Collecting tox==3.20.1 Using cached tox-3.20.1-py2.py3-none-any.whl (83 kB) Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_Combine_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from -r <https://ci-beam.apache.org/job/beam_LoadTests_Python_Combine_Dataflow_Batch/ws/src/sdks/python/build-requirements.txt> (line 20)) (47.1.0) Collecting setuptools Using cached setuptools-65.5.0-py3-none-any.whl (1.2 MB) Collecting wheel>=0.36.0 Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB) Collecting grpcio-tools==1.37.0 Using cached grpcio_tools-1.37.0-cp37-cp37m-manylinux2014_x86_64.whl (2.5 MB) Collecting mypy-protobuf==1.18 Using cached mypy_protobuf-1.18-py3-none-any.whl (7.3 kB) Collecting distlib==0.3.1 Using cached distlib-0.3.1-py2.py3-none-any.whl (335 kB) Collecting numpy<1.23.0,>=1.14.3 Using cached numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB) Collecting virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0 Using cached virtualenv-20.16.5-py3-none-any.whl (8.8 MB) Collecting pluggy>=0.12.0 Using cached pluggy-1.0.0-py2.py3-none-any.whl (13 kB) Collecting importlib-metadata<3,>=0.12 Using cached importlib_metadata-2.1.3-py2.py3-none-any.whl (10 kB) Collecting six>=1.14.0 Using cached six-1.16.0-py2.py3-none-any.whl (11 kB) Collecting packaging>=14 Using cached packaging-21.3-py3-none-any.whl (40 kB) Collecting filelock>=3.0.0 Using cached filelock-3.8.0-py3-none-any.whl (10 kB) Collecting toml>=0.9.4 Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB) Collecting py>=1.4.17 Using cached py-1.11.0-py2.py3-none-any.whl (98 kB) Collecting grpcio>=1.37.0 Using cached grpcio-1.50.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.7 MB) Collecting protobuf<4.0dev,>=3.5.0.post1 Using cached protobuf-3.20.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB) Collecting zipp>=0.5 Using cached zipp-3.10.0-py3-none-any.whl (6.2 kB) Collecting pyparsing!=3.0.5,>=2.0.2 Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB) Collecting virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0 Using cached virtualenv-20.16.4-py3-none-any.whl (8.8 MB) Using cached virtualenv-20.16.3-py2.py3-none-any.whl (8.8 MB) Using cached virtualenv-20.16.2-py2.py3-none-any.whl (8.8 MB) Collecting platformdirs<3,>=2 Using cached platformdirs-2.5.2-py3-none-any.whl (14 kB) Installing collected packages: distlib, zipp, wheel, toml, six, setuptools, pyparsing, py, protobuf, platformdirs, numpy, filelock, packaging, mypy-protobuf, importlib-metadata, grpcio, virtualenv, pluggy, grpcio-tools, tox Attempting uninstall: setuptools Found existing installation: setuptools 47.1.0 Uninstalling setuptools-47.1.0: Successfully uninstalled setuptools-47.1.0 > Task :sdks:python:sdist UP-TO-DATE > Task :sdks:python:apache_beam:testing:load_tests:setupVirtualenv Successfully installed distlib-0.3.1 filelock-3.8.0 grpcio-1.50.0 grpcio-tools-1.37.0 importlib-metadata-2.1.3 mypy-protobuf-1.18 numpy-1.21.6 packaging-21.3 platformdirs-2.5.2 pluggy-1.0.0 protobuf-3.20.3 py-1.11.0 pyparsing-3.0.9 setuptools-65.5.0 six-1.16.0 toml-0.10.2 tox-3.20.1 virtualenv-20.16.2 wheel-0.37.1 zipp-3.10.0 > Task :sdks:python:apache_beam:testing:load_tests:installGcpTest Processing <https://ci-beam.apache.org/job/beam_LoadTests_Python_Combine_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz> Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Requirement already satisfied: protobuf<4,>3.12.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_Combine_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from apache-beam==2.44.0.dev0) (3.20.3) Collecting crcmod<2.0,>=1.7 Using cached crcmod-1.7-cp37-cp37m-linux_x86_64.whl Collecting orjson<4.0 Downloading orjson-3.8.1.tar.gz (860 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 860.3/860.3 kB 16.1 MB/s eta 0:00:00 Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'error' error: subprocess-exited-with-error × Preparing metadata (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [6 lines of output] Cargo, the Rust package manager, is not installed or is not on PATH. This package requires Rust and Cargo to compile extensions. Install it through the system's package manager or via https://rustup.rs/ Checking for Rust toolchain.... [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed × Encountered error while generating package metadata. ╰─> See above for output. note: This is an issue with the package mentioned above, not pip. hint: See above for details. > Task :sdks:python:apache_beam:testing:load_tests:installGcpTest FAILED FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':sdks:python:apache_beam:testing:load_tests:installGcpTest'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. * Get more help at https://help.gradle.org BUILD FAILED in 28s 14 actionable tasks: 3 executed, 11 up-to-date Publishing build scan... https://gradle.com/s/sn6ymapw4pido Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
