See <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/1226/display/redirect?page=changes>
Changes: [noreply] Remove unnecessary dependencies from jpms test (#23775) [noreply] Use Spark 3 job-server as default Spark job-server for PortableRunner [noreply] Support usage of custom profileName with AWS ProfileCredentialsProvider [noreply] Migrate examples and maven-archetypes (including Java Quickstart) to [Moritz Mack] Update remaining pointers to Spark runner to Spark 3 module (addresses ------------------------------------------ [...truncated 47.40 KB...] value: <JsonValue integer_value: 200>>, <Property key: 'mean' value: <JsonValue integer_value: -9221705327577078>>]>> INFO:apache_beam.runners.dataflow.dataflow_metrics:Distribution metric sum value seems to have overflowed integer_value range, the correctness of sum or mean value may not be guaranteed: <JsonValue object_value: <JsonObject properties: [<Property key: 'sum' value: <JsonValue double_value: -1.8443410655154158e+18>>, <Property key: 'mean' value: <JsonValue integer_value: -9221705327577078>>, <Property key: 'min' value: <JsonValue integer_value: -9221705327581058>>, <Property key: 'count' value: <JsonValue integer_value: 200>>, <Property key: 'max' value: <JsonValue integer_value: -9221705327572092>>]>> INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test results for test: 0f349e47f78d4e318f71a8ecf91886bc and timestamp: 1666623042.5266411: INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_runtime Value: 265 INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_0_count_latency Value: 200 INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_0_max_latency Value: -9221705327572092 INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_0_min_latency Value: -9221705327581058 INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_0_sum_latency Value: -1844341065515415808 INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric: python_dataflow_batch_gbk_6_measure_latency_0_mean_latency Value: -9221705327577080 Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0. You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins. See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 14m 13s 15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date Publishing build scan... https://gradle.com/s/d25qxlu45645g Build step 'Invoke Gradle script' changed build result to SUCCESS [beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch] $ /bin/bash -xe /tmp/jenkins8179653558047151000.sh + echo '*** GroupByKey Python Load test: reiterate 4 times 2MB values ***' *** GroupByKey Python Load test: reiterate 4 times 2MB values *** [Gradle] - Launching build. [src] $ <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/gradlew> -PloadTest.mainClass=apache_beam.testing.load_tests.group_by_key_test -Prunner=DataflowRunner '-PloadTest.args=--project=apache-beam-testing --region=us-central1 --job_name=load-tests-python-dataflow-batch-gbk-7-1024125445 --temp_location=gs://temp-storage-for-perf-tests/loadtests --publish_to_big_query=true --metrics_dataset=load_test --metrics_table=python_dataflow_batch_gbk_7 --influx_measurement=python_batch_gbk_7 --input_options='{"num_records": 20000000,"key_size": 10,"value_size": 90,"num_hot_keys": 10,"hot_key_fraction": 1}' --iterations=4 --fanout=1 --num_****s=5 --autoscaling_algorithm=NONE --influx_db_name=beam_test_metrics --influx_hostname=http://10.128.0.96:8086 --runner=DataflowRunner' -PpythonVersion=3.7 --continue --max-****s=8 -Dorg.gradle.jvmargs=-Xms2g -Dorg.gradle.jvmargs=-Xmx6g -Dorg.gradle.vfs.watch=false -Pdocker-pull-licenses :sdks:python:apache_beam:testing:load_tests:run Configuration on demand is an incubating feature. > Task :buildSrc:compileJava NO-SOURCE > Task :buildSrc:compileGroovy UP-TO-DATE > Task :buildSrc:pluginDescriptors UP-TO-DATE > Task :buildSrc:processResources UP-TO-DATE > Task :buildSrc:classes UP-TO-DATE > Task :buildSrc:jar UP-TO-DATE > Task :buildSrc:assemble UP-TO-DATE > Task :buildSrc:spotlessGroovy UP-TO-DATE > Task :buildSrc:spotlessGroovyCheck UP-TO-DATE > Task :buildSrc:spotlessGroovyGradle UP-TO-DATE > Task :buildSrc:spotlessGroovyGradleCheck UP-TO-DATE > Task :buildSrc:spotlessCheck UP-TO-DATE > Task :buildSrc:pluginUnderTestMetadata UP-TO-DATE > Task :buildSrc:compileTestJava NO-SOURCE > Task :buildSrc:compileTestGroovy NO-SOURCE > Task :buildSrc:processTestResources NO-SOURCE > Task :buildSrc:testClasses UP-TO-DATE > Task :buildSrc:test NO-SOURCE > Task :buildSrc:validatePlugins UP-TO-DATE > Task :buildSrc:check UP-TO-DATE > Task :buildSrc:build UP-TO-DATE > Task :sdks:python:setupVirtualenv Collecting pip Using cached pip-22.3-py3-none-any.whl (2.1 MB) Installing collected packages: pip Attempting uninstall: pip Found existing installation: pip 20.1.1 Uninstalling pip-20.1.1: Successfully uninstalled pip-20.1.1 Successfully installed pip-22.3 Ignoring grpcio: markers 'sys_platform == "darwin"' don't match your environment Ignoring protobuf: markers 'python_version == "3.10" and sys_platform == "darwin"' don't match your environment Collecting tox==3.20.1 Using cached tox-3.20.1-py2.py3-none-any.whl (83 kB) Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1922375555/lib/python3.7/site-packages> (from -r <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build-requirements.txt> (line 20)) (47.1.0) Collecting setuptools Using cached setuptools-60.9.3-py3-none-any.whl (1.1 MB) Collecting wheel>=0.36.0 Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB) Collecting grpcio-tools==1.37.0 Using cached grpcio_tools-1.37.0-cp37-cp37m-manylinux2014_x86_64.whl (2.5 MB) Collecting mypy-protobuf==1.18 Using cached mypy_protobuf-1.18-py3-none-any.whl (7.3 kB) Collecting distlib==0.3.1 Using cached distlib-0.3.1-py2.py3-none-any.whl (335 kB) Collecting numpy<1.23.0,>=1.14.3 Downloading numpy-1.21.5-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.7/15.7 MB 71.5 MB/s eta 0:00:00 Collecting virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0 Downloading virtualenv-20.13.2-py2.py3-none-any.whl (8.7 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.7/8.7 MB 89.6 MB/s eta 0:00:00 Collecting six>=1.14.0 Using cached six-1.16.0-py2.py3-none-any.whl (11 kB) Collecting pluggy>=0.12.0 Using cached pluggy-1.0.0-py2.py3-none-any.whl (13 kB) Collecting importlib-metadata<3,>=0.12 Downloading importlib_metadata-2.1.2-py2.py3-none-any.whl (10 kB) Collecting py>=1.4.17 Using cached py-1.11.0-py2.py3-none-any.whl (98 kB) Collecting packaging>=14 Using cached packaging-21.3-py3-none-any.whl (40 kB) Collecting toml>=0.9.4 Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB) Collecting filelock>=3.0.0 Downloading filelock-3.4.2-py3-none-any.whl (9.9 kB) Collecting grpcio>=1.37.0 Using cached grpcio-1.50.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.7 MB) Collecting protobuf<4.0dev,>=3.5.0.post1 Using cached protobuf-3.20.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB) Collecting zipp>=0.5 Using cached zipp-3.10.0-py3-none-any.whl (6.2 kB) Collecting pyparsing!=3.0.5,>=2.0.2 Downloading pyparsing-3.0.6-py3-none-any.whl (97 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.6/97.6 kB 2.2 MB/s eta 0:00:00 Collecting platformdirs<3,>=2 Downloading platformdirs-2.4.1-py3-none-any.whl (14 kB) Installing collected packages: distlib, zipp, wheel, toml, six, setuptools, pyparsing, py, protobuf, platformdirs, numpy, filelock, packaging, mypy-protobuf, importlib-metadata, grpcio, virtualenv, pluggy, grpcio-tools, tox Attempting uninstall: setuptools Found existing installation: setuptools 47.1.0 Uninstalling setuptools-47.1.0: Successfully uninstalled setuptools-47.1.0 Successfully installed distlib-0.3.1 filelock-3.4.2 grpcio-1.50.0 grpcio-tools-1.37.0 importlib-metadata-2.1.2 mypy-protobuf-1.18 numpy-1.21.5 packaging-21.3 platformdirs-2.4.1 pluggy-1.0.0 protobuf-3.20.3 py-1.11.0 pyparsing-3.0.6 setuptools-60.9.3 six-1.16.0 toml-0.10.2 tox-3.20.1 virtualenv-20.13.2 wheel-0.37.1 zipp-3.10.0 > Task :sdks:python:apache_beam:testing:load_tests:setupVirtualenv Collecting pip Downloading pip-21.3.1-py3-none-any.whl (1.7 MB) Installing collected packages: pip Attempting uninstall: pip Found existing installation: pip 20.1.1 Uninstalling pip-20.1.1: Successfully uninstalled pip-20.1.1 Successfully installed pip-21.3.1 Ignoring grpcio: markers 'sys_platform == "darwin"' don't match your environment Ignoring protobuf: markers 'python_version == "3.10" and sys_platform == "darwin"' don't match your environment Collecting tox==3.20.1 Using cached tox-3.20.1-py2.py3-none-any.whl (83 kB) Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from -r <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build-requirements.txt> (line 20)) (47.1.0) Collecting setuptools Using cached setuptools-60.9.3-py3-none-any.whl (1.1 MB) Collecting wheel>=0.36.0 Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB) Collecting grpcio-tools==1.37.0 Using cached grpcio_tools-1.37.0-cp37-cp37m-manylinux2014_x86_64.whl (2.5 MB) Collecting mypy-protobuf==1.18 Using cached mypy_protobuf-1.18-py3-none-any.whl (7.3 kB) Collecting distlib==0.3.1 Using cached distlib-0.3.1-py2.py3-none-any.whl (335 kB) Collecting numpy<1.23.0,>=1.14.3 Using cached numpy-1.21.5-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB) Collecting pluggy>=0.12.0 Using cached pluggy-1.0.0-py2.py3-none-any.whl (13 kB) Collecting six>=1.14.0 Using cached six-1.16.0-py2.py3-none-any.whl (11 kB) Collecting virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0 Using cached virtualenv-20.16.5-py3-none-any.whl (8.8 MB) Collecting packaging>=14 Using cached packaging-21.3-py3-none-any.whl (40 kB) Collecting filelock>=3.0.0 Using cached filelock-3.8.0-py3-none-any.whl (10 kB) Collecting importlib-metadata<3,>=0.12 Using cached importlib_metadata-2.1.2-py2.py3-none-any.whl (10 kB) Collecting toml>=0.9.4 Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB) Collecting py>=1.4.17 Using cached py-1.11.0-py2.py3-none-any.whl (98 kB) Collecting protobuf<4.0dev,>=3.5.0.post1 Using cached protobuf-3.20.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB) Collecting grpcio>=1.37.0 Using cached grpcio-1.50.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.7 MB) Collecting zipp>=0.5 Using cached zipp-3.10.0-py3-none-any.whl (6.2 kB) Collecting pyparsing!=3.0.5,>=2.0.2 Using cached pyparsing-3.0.6-py3-none-any.whl (97 kB) Collecting virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0 Using cached virtualenv-20.16.4-py3-none-any.whl (8.8 MB) Using cached virtualenv-20.16.3-py2.py3-none-any.whl (8.8 MB) Using cached virtualenv-20.16.2-py2.py3-none-any.whl (8.8 MB) Collecting platformdirs<3,>=2 Using cached platformdirs-2.4.1-py3-none-any.whl (14 kB) Installing collected packages: zipp, six, pyparsing, platformdirs, importlib-metadata, filelock, distlib, virtualenv, toml, setuptools, py, protobuf, pluggy, packaging, grpcio, wheel, tox, numpy, mypy-protobuf, grpcio-tools Attempting uninstall: setuptools Found existing installation: setuptools 47.1.0 Uninstalling setuptools-47.1.0: Successfully uninstalled setuptools-47.1.0 > Task :sdks:python:sdist <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1922375555/lib/python3.7/site-packages/setuptools/dist.py>:505: UserWarning: Normalizing '2.44.0.dev' to '2.44.0.dev0' warnings.warn(tmpl.format(**locals())) warning: no files found matching 'README.md' warning: no files found matching 'NOTICE' warning: no files found matching 'LICENSE' warning: no files found matching 'LICENSE.python' warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md > Task :sdks:python:apache_beam:testing:load_tests:setupVirtualenv Successfully installed distlib-0.3.1 filelock-3.8.0 grpcio-1.50.0 grpcio-tools-1.37.0 importlib-metadata-2.1.2 mypy-protobuf-1.18 numpy-1.21.5 packaging-21.3 platformdirs-2.4.1 pluggy-1.0.0 protobuf-3.20.3 py-1.11.0 pyparsing-3.0.6 setuptools-60.9.3 six-1.16.0 toml-0.10.2 tox-3.20.1 virtualenv-20.16.2 wheel-0.37.1 zipp-3.10.0 WARNING: You are using pip version 21.3.1; however, version 22.3 is available. You should consider upgrading via the '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/bin/python3.7> -m pip install --upgrade pip' command. > Task :sdks:python:apache_beam:testing:load_tests:installGcpTest Processing <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/sdks/python/build/apache-beam.tar.gz> Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Requirement already satisfied: protobuf<4,>3.12.2 in <https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from apache-beam==2.44.0.dev0) (3.20.3) Collecting crcmod<2.0,>=1.7 Using cached crcmod-1.7-cp37-cp37m-linux_x86_64.whl Collecting orjson<4.0 Downloading orjson-3.6.5-cp37-cp37m-manylinux_2_24_x86_64.whl (247 kB) Collecting dill<0.3.2,>=0.3.1.1 Using cached dill-0.3.1.1-py3-none-any.whl ERROR: Could not find a version that satisfies the requirement cloudpickle~=2.2.0 (from apache-beam[aws,azure,dataframe,gcp,test]) (from versions: 0.1.0, 0.1.1, 0.2.1, 0.2.2, 0.3.0, 0.3.1, 0.4.0, 0.4.1, 0.4.2, 0.4.3, 0.4.4, 0.5.0, 0.5.1, 0.5.2, 0.5.3, 0.5.4, 0.5.5, 0.5.6, 0.6.0, 0.6.1, 0.7.0, 0.8.0, 0.8.1, 1.0.0, 1.1.1, 1.2.0, 1.2.1, 1.2.2, 1.3.0, 1.4.0, 1.4.1, 1.5.0, 1.6.0, 2.0.0) ERROR: No matching distribution found for cloudpickle~=2.2.0 WARNING: You are using pip version 21.3.1; however, version 22.3 is available. You should consider upgrading via the '<https://ci-beam.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/ws/src/build/gradleenv/1329484227/bin/python3.7> -m pip install --upgrade pip' command. > Task :sdks:python:apache_beam:testing:load_tests:installGcpTest FAILED FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':sdks:python:apache_beam:testing:load_tests:installGcpTest'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. * Get more help at https://help.gradle.org BUILD FAILED in 27s 14 actionable tasks: 4 executed, 10 up-to-date Publishing build scan... https://gradle.com/s/zbtnoanpyw6vk Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
