See <https://ci-beam.apache.org/job/beam_LoadTests_Python_ParDo_Flink_Batch/1056/display/redirect?page=changes>
Changes: [noreply] Pass user specified destination type to UpdateSchemaDestination [noreply] [Go SDK] Stream decode values in single iterations (#22904) [noreply] Enable autosharding for BQ: #22818 [noreply] Update wordcount_minimal.py by removing pipeline_args.extend (#22786) [noreply] Add map state in the Go Sdk (#22897) [noreply] [BEAM-12164] Feat: Added support to Cloud Spanner Change Streams ------------------------------------------ [...truncated 12.38 KB...] Digest: sha256:911283585036550af2b3ab86dd87aab88a229103f63aabf080ebec9be3589a3b Status: Downloaded newer image for gcr.io/apache-beam-testing/beam_portability/beam_flink1.12_job_server:latest 8b56aa66621fe53e48f0657f67f1baf747b3f06eefd67c2cdab8e92d5d432913 + start_tunnel ++ gcloud compute ssh --quiet --zone=us-central1-a yarn@beam-loadtests-python-pardo-flink-batch-1056-m '--command=curl -s "http://beam-loadtests-python-pardo-flink-batch-1056-w-5.c.apache-beam-testing.internal:38127/jobmanager/config";' Existing host keys found in /home/jenkins/.ssh/google_compute_known_hosts + local 'job_server_config=[{"key":"yarn.flink-dist-jar","value":"file:/usr/lib/flink/lib/flink-dist_2.12-1.12.5.jar"},{"key":"jobmanager.execution.failover-strategy","value":"region"},{"key":"high-availability.cluster-id","value":"application_1661608171142_0002"},{"key":"jobmanager.rpc.address","value":"beam-loadtests-python-pardo-flink-batch-1056-w-5.c.apache-beam-testing.internal"},{"key":"jobmanager.memory.jvm-overhead.min","value":"1073741824b"},{"key":"io.tmp.dirs","value":"/hadoop/yarn/nm-local-dir/usercache/root/appcache/application_1661608171142_0002"},{"key":"taskmanager.network.numberOfBuffers","value":"2048"},{"key":"parallelism.default","value":"20"},{"key":"taskmanager.numberOfTaskSlots","value":"4"},{"key":"env.hadoop.conf.dir","value":"/etc/hadoop/conf"},{"key":"yarn.application.name","value":"flink-dataproc"},{"key":"taskmanager.heap.mb","value":"11992"},{"key":"taskmanager.memory.process.size","value":"11992 mb"},{"key":"web.port","value":"0"},{"key":"jobmanager.heap.mb","value":"11992"},{"key":"jobmanager.memory.off-heap.size","value":"134217728b"},{"key":"execution.target","value":"yarn-session"},{"key":"jobmanager.memory.process.size","value":"11992 mb"},{"key":"web.tmpdir","value":"/tmp/flink-web-9328e25f-4cf2-4a07-a6db-fffbb4884c98"},{"key":"jobmanager.rpc.port","value":"46769"},{"key":"internal.io.tmpdirs.use-local-default","value":"true"},{"key":"execution.attached","value":"false"},{"key":"internal.cluster.execution-mode","value":"NORMAL"},{"key":"fs.hdfs.hadoopconf","value":"/etc/hadoop/conf"},{"key":"rest.address","value":"beam-loadtests-python-pardo-flink-batch-1056-w-5.c.apache-beam-testing.internal"},{"key":"jobmanager.memory.jvm-metaspace.size","value":"268435456b"},{"key":"$internal.deployment.config-dir","value":"/usr/lib/flink/conf"},{"key":"$internal.yarn.log-config-file","value":"/usr/lib/flink/conf/log4j.properties"},{"key":"jobmanager.memory.heap.size","value":"11098128384b"},{"key":"jobmanager.memory.jvm-overhead.max","value":"1073741824b"}]' + local key=jobmanager.rpc.port ++ echo beam-loadtests-python-pardo-flink-batch-1056-w-5.c.apache-beam-testing.internal:38127 ++ cut -d : -f1 + local yarn_application_master_host=beam-loadtests-python-pardo-flink-batch-1056-w-5.c.apache-beam-testing.internal ++ echo '[{"key":"yarn.flink-dist-jar","value":"file:/usr/lib/flink/lib/flink-dist_2.12-1.12.5.jar"},{"key":"jobmanager.execution.failover-strategy","value":"region"},{"key":"high-availability.cluster-id","value":"application_1661608171142_0002"},{"key":"jobmanager.rpc.address","value":"beam-loadtests-python-pardo-flink-batch-1056-w-5.c.apache-beam-testing.internal"},{"key":"jobmanager.memory.jvm-overhead.min","value":"1073741824b"},{"key":"io.tmp.dirs","value":"/hadoop/yarn/nm-local-dir/usercache/root/appcache/application_1661608171142_0002"},{"key":"taskmanager.network.numberOfBuffers","value":"2048"},{"key":"parallelism.default","value":"20"},{"key":"taskmanager.numberOfTaskSlots","value":"4"},{"key":"env.hadoop.conf.dir","value":"/etc/hadoop/conf"},{"key":"yarn.application.name","value":"flink-dataproc"},{"key":"taskmanager.heap.mb","value":"11992"},{"key":"taskmanager.memory.process.size","value":"11992' 'mb"},{"key":"web.port","value":"0"},{"key":"jobmanager.heap.mb","value":"11992"},{"key":"jobmanager.memory.off-heap.size","value":"134217728b"},{"key":"execution.target","value":"yarn-session"},{"key":"jobmanager.memory.process.size","value":"11992' 'mb"},{"key":"web.tmpdir","value":"/tmp/flink-web-9328e25f-4cf2-4a07-a6db-fffbb4884c98"},{"key":"jobmanager.rpc.port","value":"46769"},{"key":"internal.io.tmpdirs.use-local-default","value":"true"},{"key":"execution.attached","value":"false"},{"key":"internal.cluster.execution-mode","value":"NORMAL"},{"key":"fs.hdfs.hadoopconf","value":"/etc/hadoop/conf"},{"key":"rest.address","value":"beam-loadtests-python-pardo-flink-batch-1056-w-5.c.apache-beam-testing.internal"},{"key":"jobmanager.memory.jvm-metaspace.size","value":"268435456b"},{"key":"$internal.deployment.config-dir","value":"/usr/lib/flink/conf"},{"key":"$internal.yarn.log-config-file","value":"/usr/lib/flink/conf/log4j.properties"},{"key":"jobmanager.memory.heap.size","value":"11098128384b"},{"key":"jobmanager.memory.jvm-overhead.max","value":"1073741824b"}]' ++ python -c 'import sys, json; print([e['\''value'\''] for e in json.load(sys.stdin) if e['\''key'\''] == u'\''jobmanager.rpc.port'\''][0])' + local jobmanager_rpc_port=46769 ++ [[ true == \t\r\u\e ]] ++ echo ' -Nf >& /dev/null' + local 'detached_mode_params= -Nf >& /dev/null' ++ [[ -n gcr.io/apache-beam-testing/beam_portability/beam_flink1.12_job_server:latest ]] ++ echo '-L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097' + local 'job_server_ports_forwarding=-L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097' + local 'tunnel_command=gcloud compute ssh --zone=us-central1-a --quiet yarn@beam-loadtests-python-pardo-flink-batch-1056-m -- -L 8081:beam-loadtests-python-pardo-flink-batch-1056-w-5.c.apache-beam-testing.internal:38127 -L 46769:beam-loadtests-python-pardo-flink-batch-1056-w-5.c.apache-beam-testing.internal:46769 -L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097 -D 1080 -Nf >& /dev/null' + eval gcloud compute ssh --zone=us-central1-a --quiet yarn@beam-loadtests-python-pardo-flink-batch-1056-m -- -L 8081:beam-loadtests-python-pardo-flink-batch-1056-w-5.c.apache-beam-testing.internal:38127 -L 46769:beam-loadtests-python-pardo-flink-batch-1056-w-5.c.apache-beam-testing.internal:46769 -L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097 -D 1080 -Nf '>&' /dev/null ++ gcloud compute ssh --zone=us-central1-a --quiet yarn@beam-loadtests-python-pardo-flink-batch-1056-m -- -L 8081:beam-loadtests-python-pardo-flink-batch-1056-w-5.c.apache-beam-testing.internal:38127 -L 46769:beam-loadtests-python-pardo-flink-batch-1056-w-5.c.apache-beam-testing.internal:46769 -L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097 -D 1080 -Nf [beam_LoadTests_Python_ParDo_Flink_Batch] $ /bin/bash -xe /tmp/jenkins4636764496347516038.sh + echo '*** ParDo Python Load test: 20M 100 byte records 10 iterations ***' *** ParDo Python Load test: 20M 100 byte records 10 iterations *** [Gradle] - Launching build. [src] $ <https://ci-beam.apache.org/job/beam_LoadTests_Python_ParDo_Flink_Batch/ws/src/gradlew> -PloadTest.mainClass=apache_beam.testing.load_tests.pardo_test -Prunner=PortableRunner '-PloadTest.args=--job_name=load-tests-python-flink-batch-pardo-1-0827125446 --project=apache-beam-testing --publish_to_big_query=true --metrics_dataset=load_test --metrics_table=python_flink_batch_pardo_1 --influx_measurement=python_batch_pardo_1 --input_options='{"num_records": 20000000,"key_size": 10,"value_size": 90}' --iterations=10 --number_of_counter_operations=0 --number_of_counters=0 --parallelism=5 --job_endpoint=localhost:8099 --environment_type=DOCKER --environment_config=gcr.io/apache-beam-testing/beam_portability/beam_python3.7_sdk:latest --influx_db_name=beam_test_metrics --influx_hostname=http://10.128.0.96:8086 --runner=PortableRunner' -PpythonVersion=3.7 --continue --max-****s=8 -Dorg.gradle.jvmargs=-Xms2g -Dorg.gradle.jvmargs=-Xmx6g -Dorg.gradle.vfs.watch=false -Pdocker-pull-licenses :sdks:python:apache_beam:testing:load_tests:run Starting a Gradle Daemon (subsequent builds will be faster) Configuration on demand is an incubating feature. > Task :buildSrc:compileJava NO-SOURCE > Task :buildSrc:compileGroovy FROM-CACHE > Task :buildSrc:pluginDescriptors > Task :buildSrc:processResources > Task :buildSrc:classes > Task :buildSrc:jar > Task :buildSrc:assemble > Task :buildSrc:spotlessGroovy FROM-CACHE > Task :buildSrc:spotlessGroovyCheck UP-TO-DATE > Task :buildSrc:spotlessGroovyGradle FROM-CACHE > Task :buildSrc:spotlessGroovyGradleCheck UP-TO-DATE > Task :buildSrc:spotlessCheck UP-TO-DATE > Task :buildSrc:pluginUnderTestMetadata > Task :buildSrc:compileTestJava NO-SOURCE > Task :buildSrc:compileTestGroovy NO-SOURCE > Task :buildSrc:processTestResources NO-SOURCE > Task :buildSrc:testClasses UP-TO-DATE > Task :buildSrc:test NO-SOURCE > Task :buildSrc:validatePlugins FROM-CACHE > Task :buildSrc:check UP-TO-DATE > Task :buildSrc:build > Task :sdks:python:setupVirtualenv Collecting pip Using cached pip-22.2.2-py3-none-any.whl (2.0 MB) Installing collected packages: pip Attempting uninstall: pip Found existing installation: pip 20.1.1 Uninstalling pip-20.1.1: Successfully uninstalled pip-20.1.1 Successfully installed pip-22.2.2 Collecting tox==3.20.1 Using cached tox-3.20.1-py2.py3-none-any.whl (83 kB) Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_ParDo_Flink_Batch/ws/src/build/gradleenv/1922375555/lib/python3.7/site-packages> (from -r <https://ci-beam.apache.org/job/beam_LoadTests_Python_ParDo_Flink_Batch/ws/src/sdks/python/build-requirements.txt> (line 20)) (47.1.0) Collecting setuptools Using cached setuptools-65.3.0-py3-none-any.whl (1.2 MB) Collecting wheel>=0.36.0 Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB) Collecting grpcio-tools==1.37.0 Using cached grpcio_tools-1.37.0-cp37-cp37m-manylinux2014_x86_64.whl (2.5 MB) Collecting mypy-protobuf==1.18 Using cached mypy_protobuf-1.18-py3-none-any.whl (7.3 kB) Collecting distlib==0.3.1 Using cached distlib-0.3.1-py2.py3-none-any.whl (335 kB) Collecting packaging>=14 Using cached packaging-21.3-py3-none-any.whl (40 kB) Collecting importlib-metadata<3,>=0.12 Using cached importlib_metadata-2.1.3-py2.py3-none-any.whl (10 kB) Collecting virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0 Using cached virtualenv-20.16.3-py2.py3-none-any.whl (8.8 MB) Collecting filelock>=3.0.0 Using cached filelock-3.8.0-py3-none-any.whl (10 kB) Collecting six>=1.14.0 Using cached six-1.16.0-py2.py3-none-any.whl (11 kB) Collecting pluggy>=0.12.0 Using cached pluggy-1.0.0-py2.py3-none-any.whl (13 kB) Collecting toml>=0.9.4 Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB) Collecting py>=1.4.17 Using cached py-1.11.0-py2.py3-none-any.whl (98 kB) Collecting grpcio>=1.37.0 Using cached grpcio-1.47.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.5 MB) Collecting protobuf<4.0dev,>=3.5.0.post1 Using cached protobuf-3.20.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB) Collecting zipp>=0.5 Using cached zipp-3.8.1-py3-none-any.whl (5.6 kB) Collecting pyparsing!=3.0.5,>=2.0.2 Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB) Collecting virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0 Using cached virtualenv-20.16.2-py2.py3-none-any.whl (8.8 MB) Collecting platformdirs<3,>=2 Using cached platformdirs-2.5.2-py3-none-any.whl (14 kB) Installing collected packages: distlib, zipp, wheel, toml, six, setuptools, pyparsing, py, protobuf, platformdirs, filelock, packaging, mypy-protobuf, importlib-metadata, grpcio, virtualenv, pluggy, grpcio-tools, tox Attempting uninstall: setuptools Found existing installation: setuptools 47.1.0 Uninstalling setuptools-47.1.0: Successfully uninstalled setuptools-47.1.0 Successfully installed distlib-0.3.1 filelock-3.8.0 grpcio-1.47.0 grpcio-tools-1.37.0 importlib-metadata-2.1.3 mypy-protobuf-1.18 packaging-21.3 platformdirs-2.5.2 pluggy-1.0.0 protobuf-3.20.1 py-1.11.0 pyparsing-3.0.9 setuptools-65.3.0 six-1.16.0 toml-0.10.2 tox-3.20.1 virtualenv-20.16.2 wheel-0.37.1 zipp-3.8.1 > Task :sdks:python:apache_beam:testing:load_tests:setupVirtualenv Collecting pip Using cached pip-22.2.2-py3-none-any.whl (2.0 MB) Installing collected packages: pip Attempting uninstall: pip Found existing installation: pip 20.1.1 Uninstalling pip-20.1.1: Successfully uninstalled pip-20.1.1 Successfully installed pip-22.2.2 Collecting tox==3.20.1 Using cached tox-3.20.1-py2.py3-none-any.whl (83 kB) Requirement already satisfied: setuptools in <https://ci-beam.apache.org/job/beam_LoadTests_Python_ParDo_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python3.7/site-packages> (from -r <https://ci-beam.apache.org/job/beam_LoadTests_Python_ParDo_Flink_Batch/ws/src/sdks/python/build-requirements.txt> (line 20)) (47.1.0) Collecting setuptools Using cached setuptools-65.3.0-py3-none-any.whl (1.2 MB) Collecting wheel>=0.36.0 Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB) Collecting grpcio-tools==1.37.0 Using cached grpcio_tools-1.37.0-cp37-cp37m-manylinux2014_x86_64.whl (2.5 MB) Collecting mypy-protobuf==1.18 Using cached mypy_protobuf-1.18-py3-none-any.whl (7.3 kB) Collecting distlib==0.3.1 Using cached distlib-0.3.1-py2.py3-none-any.whl (335 kB) Collecting six>=1.14.0 Using cached six-1.16.0-py2.py3-none-any.whl (11 kB) Collecting filelock>=3.0.0 Using cached filelock-3.8.0-py3-none-any.whl (10 kB) Collecting pluggy>=0.12.0 Using cached pluggy-1.0.0-py2.py3-none-any.whl (13 kB) Collecting toml>=0.9.4 Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB) Collecting packaging>=14 Using cached packaging-21.3-py3-none-any.whl (40 kB) Collecting virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0 Using cached virtualenv-20.16.3-py2.py3-none-any.whl (8.8 MB) Collecting importlib-metadata<3,>=0.12 Using cached importlib_metadata-2.1.3-py2.py3-none-any.whl (10 kB) Collecting py>=1.4.17 Using cached py-1.11.0-py2.py3-none-any.whl (98 kB) Collecting grpcio>=1.37.0 Using cached grpcio-1.47.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.5 MB) Collecting protobuf<4.0dev,>=3.5.0.post1 Using cached protobuf-3.20.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB) Collecting zipp>=0.5 Using cached zipp-3.8.1-py3-none-any.whl (5.6 kB) Collecting pyparsing!=3.0.5,>=2.0.2 Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB) Collecting virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0 Using cached virtualenv-20.16.2-py2.py3-none-any.whl (8.8 MB) Collecting platformdirs<3,>=2 Using cached platformdirs-2.5.2-py3-none-any.whl (14 kB) Installing collected packages: distlib, zipp, wheel, toml, six, setuptools, pyparsing, py, protobuf, platformdirs, filelock, packaging, mypy-protobuf, importlib-metadata, grpcio, virtualenv, pluggy, grpcio-tools, tox Attempting uninstall: setuptools Found existing installation: setuptools 47.1.0 Uninstalling setuptools-47.1.0: Successfully uninstalled setuptools-47.1.0 Successfully installed distlib-0.3.1 filelock-3.8.0 grpcio-1.47.0 grpcio-tools-1.37.0 importlib-metadata-2.1.3 mypy-protobuf-1.18 packaging-21.3 platformdirs-2.5.2 pluggy-1.0.0 protobuf-3.20.1 py-1.11.0 pyparsing-3.0.9 setuptools-65.3.0 six-1.16.0 toml-0.10.2 tox-3.20.1 virtualenv-20.16.2 wheel-0.37.1 zipp-3.8.1 > Task :sdks:python:sdist org/apache/beam/model/interactive/v1/beam_interactive_api.proto:36:1: warning: Import google/protobuf/timestamp.proto is unused. Writing mypy to org/apache/beam/model/pipeline/v1/external_transforms_pb2.pyi Writing mypy to org/apache/beam/model/pipeline/v1/beam_runner_api_pb2.pyi Writing mypy to org/apache/beam/model/pipeline/v1/standard_window_fns_pb2.pyi Writing mypy to org/apache/beam/model/pipeline/v1/metrics_pb2.pyi Writing mypy to org/apache/beam/model/pipeline/v1/endpoints_pb2.pyi Writing mypy to org/apache/beam/model/pipeline/v1/schema_pb2.pyi Writing mypy to org/apache/beam/model/job_management/v1/beam_artifact_api_pb2.pyi Writing mypy to org/apache/beam/model/job_management/v1/beam_expansion_api_pb2.pyi Writing mypy to org/apache/beam/model/job_management/v1/beam_job_api_pb2.pyi Writing mypy to org/apache/beam/model/fn_execution/v1/beam_provision_api_pb2.pyi Writing mypy to org/apache/beam/model/fn_execution/v1/beam_fn_api_pb2.pyi Writing mypy to org/apache/beam/model/interactive/v1/beam_interactive_api_pb2.pyi <https://ci-beam.apache.org/job/beam_LoadTests_Python_ParDo_Flink_Batch/ws/src/build/gradleenv/1922375555/lib/python3.7/site-packages/setuptools/dist.py>:530: UserWarning: Normalizing '2.42.0.dev' to '2.42.0.dev0' warnings.warn(tmpl.format(**locals())) warning: no files found matching 'README.md' warning: no files found matching 'NOTICE' warning: no files found matching 'LICENSE' warning: no files found matching 'LICENSE.python' warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md > Task :sdks:python:apache_beam:testing:load_tests:installGcpTest Processing <https://ci-beam.apache.org/job/beam_LoadTests_Python_ParDo_Flink_Batch/ws/src/sdks/python/build/apache-beam.tar.gz> Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Collecting crcmod<2.0,>=1.7 Using cached crcmod-1.7-cp37-cp37m-linux_x86_64.whl Collecting orjson<4.0 Downloading orjson-3.8.0.tar.gz (949 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 949.1/949.1 kB 15.8 MB/s eta 0:00:00 Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'error' error: subprocess-exited-with-error × Preparing metadata (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [6 lines of output] Cargo, the Rust package manager, is not installed or is not on PATH. This package requires Rust and Cargo to compile extensions. Install it through the system's package manager or via https://rustup.rs/ Checking for Rust toolchain.... [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed × Encountered error while generating package metadata. ╰─> See above for output. note: This is an issue with the package mentioned above, not pip. hint: See above for details. > Task :sdks:python:apache_beam:testing:load_tests:installGcpTest FAILED FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':sdks:python:apache_beam:testing:load_tests:installGcpTest'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Run with --scan to get full insights. * Get more help at https://help.gradle.org BUILD FAILED in 35s 14 actionable tasks: 8 executed, 4 from cache, 2 up-to-date Publishing build scan... https://gradle.com/s/inbcq6igvwhcu Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
