See
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/183/display/redirect?page=changes>
Changes:
[michal.walenia] [BEAM-8947] Change Flink docker image names in Jenkins jobs to
match
[ehudm] [BEAM-3713] pytest migration: py3x-{gcp,cython}
[younghoono] [BEAM-7970] Improved error help in Go PROTOBUF.md
[robertwb] Fixed instructions and the example of running Interactive Beam on
Flink.
------------------------------------------
[...truncated 151.64 KB...]
Operation completed over 3 objects/13.4 KiB.
+ create_cluster
+ local
metadata=flink-snapshot-url=https://archive.apache.org/dist/flink/flink-1.9.1/flink-1.9.1-bin-scala_2.11.tgz,
+ metadata+=flink-start-yarn-session=true,
+ metadata+=flink-taskmanager-slots=1,
+
metadata+=hadoop-jar-url=https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-7.0/flink-shaded-hadoop-2-uber-2.8.3-7.0.jar
+ [[ -n gcr.io/apache-beam-testing/beam_portability/python2.7_sdk:latest ]]
+
metadata+=,beam-sdk-harness-images-to-pull=gcr.io/apache-beam-testing/beam_portability/python2.7_sdk:latest
+ [[ -n gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server:latest
]]
+
metadata+=,beam-job-server-image=gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server:latest
+ local image_version=1.2
+ echo 'Starting dataproc cluster. Dataproc version: 1.2'
Starting dataproc cluster. Dataproc version: 1.2
+ local num_dataproc_workers=6
+ gcloud dataproc clusters create beam-loadtests-python-gbk-flink-batch-183
--region=global --num-workers=6 --initialization-actions
gs://beam-flink-cluster/init-actions/docker.sh,gs://beam-flink-cluster/init-actions/beam.sh,gs://beam-flink-cluster/init-actions/flink.sh
--metadata
flink-snapshot-url=https://archive.apache.org/dist/flink/flink-1.9.1/flink-1.9.1-bin-scala_2.11.tgz,flink-start-yarn-session=true,flink-taskmanager-slots=1,hadoop-jar-url=https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-7.0/flink-shaded-hadoop-2-uber-2.8.3-7.0.jar,beam-sdk-harness-images-to-pull=gcr.io/apache-beam-testing/beam_portability/python2.7_sdk:latest,beam-job-server-image=gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server:latest,
--image-version=1.2 --zone=us-central1-a --quiet
Waiting on operation
[projects/apache-beam-testing/regions/global/operations/7542ab1f-d3b7-3834-8a61-bedad6f2548c].
Waiting for cluster creation operation...
WARNING: For PD-Standard without local SSDs, we strongly recommend provisioning
1TB or larger to ensure consistently high I/O performance. See
https://cloud.google.com/compute/docs/disks/performance for information on disk
I/O performance.
..........................................................................................................................................................................................done.
Created
[https://dataproc.googleapis.com/v1/projects/apache-beam-testing/regions/global/clusters/beam-loadtests-python-gbk-flink-batch-183]
Cluster placed in zone [us-central1-a].
+ get_leader
+ local i=0
+ local application_ids
+ local application_masters
+ echo 'Yarn Applications'
Yarn Applications
++ gcloud compute ssh --zone=us-central1-a --quiet
yarn@beam-loadtests-python-gbk-flink-batch-183-m '--command=yarn application
-list'
++ grep beam-loadtests-python-gbk-flink-batch-183
Warning: Permanently added 'compute.2795878091033502135' (ECDSA) to the list of
known hosts.
19/12/14 13:13:56 INFO client.RMProxy: Connecting to ResourceManager at
beam-loadtests-python-gbk-flink-batch-183-m/10.128.0.50:8032
+ read line
+ echo application_1576329159792_0001 flink-dataproc Apache Flink yarn default
RUNNING UNDEFINED 100%
http://beam-loadtests-python-gbk-flink-batch-183-w-0.c.apache-beam-testing.internal:39911
application_1576329159792_0001 flink-dataproc Apache Flink yarn default RUNNING
UNDEFINED 100%
http://beam-loadtests-python-gbk-flink-batch-183-w-0.c.apache-beam-testing.internal:39911
++ echo application_1576329159792_0001 flink-dataproc Apache Flink yarn default
RUNNING UNDEFINED 100%
http://beam-loadtests-python-gbk-flink-batch-183-w-0.c.apache-beam-testing.internal:39911
++ sed 's/ .*//'
+ application_ids[$i]=application_1576329159792_0001
++ echo application_1576329159792_0001 flink-dataproc Apache Flink yarn default
RUNNING UNDEFINED 100%
http://beam-loadtests-python-gbk-flink-batch-183-w-0.c.apache-beam-testing.internal:39911
++ sed
's/.*beam-loadtests-python-gbk-flink-batch-183/beam-loadtests-python-gbk-flink-batch-183/'
++ sed 's/ .*//'
+
application_masters[$i]=beam-loadtests-python-gbk-flink-batch-183-w-0.c.apache-beam-testing.internal:39911
+ i=1
+ read line
+ '[' 1 '!=' 1 ']'
+
YARN_APPLICATION_MASTER=beam-loadtests-python-gbk-flink-batch-183-w-0.c.apache-beam-testing.internal:39911
+ echo 'Using Yarn Application master:
beam-loadtests-python-gbk-flink-batch-183-w-0.c.apache-beam-testing.internal:39911'
Using Yarn Application master:
beam-loadtests-python-gbk-flink-batch-183-w-0.c.apache-beam-testing.internal:39911
+ [[ -n gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server:latest
]]
+ start_job_server
+ gcloud compute ssh --zone=us-central1-a --quiet
yarn@beam-loadtests-python-gbk-flink-batch-183-m '--command=sudo --user yarn
docker run --detach --publish 8099:8099 --publish 8098:8098 --publish 8097:8097
--volume ~/.config/gcloud:/root/.config/gcloud
gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server:latest
--flink-master=beam-loadtests-python-gbk-flink-batch-183-w-0.c.apache-beam-testing.internal:39911
--artifacts-dir=gs://beam-flink-cluster/beam-loadtests-python-gbk-flink-batch-183'
60e1a7f37690ce49d9ab999cf55a9a4a247d16f3ad2642dac2e382971eac3b83
+ start_tunnel
++ gcloud compute ssh --quiet --zone=us-central1-a
yarn@beam-loadtests-python-gbk-flink-batch-183-m '--command=curl -s
"http://beam-loadtests-python-gbk-flink-batch-183-w-0.c.apache-beam-testing.internal:39911/jobmanager/config";'
+ local
'job_server_config=[{"key":"web.port","value":"0"},{"key":"jobmanager.execution.failover-strategy","value":"region"},{"key":"high-availability.cluster-id","value":"application_1576329159792_0001"},{"key":"jobmanager.rpc.address","value":"beam-loadtests-python-gbk-flink-batch-183-w-0.c.apache-beam-testing.internal"},{"key":"jobmanager.heap.mb","value":"12288"},{"key":"FLINK_PLUGINS_DIR","value":"/usr/lib/flink/plugins"},{"key":"web.tmpdir","value":"/tmp/flink-web-c1755b00-47cc-4eca-8e41-1d0433afca10"},{"key":"jobmanager.rpc.port","value":"39597"},{"key":"internal.io.tmpdirs.use-local-default","value":"true"},{"key":"io.tmp.dirs","value":"/hadoop/yarn/nm-local-dir/usercache/yarn/appcache/application_1576329159792_0001"},{"key":"taskmanager.network.numberOfBuffers","value":"2048"},{"key":"internal.cluster.execution-mode","value":"NORMAL"},{"key":"parallelism.default","value":"5"},{"key":"taskmanager.numberOfTaskSlots","value":"1"},{"key":"fs.hdfs.hadoopconf","value":"/etc/hadoop/conf"},{"key":"rest.address","value":"beam-loadtests-python-gbk-flink-batch-183-w-0.c.apache-beam-testing.internal"},{"key":"taskmanager.heap.mb","value":"12288"},{"key":"taskmanager.heap.size","value":"12288m"},{"key":"jobmanager.heap.size","value":"12288m"}]'
+ local key=jobmanager.rpc.port
++ echo
beam-loadtests-python-gbk-flink-batch-183-w-0.c.apache-beam-testing.internal:39911
++ cut -d : -f1
+ local
yarn_application_master_host=beam-loadtests-python-gbk-flink-batch-183-w-0.c.apache-beam-testing.internal
++ echo
'[{"key":"web.port","value":"0"},{"key":"jobmanager.execution.failover-strategy","value":"region"},{"key":"high-availability.cluster-id","value":"application_1576329159792_0001"},{"key":"jobmanager.rpc.address","value":"beam-loadtests-python-gbk-flink-batch-183-w-0.c.apache-beam-testing.internal"},{"key":"jobmanager.heap.mb","value":"12288"},{"key":"FLINK_PLUGINS_DIR","value":"/usr/lib/flink/plugins"},{"key":"web.tmpdir","value":"/tmp/flink-web-c1755b00-47cc-4eca-8e41-1d0433afca10"},{"key":"jobmanager.rpc.port","value":"39597"},{"key":"internal.io.tmpdirs.use-local-default","value":"true"},{"key":"io.tmp.dirs","value":"/hadoop/yarn/nm-local-dir/usercache/yarn/appcache/application_1576329159792_0001"},{"key":"taskmanager.network.numberOfBuffers","value":"2048"},{"key":"internal.cluster.execution-mode","value":"NORMAL"},{"key":"parallelism.default","value":"5"},{"key":"taskmanager.numberOfTaskSlots","value":"1"},{"key":"fs.hdfs.hadoopconf","value":"/etc/hadoop/conf"},{"key":"rest.address++
python -c 'import sys, json; print [ e['\''value'\''] for e in
json.load(sys.stdin) if e['\''key'\''] == u'\''jobmanager.rpc.port'\''][0]'
","value":"beam-loadtests-python-gbk-flink-batch-183-w-0.c.apache-beam-testing.internal"},{"key":"taskmanager.heap.mb","value":"12288"},{"key":"taskmanager.heap.size","value":"12288m"},{"key":"jobmanager.heap.size","value":"12288m"}]'
+ local jobmanager_rpc_port=39597
++ [[ true == \t\r\u\e ]]
++ echo ' -Nf >& /dev/null'
+ local 'detached_mode_params= -Nf >& /dev/null'
++ [[ -n gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server:latest
]]
++ echo '-L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097'
+ local 'job_server_ports_forwarding=-L 8099:localhost:8099 -L
8098:localhost:8098 -L 8097:localhost:8097'
+ local 'tunnel_command=gcloud compute ssh --zone=us-central1-a --quiet
yarn@beam-loadtests-python-gbk-flink-batch-183-m -- -L
8081:beam-loadtests-python-gbk-flink-batch-183-w-0.c.apache-beam-testing.internal:39911
-L
39597:beam-loadtests-python-gbk-flink-batch-183-w-0.c.apache-beam-testing.internal:39597
-L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097 -D 1080
-Nf >& /dev/null'
+ eval gcloud compute ssh --zone=us-central1-a --quiet
yarn@beam-loadtests-python-gbk-flink-batch-183-m -- -L
8081:beam-loadtests-python-gbk-flink-batch-183-w-0.c.apache-beam-testing.internal:39911
-L
39597:beam-loadtests-python-gbk-flink-batch-183-w-0.c.apache-beam-testing.internal:39597
-L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097 -D 1080
-Nf '>&' /dev/null
++ gcloud compute ssh --zone=us-central1-a --quiet
yarn@beam-loadtests-python-gbk-flink-batch-183-m -- -L
8081:beam-loadtests-python-gbk-flink-batch-183-w-0.c.apache-beam-testing.internal:39911
-L
39597:beam-loadtests-python-gbk-flink-batch-183-w-0.c.apache-beam-testing.internal:39597
-L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097 -D 1080
-Nf
[beam_LoadTests_Python_GBK_Flink_Batch] $ /bin/bash -xe
/tmp/jenkins8848763266173780123.sh
+ echo src Load test: 2GB of 10B records src
src Load test: 2GB of 10B records src
[Gradle] - Launching build.
[src] $
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/gradlew>
--continue --max-workers=12 -Dorg.gradle.jvmargs=-Xms2g
-Dorg.gradle.jvmargs=-Xmx4g
-PloadTest.mainClass=apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey
-Prunner=PortableRunner
'-PloadTest.args=--job_name=load_tests_Python_Flink_Batch_GBK_1_1214100241
--publish_to_big_query=true --project=apache-beam-testing
--metrics_dataset=load_test --metrics_table=python_flink_batch_GBK_1
--input_options='{"num_records": 200000000,"key_size": 1,"value_size":9}'
--iterations=1 --fanout=1 --parallelism=5 --job_endpoint=localhost:8099
--environment_config=gcr.io/apache-beam-testing/beam_portability/python2.7_sdk:latest
--environment_type=DOCKER --runner=PortableRunner'
:sdks:python:apache_beam:testing:load_tests:run
> Task :buildSrc:compileJava NO-SOURCE
> Task :buildSrc:compileGroovy UP-TO-DATE
> Task :buildSrc:pluginDescriptors UP-TO-DATE
> Task :buildSrc:processResources UP-TO-DATE
> Task :buildSrc:classes UP-TO-DATE
> Task :buildSrc:jar UP-TO-DATE
> Task :buildSrc:assemble UP-TO-DATE
> Task :buildSrc:spotlessGroovy UP-TO-DATE
> Task :buildSrc:spotlessGroovyCheck UP-TO-DATE
> Task :buildSrc:spotlessGroovyGradle UP-TO-DATE
> Task :buildSrc:spotlessGroovyGradleCheck UP-TO-DATE
> Task :buildSrc:spotlessCheck UP-TO-DATE
> Task :buildSrc:pluginUnderTestMetadata UP-TO-DATE
> Task :buildSrc:compileTestJava NO-SOURCE
> Task :buildSrc:compileTestGroovy NO-SOURCE
> Task :buildSrc:processTestResources NO-SOURCE
> Task :buildSrc:testClasses UP-TO-DATE
> Task :buildSrc:test NO-SOURCE
> Task :buildSrc:validateTaskProperties UP-TO-DATE
> Task :buildSrc:check UP-TO-DATE
> Task :buildSrc:build UP-TO-DATE
Configuration on demand is an incubating feature.
> Task :sdks:python:apache_beam:testing:load_tests:setupVirtualenv UP-TO-DATE
> Task :sdks:python:apache_beam:testing:load_tests:installGcpTest
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020.
Please upgrade your Python as Python 2.7 won't be maintained after that date. A
future version of pip will drop support for Python 2.7. More details about
Python 2 support in pip, can be found at
https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Obtaining
file://<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/sdks/python>
Requirement already satisfied: crcmod<2.0,>=1.7 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (1.7)
Requirement already satisfied: dill<0.3.2,>=0.3.1.1 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (0.3.1.1)
Requirement already satisfied: fastavro<0.22,>=0.21.4 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (0.21.24)
Requirement already satisfied: future<1.0.0,>=0.16.0 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (0.16.0)
Requirement already satisfied: grpcio<2,>=1.12.1 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (1.25.0)
Requirement already satisfied: hdfs<3.0.0,>=2.1.0 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (2.5.8)
Requirement already satisfied: httplib2<=0.12.0,>=0.8 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (0.12.0)
Requirement already satisfied: mock<3.0.0,>=1.0.1 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (2.0.0)
Requirement already satisfied: numpy<2,>=1.14.3 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (1.16.5)
Requirement already satisfied: pymongo<4.0.0,>=3.8.0 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (3.10.0)
Requirement already satisfied: oauth2client<4,>=2.0.1 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (3.0.0)
Requirement already satisfied: protobuf<4,>=3.5.0.post1 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (3.11.1)
Requirement already satisfied: pydot<2,>=1.2.0 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (1.4.1)
Requirement already satisfied: python-dateutil<3,>=2.8.0 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (2.8.1)
Requirement already satisfied: pytz>=2018.3 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (2019.3)
Requirement already satisfied: avro<2.0.0,>=1.8.1 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (1.9.1)
Requirement already satisfied: funcsigs<2,>=1.0.2 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (1.0.2)
Requirement already satisfied: futures<4.0.0,>=3.2.0 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (3.3.0)
Requirement already satisfied: pyvcf<0.7.0,>=0.6.8 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (0.6.8)
Requirement already satisfied: pyarrow<0.16.0,>=0.15.1 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (0.15.1)
Requirement already satisfied: typing<3.8.0,>=3.7.0 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (3.7.4.1)
Requirement already satisfied: typing-extensions<3.8.0,>=3.7.0 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (3.7.4.1)
Requirement already satisfied: cachetools<4,>=3.1.0 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (3.1.1)
Requirement already satisfied: google-apitools<0.5.29,>=0.5.28 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (0.5.28)
Requirement already satisfied: google-cloud-datastore<1.8.0,>=1.7.1 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (1.7.4)
Requirement already satisfied: google-cloud-pubsub<1.1.0,>=0.39.0 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (1.0.2)
Requirement already satisfied: google-cloud-bigquery<1.18.0,>=1.6.0 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (1.17.1)
Requirement already satisfied: google-cloud-core<2,>=0.28.1 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (1.1.0)
Requirement already satisfied: google-cloud-bigtable<1.1.0,>=0.31.1 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (1.0.0)
Requirement already satisfied: googledatastore<7.1,>=7.0.1 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (7.0.2)
Requirement already satisfied: proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0
in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (0.90.4)
Requirement already satisfied: nose>=1.3.7 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (1.3.7)
Requirement already satisfied: nose_xunitmp>=0.4.1 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (0.4.1)
Requirement already satisfied: pandas<0.25,>=0.23.4 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (0.24.2)
Requirement already satisfied: parameterized<0.7.0,>=0.6.0 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (0.6.3)
Requirement already satisfied: pyhamcrest<2.0,>=1.9 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (1.9.0)
Requirement already satisfied: pyyaml<6.0.0,>=3.12 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (5.2)
Requirement already satisfied: requests_mock<2.0,>=1.7 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (1.7.0)
Requirement already satisfied: tenacity<6.0,>=5.0.2 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (5.1.5)
Requirement already satisfied: pytest<5.0,>=4.4.0 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (4.6.7)
Requirement already satisfied: pytest-xdist<2,>=1.29.0 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from apache-beam==2.19.0.dev0) (1.30.0)
Requirement already satisfied: six>=1.5.2 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from grpcio<2,>=1.12.1->apache-beam==2.19.0.dev0) (1.13.0)
Requirement already satisfied: enum34>=1.0.4; python_version < "3.4" in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from grpcio<2,>=1.12.1->apache-beam==2.19.0.dev0) (1.1.6)
Requirement already satisfied: docopt in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from hdfs<3.0.0,>=2.1.0->apache-beam==2.19.0.dev0) (0.6.2)
Requirement already satisfied: requests>=2.7.0 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from hdfs<3.0.0,>=2.1.0->apache-beam==2.19.0.dev0) (2.22.0)
Requirement already satisfied: pbr>=0.11 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from mock<3.0.0,>=1.0.1->apache-beam==2.19.0.dev0) (5.4.4)
Requirement already satisfied: pyasn1>=0.1.7 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from oauth2client<4,>=2.0.1->apache-beam==2.19.0.dev0) (0.4.8)
Requirement already satisfied: pyasn1-modules>=0.0.5 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from oauth2client<4,>=2.0.1->apache-beam==2.19.0.dev0) (0.2.7)
Requirement already satisfied: rsa>=3.1.4 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from oauth2client<4,>=2.0.1->apache-beam==2.19.0.dev0) (4.0)
Requirement already satisfied: setuptools in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from protobuf<4,>=3.5.0.post1->apache-beam==2.19.0.dev0) (42.0.2)
Requirement already satisfied: pyparsing>=2.1.4 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from pydot<2,>=1.2.0->apache-beam==2.19.0.dev0) (2.4.5)
Requirement already satisfied: fasteners>=0.14 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from google-apitools<0.5.29,>=0.5.28->apache-beam==2.19.0.dev0) (0.15)
Requirement already satisfied: google-api-core[grpc]<2.0.0dev,>=1.6.0 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from google-cloud-datastore<1.8.0,>=1.7.1->apache-beam==2.19.0.dev0) (1.14.3)
Requirement already satisfied: grpc-google-iam-v1<0.13dev,>=0.12.3 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from google-cloud-pubsub<1.1.0,>=0.39.0->apache-beam==2.19.0.dev0) (0.12.3)
Requirement already satisfied: google-resumable-media<0.5.0dev,>=0.3.1 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from google-cloud-bigquery<1.18.0,>=1.6.0->apache-beam==2.19.0.dev0) (0.4.1)
Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.5.2 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from
proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0->apache-beam==2.19.0.dev0)
(1.6.0)
Requirement already satisfied: monotonic>=0.6; python_version == "2.7" in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from tenacity<6.0,>=5.0.2->apache-beam==2.19.0.dev0) (1.5)
Requirement already satisfied: atomicwrites>=1.0 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from pytest<5.0,>=4.4.0->apache-beam==2.19.0.dev0) (1.3.0)
Requirement already satisfied: packaging in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from pytest<5.0,>=4.4.0->apache-beam==2.19.0.dev0) (19.2)
Requirement already satisfied: wcwidth in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from pytest<5.0,>=4.4.0->apache-beam==2.19.0.dev0) (0.1.7)
Requirement already satisfied: importlib-metadata>=0.12; python_version < "3.8"
in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from pytest<5.0,>=4.4.0->apache-beam==2.19.0.dev0) (1.3.0)
Requirement already satisfied: py>=1.5.0 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from pytest<5.0,>=4.4.0->apache-beam==2.19.0.dev0) (1.8.0)
Requirement already satisfied: pathlib2>=2.2.0; python_version < "3.6" in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from pytest<5.0,>=4.4.0->apache-beam==2.19.0.dev0) (2.3.5)
Requirement already satisfied: pluggy<1.0,>=0.12 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from pytest<5.0,>=4.4.0->apache-beam==2.19.0.dev0) (0.13.1)
Requirement already satisfied: attrs>=17.4.0 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from pytest<5.0,>=4.4.0->apache-beam==2.19.0.dev0) (19.3.0)
Requirement already satisfied: more-itertools<6.0.0,>=4.0.0; python_version <=
"2.7" in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from pytest<5.0,>=4.4.0->apache-beam==2.19.0.dev0) (5.0.0)
Requirement already satisfied: pytest-forked in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from pytest-xdist<2,>=1.29.0->apache-beam==2.19.0.dev0) (1.1.3)
Requirement already satisfied: execnet>=1.1 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from pytest-xdist<2,>=1.29.0->apache-beam==2.19.0.dev0) (1.7.1)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.19.0.dev0) (1.25.7)
Requirement already satisfied: certifi>=2017.4.17 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.19.0.dev0)
(2019.11.28)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.19.0.dev0) (3.0.4)
Requirement already satisfied: idna<2.9,>=2.5 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.19.0.dev0) (2.8)
Requirement already satisfied: google-auth<2.0dev,>=0.4.0 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from
google-api-core[grpc]<2.0.0dev,>=1.6.0->google-cloud-datastore<1.8.0,>=1.7.1->apache-beam==2.19.0.dev0)
(1.8.2)
Requirement already satisfied: contextlib2; python_version < "3" in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from importlib-metadata>=0.12; python_version <
"3.8"->pytest<5.0,>=4.4.0->apache-beam==2.19.0.dev0) (0.6.0.post1)
Requirement already satisfied: zipp>=0.5 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from importlib-metadata>=0.12; python_version <
"3.8"->pytest<5.0,>=4.4.0->apache-beam==2.19.0.dev0) (0.6.0)
Requirement already satisfied: configparser>=3.5; python_version < "3" in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from importlib-metadata>=0.12; python_version <
"3.8"->pytest<5.0,>=4.4.0->apache-beam==2.19.0.dev0) (4.0.2)
Requirement already satisfied: scandir; python_version < "3.5" in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from pathlib2>=2.2.0; python_version <
"3.6"->pytest<5.0,>=4.4.0->apache-beam==2.19.0.dev0) (1.10.0)
Requirement already satisfied: apipkg>=1.4 in
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/lib/python2.7/site-packages>
(from execnet>=1.1->pytest-xdist<2,>=1.29.0->apache-beam==2.19.0.dev0) (1.5)
Installing collected packages: apache-beam
Found existing installation: apache-beam 2.19.0.dev0
Not uninstalling apache-beam at
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/sdks/python,>
outside environment
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227>
Can't uninstall 'apache-beam'. No files were found to uninstall.
Running setup.py develop for apache-beam
Successfully installed apache-beam
> Task :sdks:python:apache_beam:testing:load_tests:run
setup.py:232: UserWarning: You are using Apache Beam with Python 2. New
releases of Apache Beam will soon support Python 3 only.
'You are using Apache Beam with Python 2. '
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/local/lib/python2.7/site-packages/setuptools/dist.py>:476:
UserWarning: Normalizing '2.19.0.dev' to '2.19.0.dev0'
normalized_version,
running nosetests
running egg_info
writing requirements to apache_beam.egg-info/requires.txt
writing apache_beam.egg-info/PKG-INFO
writing top-level names to apache_beam.egg-info/top_level.txt
writing dependency_links to apache_beam.egg-info/dependency_links.txt
writing entry points to apache_beam.egg-info/entry_points.txt
reading manifest file 'apache_beam.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'README.md'
warning: no files found matching 'NOTICE'
warning: no files found matching 'LICENSE'
writing manifest file 'apache_beam.egg-info/SOURCES.txt'
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/sdks/python/apache_beam/__init__.py>:84:
UserWarning: You are using Apache Beam with Python 2. New releases of Apache
Beam will soon support Python 3 only.
'You are using Apache Beam with Python 2. '
testGroupByKey
(apache_beam.testing.load_tests.group_by_key_test.GroupByKeyTest) ... Terminated
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
The message received from the daemon indicates that the daemon has disappeared.
Build request sent: Build{id=4b96ec81-b578-44cc-994b-e619bd9a13eb,
currentDir=<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src}>
Attempting to read last messages from the daemon log...
Daemon pid: 2806
log file: /home/jenkins/.gradle/daemon/5.2.1/daemon-2806.out.log
----- Last 20 lines from daemon log file - daemon-2806.out.log -----
'You are using Apache Beam with Python 2. '
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/build/gradleenv/1329484227/local/lib/python2.7/site-packages/setuptools/dist.py>:476:
UserWarning: Normalizing '2.19.0.dev' to '2.19.0.dev0'
normalized_version,
running nosetests
running egg_info
writing requirements to apache_beam.egg-info/requires.txt
writing apache_beam.egg-info/PKG-INFO
writing top-level names to apache_beam.egg-info/top_level.txt
writing dependency_links to apache_beam.egg-info/dependency_links.txt
writing entry points to apache_beam.egg-info/entry_points.txt
reading manifest file 'apache_beam.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'README.md'
warning: no files found matching 'NOTICE'
warning: no files found matching 'LICENSE'
writing manifest file 'apache_beam.egg-info/SOURCES.txt'
<https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/ws/src/sdks/python/apache_beam/__init__.py>:84:
UserWarning: You are using Apache Beam with Python 2. New releases of Apache
Beam will soon support Python 3 only.
'You are using Apache Beam with Python 2. '
testGroupByKey
(apache_beam.testing.load_tests.group_by_key_test.GroupByKeyTest) ... Terminated
Daemon vm is shutting down... The daemon has exited normally or was terminated
in response to a user interrupt.
----- End of the daemon log -----
FAILURE: Build failed with an exception.
* What went wrong:
Gradle build daemon disappeared unexpectedly (it may have been killed or may
have crashed)
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
