See <https://builds.apache.org/job/beam_LoadTests_Java_GBK_Flink_Streaming/1/display/redirect>
Changes: ------------------------------------------ [...truncated 72.57 KB...] / [1 files][ 2.3 KiB/ 6.0 KiB] / [2 files][ 6.0 KiB/ 6.0 KiB] Copying file://init-actions/flink.sh [Content-Type=text/x-sh]... / [2 files][ 6.0 KiB/ 13.4 KiB] / [3 files][ 13.4 KiB/ 13.4 KiB] - Operation completed over 3 objects/13.4 KiB. + create_cluster + local metadata=flink-snapshot-url=https://archive.apache.org/dist/flink/flink-1.9.1/flink-1.9.1-bin-scala_2.11.tgz, + metadata+=flink-start-yarn-session=true, + metadata+=flink-taskmanager-slots=1, + metadata+=hadoop-jar-url=https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-7.0/flink-shaded-hadoop-2-uber-2.8.3-7.0.jar + [[ -n gcr.io/apache-beam-testing/beam_portability/java_sdk:latest ]] + metadata+=,beam-sdk-harness-images-to-pull=gcr.io/apache-beam-testing/beam_portability/java_sdk:latest + [[ -n gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server:latest ]] + metadata+=,beam-job-server-image=gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server:latest + local image_version=1.2 + echo 'Starting dataproc cluster. Dataproc version: 1.2' Starting dataproc cluster. Dataproc version: 1.2 + local num_dataproc_workers=17 + gcloud dataproc clusters create beam-loadtests-python-gbk-flink-batch-1 --region=global --num-workers=17 --initialization-actions gs://beam-flink-cluster/init-actions/docker.sh,gs://beam-flink-cluster/init-actions/beam.sh,gs://beam-flink-cluster/init-actions/flink.sh --metadata flink-snapshot-url=https://archive.apache.org/dist/flink/flink-1.9.1/flink-1.9.1-bin-scala_2.11.tgz,flink-start-yarn-session=true,flink-taskmanager-slots=1,hadoop-jar-url=https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-7.0/flink-shaded-hadoop-2-uber-2.8.3-7.0.jar,beam-sdk-harness-images-to-pull=gcr.io/apache-beam-testing/beam_portability/java_sdk:latest,beam-job-server-image=gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server:latest, --image-version=1.2 --zone=us-central1-a --quiet Waiting on operation [projects/apache-beam-testing/regions/global/operations/d2d3188a-2e6c-3c9c-b1af-b2f9d4d973dd]. Waiting for cluster creation operation... WARNING: For PD-Standard without local SSDs, we strongly recommend provisioning 1TB or larger to ensure consistently high I/O performance. See https://cloud.google.com/compute/docs/disks/performance for information on disk I/O performance. .....................................................................................................................................................................................done. Created [https://dataproc.googleapis.com/v1/projects/apache-beam-testing/regions/global/clusters/beam-loadtests-python-gbk-flink-batch-1] Cluster placed in zone [us-central1-a]. + get_leader + local i=0 + local application_ids + local application_masters + echo 'Yarn Applications' Yarn Applications ++ gcloud compute ssh --zone=us-central1-a --quiet yarn@beam-loadtests-python-gbk-flink-batch-1-m '--command=yarn application -list' ++ grep beam-loadtests-python-gbk-flink-batch-1 Warning: Permanently added 'compute.5437252805928652547' (ECDSA) to the list of known hosts. 19/12/12 12:29:24 INFO client.RMProxy: Connecting to ResourceManager at beam-loadtests-python-gbk-flink-batch-1-m/10.128.0.26:8032 + read line + echo application_1576153691181_0001 flink-dataproc Apache Flink yarn default RUNNING UNDEFINED 100% http://beam-loadtests-python-gbk-flink-batch-1-w-16.c.apache-beam-testing.internal:34003 application_1576153691181_0001 flink-dataproc Apache Flink yarn default RUNNING UNDEFINED 100% http://beam-loadtests-python-gbk-flink-batch-1-w-16.c.apache-beam-testing.internal:34003 ++ echo application_1576153691181_0001 flink-dataproc Apache Flink yarn default RUNNING UNDEFINED 100% http://beam-loadtests-python-gbk-flink-batch-1-w-16.c.apache-beam-testing.internal:34003 ++ sed 's/ .*//' + application_ids[$i]=application_1576153691181_0001 ++ echo application_1576153691181_0001 flink-dataproc Apache Flink yarn default RUNNING UNDEFINED 100% http://beam-loadtests-python-gbk-flink-batch-1-w-16.c.apache-beam-testing.internal:34003 ++ sed 's/.*beam-loadtests-python-gbk-flink-batch-1/beam-loadtests-python-gbk-flink-batch-1/' ++ sed 's/ .*//' + application_masters[$i]=beam-loadtests-python-gbk-flink-batch-1-w-16.c.apache-beam-testing.internal:34003 + i=1 + read line + '[' 1 '!=' 1 ']' + YARN_APPLICATION_MASTER=beam-loadtests-python-gbk-flink-batch-1-w-16.c.apache-beam-testing.internal:34003 + echo 'Using Yarn Application master: beam-loadtests-python-gbk-flink-batch-1-w-16.c.apache-beam-testing.internal:34003' Using Yarn Application master: beam-loadtests-python-gbk-flink-batch-1-w-16.c.apache-beam-testing.internal:34003 + [[ -n gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server:latest ]] + start_job_server + gcloud compute ssh --zone=us-central1-a --quiet yarn@beam-loadtests-python-gbk-flink-batch-1-m '--command=sudo --user yarn docker run --detach --publish 8099:8099 --publish 8098:8098 --publish 8097:8097 --volume ~/.config/gcloud:/root/.config/gcloud gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server:latest --flink-master=beam-loadtests-python-gbk-flink-batch-1-w-16.c.apache-beam-testing.internal:34003 --artifacts-dir=gs://beam-flink-cluster/beam-loadtests-python-gbk-flink-batch-1' 1efef7e4acfadb4f7869edf5926981e02fc7e25c446731eabe6b42bca60fbe5b + start_tunnel ++ gcloud compute ssh --quiet --zone=us-central1-a yarn@beam-loadtests-python-gbk-flink-batch-1-m '--command=curl -s "http://beam-loadtests-python-gbk-flink-batch-1-w-16.c.apache-beam-testing.internal:34003/jobmanager/config";' + local 'job_server_config=[{"key":"web.port","value":"0"},{"key":"jobmanager.execution.failover-strategy","value":"region"},{"key":"high-availability.cluster-id","value":"application_1576153691181_0001"},{"key":"jobmanager.rpc.address","value":"beam-loadtests-python-gbk-flink-batch-1-w-16.c.apache-beam-testing.internal"},{"key":"jobmanager.heap.mb","value":"12288"},{"key":"FLINK_PLUGINS_DIR","value":"/usr/lib/flink/plugins"},{"key":"web.tmpdir","value":"/tmp/flink-web-27d9b034-c33e-41c7-a381-2debed855ca0"},{"key":"jobmanager.rpc.port","value":"36249"},{"key":"internal.io.tmpdirs.use-local-default","value":"true"},{"key":"io.tmp.dirs","value":"/hadoop/yarn/nm-local-dir/usercache/yarn/appcache/application_1576153691181_0001"},{"key":"taskmanager.network.numberOfBuffers","value":"2048"},{"key":"internal.cluster.execution-mode","value":"NORMAL"},{"key":"parallelism.default","value":"16"},{"key":"taskmanager.numberOfTaskSlots","value":"1"},{"key":"fs.hdfs.hadoopconf","value":"/etc/hadoop/conf"},{"key":"rest.address","value":"beam-loadtests-python-gbk-flink-batch-1-w-16.c.apache-beam-testing.internal"},{"key":"taskmanager.heap.mb","value":"12288"},{"key":"taskmanager.heap.size","value":"12288m"},{"key":"jobmanager.heap.size","value":"12288m"}]' + local key=jobmanager.rpc.port ++ cut -d : -f1 ++ echo beam-loadtests-python-gbk-flink-batch-1-w-16.c.apache-beam-testing.internal:34003 + local yarn_application_master_host=beam-loadtests-python-gbk-flink-batch-1-w-16.c.apache-beam-testing.internal ++ python -c 'import sys, json; print [ e['\''value'\''] for e in json.load(sys.stdin) if e['\''key'\''] == u'\''jobmanager.rpc.port'\''][0]' ++ echo '[{"key":"web.port","value":"0"},{"key":"jobmanager.execution.failover-strategy","value":"region"},{"key":"high-availability.cluster-id","value":"application_1576153691181_0001"},{"key":"jobmanager.rpc.address","value":"beam-loadtests-python-gbk-flink-batch-1-w-16.c.apache-beam-testing.internal"},{"key":"jobmanager.heap.mb","value":"12288"},{"key":"FLINK_PLUGINS_DIR","value":"/usr/lib/flink/plugins"},{"key":"web.tmpdir","value":"/tmp/flink-web-27d9b034-c33e-41c7-a381-2debed855ca0"},{"key":"jobmanager.rpc.port","value":"36249"},{"key":"internal.io.tmpdirs.use-local-default","value":"true"},{"key":"io.tmp.dirs","value":"/hadoop/yarn/nm-local-dir/usercache/yarn/appcache/application_1576153691181_0001"},{"key":"taskmanager.network.numberOfBuffers","value":"2048"},{"key":"internal.cluster.execution-mode","value":"NORMAL"},{"key":"parallelism.default","value":"16"},{"key":"taskmanager.numberOfTaskSlots","value":"1"},{"key":"fs.hdfs.hadoopconf","value":"/etc/hadoop/conf"},{"key":"rest.address","value":"beam-loadtests-python-gbk-flink-batch-1-w-16.c.apache-beam-testing.internal"},{"key":"taskmanager.heap.mb","value":"12288"},{"key":"taskmanager.heap.size","value":"12288m"},{"key":"jobmanager.heap.size","value":"12288m"}]' + local jobmanager_rpc_port=36249 ++ [[ true == \t\r\u\e ]] ++ echo ' -Nf >& /dev/null' + local 'detached_mode_params= -Nf >& /dev/null' ++ [[ -n gcr.io/apache-beam-testing/beam_portability/flink1.9_job_server:latest ]] ++ echo '-L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097' + local 'job_server_ports_forwarding=-L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097' + local 'tunnel_command=gcloud compute ssh --zone=us-central1-a --quiet yarn@beam-loadtests-python-gbk-flink-batch-1-m -- -L 8081:beam-loadtests-python-gbk-flink-batch-1-w-16.c.apache-beam-testing.internal:34003 -L 36249:beam-loadtests-python-gbk-flink-batch-1-w-16.c.apache-beam-testing.internal:36249 -L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097 -D 1080 -Nf >& /dev/null' + eval gcloud compute ssh --zone=us-central1-a --quiet yarn@beam-loadtests-python-gbk-flink-batch-1-m -- -L 8081:beam-loadtests-python-gbk-flink-batch-1-w-16.c.apache-beam-testing.internal:34003 -L 36249:beam-loadtests-python-gbk-flink-batch-1-w-16.c.apache-beam-testing.internal:36249 -L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097 -D 1080 -Nf '>&' /dev/null ++ gcloud compute ssh --zone=us-central1-a --quiet yarn@beam-loadtests-python-gbk-flink-batch-1-m -- -L 8081:beam-loadtests-python-gbk-flink-batch-1-w-16.c.apache-beam-testing.internal:34003 -L 36249:beam-loadtests-python-gbk-flink-batch-1-w-16.c.apache-beam-testing.internal:36249 -L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097 -D 1080 -Nf [beam_LoadTests_Java_GBK_Flink_Streaming] $ /bin/bash -xe /tmp/jenkins5443152237852459691.sh + echo src Load test: fanout 4 times with 2GB 10-byte records total src src Load test: fanout 4 times with 2GB 10-byte records total src [Gradle] - Launching build. [src] $ <https://builds.apache.org/job/beam_LoadTests_Java_GBK_Flink_Streaming/ws/src/gradlew> --continue --max-workers=12 -Dorg.gradle.jvmargs=-Xms2g -Dorg.gradle.jvmargs=-Xmx4g -PloadTest.mainClass=org.apache.beam.sdk.loadtests.GroupByKeyLoadTest -Prunner=:runners:portability:java '-PloadTest.args=--project=apache-beam-testing --appName=load_tests_Java_Flink_streaming_GBK_4 --tempLocation=gs://temp-storage-for-perf-tests/loadtests --publishToBigQuery=true --bigQueryDataset=load_test --bigQueryTable=java_flink_streaming_GBK_4 --sourceOptions={"numRecords":5000000,"keySizeBytes":10,"valueSizeBytes":90} --fanout=4 --iterations=1 --sdkWorkerParallelism=16 --streaming=true --jobEndpoint=localhost:8099 --defaultEnvironmentConfig=gcr.io/apache-beam-testing/beam_portability/java_sdk:latest --defaultEnvironmentType=DOCKER --runner=PortableRunner' :sdks:java:testing:load-tests:run > Task :buildSrc:compileJava NO-SOURCE > Task :buildSrc:compileGroovy UP-TO-DATE > Task :buildSrc:pluginDescriptors UP-TO-DATE > Task :buildSrc:processResources UP-TO-DATE > Task :buildSrc:classes UP-TO-DATE > Task :buildSrc:jar UP-TO-DATE > Task :buildSrc:assemble UP-TO-DATE > Task :buildSrc:spotlessGroovy UP-TO-DATE > Task :buildSrc:spotlessGroovyCheck UP-TO-DATE > Task :buildSrc:spotlessGroovyGradle UP-TO-DATE > Task :buildSrc:spotlessGroovyGradleCheck UP-TO-DATE > Task :buildSrc:spotlessCheck UP-TO-DATE > Task :buildSrc:pluginUnderTestMetadata UP-TO-DATE > Task :buildSrc:compileTestJava NO-SOURCE > Task :buildSrc:compileTestGroovy NO-SOURCE > Task :buildSrc:processTestResources NO-SOURCE > Task :buildSrc:testClasses UP-TO-DATE > Task :buildSrc:test NO-SOURCE > Task :buildSrc:validateTaskProperties UP-TO-DATE > Task :buildSrc:check UP-TO-DATE > Task :buildSrc:build UP-TO-DATE Configuration on demand is an incubating feature. > Task :sdks:java:core:generateAvroProtocol NO-SOURCE > Task :runners:core-construction-java:processResources NO-SOURCE > Task :runners:core-java:processResources NO-SOURCE > Task :sdks:java:fn-execution:processResources NO-SOURCE > Task :vendor:sdks-java-extensions-protobuf:processResources NO-SOURCE > Task :model:job-management:extractProto UP-TO-DATE > Task :model:fn-execution:extractProto UP-TO-DATE > Task :sdks:java:harness:processResources NO-SOURCE > Task :runners:java-fn-execution:processResources NO-SOURCE > Task :sdks:java:extensions:google-cloud-platform-core:processResources > NO-SOURCE > Task :runners:local-java:processResources NO-SOURCE > Task :runners:direct-java:processResources NO-SOURCE > Task :model:job-management:processResources UP-TO-DATE > Task :sdks:java:io:google-cloud-platform:processResources NO-SOURCE > Task :runners:portability:java:processResources NO-SOURCE > Task :sdks:java:extensions:protobuf:extractProto UP-TO-DATE > Task :model:fn-execution:processResources UP-TO-DATE > Task :sdks:java:io:kafka:processResources NO-SOURCE > Task :sdks:java:core:generateAvroJava NO-SOURCE > Task :sdks:java:io:kinesis:processResources NO-SOURCE > Task :sdks:java:io:synthetic:processResources NO-SOURCE > Task :sdks:java:testing:test-utils:processResources NO-SOURCE > Task :sdks:java:extensions:protobuf:processResources NO-SOURCE > Task :sdks:java:testing:load-tests:processResources NO-SOURCE > Task :sdks:java:core:generateGrammarSource UP-TO-DATE > Task :sdks:java:core:processResources UP-TO-DATE > Task :model:pipeline:extractIncludeProto UP-TO-DATE > Task :model:pipeline:extractProto UP-TO-DATE > Task :model:pipeline:generateProto UP-TO-DATE > Task :model:pipeline:compileJava UP-TO-DATE > Task :model:pipeline:processResources UP-TO-DATE > Task :model:pipeline:classes UP-TO-DATE > Task :model:pipeline:jar UP-TO-DATE > Task :model:pipeline:shadowJar UP-TO-DATE > Task :model:job-management:extractIncludeProto UP-TO-DATE > Task :model:fn-execution:extractIncludeProto UP-TO-DATE > Task :model:job-management:generateProto UP-TO-DATE > Task :model:fn-execution:generateProto UP-TO-DATE > Task :model:job-management:compileJava UP-TO-DATE > Task :model:job-management:classes UP-TO-DATE > Task :model:fn-execution:compileJava UP-TO-DATE > Task :model:fn-execution:classes UP-TO-DATE > Task :model:job-management:shadowJar UP-TO-DATE > Task :model:fn-execution:shadowJar UP-TO-DATE > Task :sdks:java:core:compileJava UP-TO-DATE > Task :sdks:java:core:classes UP-TO-DATE > Task :sdks:java:core:shadowJar UP-TO-DATE > Task :sdks:java:extensions:protobuf:extractIncludeProto UP-TO-DATE > Task :sdks:java:extensions:protobuf:generateProto NO-SOURCE > Task :runners:core-construction-java:compileJava UP-TO-DATE > Task :sdks:java:fn-execution:compileJava UP-TO-DATE > Task :runners:core-construction-java:classes UP-TO-DATE > Task :sdks:java:fn-execution:classes UP-TO-DATE > Task :vendor:sdks-java-extensions-protobuf:compileJava UP-TO-DATE > Task :vendor:sdks-java-extensions-protobuf:classes UP-TO-DATE > Task :runners:local-java:compileJava FROM-CACHE > Task :runners:local-java:classes UP-TO-DATE > Task :sdks:java:extensions:google-cloud-platform-core:compileJava UP-TO-DATE > Task :sdks:java:extensions:google-cloud-platform-core:classes UP-TO-DATE > Task :sdks:java:fn-execution:jar UP-TO-DATE > Task :sdks:java:extensions:google-cloud-platform-core:jar UP-TO-DATE > Task :vendor:sdks-java-extensions-protobuf:shadowJar UP-TO-DATE > Task :runners:core-construction-java:jar UP-TO-DATE > Task :sdks:java:extensions:protobuf:compileJava UP-TO-DATE > Task :sdks:java:extensions:protobuf:classes UP-TO-DATE > Task :sdks:java:extensions:protobuf:jar UP-TO-DATE > Task :runners:local-java:jar > Task :runners:core-java:compileJava UP-TO-DATE > Task :runners:core-java:classes UP-TO-DATE > Task :runners:core-java:jar UP-TO-DATE > Task :sdks:java:harness:compileJava UP-TO-DATE > Task :sdks:java:harness:classes UP-TO-DATE > Task :sdks:java:harness:jar UP-TO-DATE > Task :sdks:java:harness:shadowJar UP-TO-DATE > Task :runners:java-fn-execution:compileJava UP-TO-DATE > Task :runners:java-fn-execution:classes UP-TO-DATE > Task :runners:java-fn-execution:jar UP-TO-DATE > Task :sdks:java:io:synthetic:compileJava FROM-CACHE > Task :sdks:java:io:synthetic:classes UP-TO-DATE > Task :sdks:java:io:synthetic:jar > Task :sdks:java:io:kafka:compileJava UP-TO-DATE > Task :sdks:java:io:kafka:classes UP-TO-DATE > Task :sdks:java:io:kafka:jar UP-TO-DATE > Task :sdks:java:testing:test-utils:compileJava FROM-CACHE > Task :sdks:java:testing:test-utils:classes UP-TO-DATE > Task :sdks:java:testing:test-utils:jar > Task :sdks:java:io:google-cloud-platform:compileJava UP-TO-DATE > Task :sdks:java:io:google-cloud-platform:classes UP-TO-DATE > Task :sdks:java:io:google-cloud-platform:jar UP-TO-DATE > Task :runners:portability:java:compileJava > Task :runners:portability:java:classes > Task :runners:portability:java:jar > Task :sdks:java:io:kinesis:compileJava > Task :sdks:java:io:kinesis:classes > Task :sdks:java:io:kinesis:jar > Task :runners:direct-java:compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. > Task :runners:direct-java:classes > Task :runners:direct-java:shadowJar > Task :sdks:java:testing:load-tests:compileJava Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. > Task :sdks:java:testing:load-tests:classes > Task :sdks:java:testing:load-tests:jar > Task :sdks:java:testing:load-tests:run FAILED SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Exception in thread "main" java.lang.IllegalStateException: GroupByKey cannot be applied to non-bounded PCollection in the GlobalWindow without a trigger. Use a Window.into or Window.triggering transform prior to GroupByKey. at org.apache.beam.sdk.transforms.GroupByKey.applicableTo(GroupByKey.java:156) at org.apache.beam.sdk.transforms.GroupByKey.expand(GroupByKey.java:226) at org.apache.beam.sdk.transforms.GroupByKey.expand(GroupByKey.java:110) at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:539) at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:490) at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:368) at org.apache.beam.sdk.loadtests.GroupByKeyLoadTest.loadTest(GroupByKeyLoadTest.java:97) at org.apache.beam.sdk.loadtests.LoadTest.run(LoadTest.java:96) at org.apache.beam.sdk.loadtests.GroupByKeyLoadTest.run(GroupByKeyLoadTest.java:57) at org.apache.beam.sdk.loadtests.GroupByKeyLoadTest.main(GroupByKeyLoadTest.java:131) FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':sdks:java:testing:load-tests:run'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 16s 61 actionable tasks: 12 executed, 3 from cache, 46 up-to-date Publishing build scan... https://scans.gradle.com/s/woctfj6eirr4i Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
