See <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/467/display/redirect?page=changes>
Changes: [yoshiki.obata] [BEAM-9980] add groovy functions for python versions [yoshiki.obata] [BEAM-9980] update dataflow test-suites to switch python versions using [katarzyna.kucharczyk] [BEAM-10672] Added streaming option to combine test [katarzyna.kucharczyk] [BEAM-10672] Added experimental dataflow param [katarzyna.kucharczyk] [BEAM-10672] Spottless Apply [katarzyna.kucharczyk] [BEAM-10672] Updated streaming commands [heejong] [BEAM-10679] improving XLang KafkaIO streaming test [heejong] fix formatting [heejong] fix incorrect coder issue [heejong] use Create [yoshiki.obata] [BEAM-9980] version switchable dataflow tasks to be invoked [yoshiki.obata] [BEAM-9980] :sdks:python:test-suites:dataflow included in [katarzyna.kucharczyk] [BEAM-10672] Fixes after review in combine python load test [katarzyna.kucharczyk] [BEAM-10672] Changed jobType variable into mode to declare if it is [tobiasz.kedzierski] Improve CI documentation [Luke Cwik] [BEAM-10670] Make key coder deterministic by using upstream PCollection [Udi Meiri] [BEAM-10697] Remove testPy2Cython from precommit [noreply] [BEAM-9891] Generate query execution summary table after finishing jobs [noreply] [BEAM-9919] Added an External Transform API to Go SDK (#12445) ------------------------------------------ [...truncated 19.77 KB...] > Task :model:job-management:classes > Task :model:fn-execution:compileJava FROM-CACHE > Task :model:fn-execution:classes > Task :model:pipeline:shadowJar > Task :model:job-management:shadowJar > Task :model:fn-execution:shadowJar > Task :sdks:java:core:compileJava FROM-CACHE > Task :sdks:java:core:classes > Task :sdks:java:core:shadowJar > Task :runners:local-java:compileJava FROM-CACHE > Task :runners:local-java:classes UP-TO-DATE > Task :runners:core-construction-java:compileJava FROM-CACHE > Task :runners:core-construction-java:classes UP-TO-DATE > Task :runners:local-java:jar > Task :sdks:java:extensions:protobuf:extractIncludeProto > Task :sdks:java:extensions:protobuf:generateProto NO-SOURCE > Task :vendor:sdks-java-extensions-protobuf:compileJava FROM-CACHE > Task :vendor:sdks-java-extensions-protobuf:classes UP-TO-DATE > Task :sdks:java:io:synthetic:compileJava FROM-CACHE > Task :sdks:java:io:synthetic:classes UP-TO-DATE > Task :sdks:java:extensions:protobuf:compileJava FROM-CACHE > Task :sdks:java:extensions:protobuf:classes UP-TO-DATE > Task :sdks:java:io:synthetic:jar > Task :sdks:java:extensions:google-cloud-platform-core:compileJava FROM-CACHE > Task :sdks:java:io:kinesis:compileJava FROM-CACHE > Task :sdks:java:extensions:google-cloud-platform-core:classes UP-TO-DATE > Task :sdks:java:io:kinesis:classes UP-TO-DATE > Task :sdks:java:extensions:protobuf:jar > Task :sdks:java:fn-execution:compileJava FROM-CACHE > Task :sdks:java:extensions:google-cloud-platform-core:jar > Task :sdks:java:fn-execution:classes UP-TO-DATE > Task :sdks:java:io:kinesis:jar > Task :sdks:java:fn-execution:jar > Task :vendor:sdks-java-extensions-protobuf:shadowJar > Task :sdks:java:testing:test-utils:compileJava FROM-CACHE > Task :sdks:java:testing:test-utils:classes UP-TO-DATE > Task :runners:core-construction-java:jar > Task :sdks:java:testing:test-utils:jar > Task :runners:core-java:compileJava FROM-CACHE > Task :runners:core-java:classes UP-TO-DATE > Task :runners:core-java:jar > Task :sdks:java:harness:compileJava FROM-CACHE > Task :sdks:java:harness:classes UP-TO-DATE > Task :sdks:java:harness:jar > Task :sdks:java:harness:shadowJar > Task :runners:java-fn-execution:compileJava FROM-CACHE > Task :runners:java-fn-execution:classes UP-TO-DATE > Task :runners:java-fn-execution:jar > Task :sdks:java:expansion-service:compileJava FROM-CACHE > Task :sdks:java:expansion-service:classes UP-TO-DATE > Task :sdks:java:expansion-service:jar > Task :runners:direct-java:compileJava FROM-CACHE > Task :runners:direct-java:classes UP-TO-DATE > Task :sdks:java:io:kafka:compileJava FROM-CACHE > Task :sdks:java:io:kafka:classes UP-TO-DATE > Task :sdks:java:io:kafka:jar > Task :sdks:java:io:google-cloud-platform:compileJava FROM-CACHE > Task :sdks:java:io:google-cloud-platform:classes UP-TO-DATE > Task :sdks:java:io:google-cloud-platform:jar > Task :runners:google-cloud-dataflow-java:compileJava FROM-CACHE > Task :runners:google-cloud-dataflow-java:classes > Task :runners:google-cloud-dataflow-java:jar > Task :runners:google-cloud-dataflow-java:****:legacy-****:compileJava > FROM-CACHE > Task :runners:google-cloud-dataflow-java:****:legacy-****:classes UP-TO-DATE > Task :runners:direct-java:shadowJar > Task :sdks:java:testing:load-tests:compileJava FROM-CACHE > Task :sdks:java:testing:load-tests:classes UP-TO-DATE > Task :sdks:java:testing:load-tests:jar > Task :runners:google-cloud-dataflow-java:****:legacy-****:shadowJar > Task :sdks:java:testing:load-tests:run Aug 18, 2020 12:07:49 PM org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory create INFO: No stagingLocation provided, falling back to gcpTempLocation Aug 18, 2020 12:07:50 PM org.apache.beam.runners.dataflow.DataflowRunner fromOptions INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 181 files. Enable logging at DEBUG level to see which files will be staged. Aug 18, 2020 12:07:50 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services. Aug 18, 2020 12:07:53 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Uploading 182 files from PipelineOptions.filesToStage to staging location to prepare for execution. Aug 18, 2020 12:07:53 PM org.apache.beam.runners.dataflow.util.PackageUtil$PackageAttributes forFileToStage INFO: Staging custom dataflow-****.jar as beam-runners-google-cloud-dataflow-java-legacy-****-2.25.0-SNAPSHOT-17jDKJCo1niyleelXT3gbXAQDKFn_WSRCfYkZmGDuKY.jar Aug 18, 2020 12:07:53 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/sdks/java/testing/load-tests/build/libs/beam-sdks-java-load-tests-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-load-tests-2.25.0-SNAPSHOT-oLEaT5Uctk8DORvK0JEVlmKtcpxIpygS7hepKb1r2I0.jar Aug 18, 2020 12:07:53 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/sdks/java/testing/test-utils/build/libs/beam-sdks-java-test-utils-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-test-utils-2.25.0-SNAPSHOT-8UkVDsuWAcKd5dDlDJxCcUA8TLIFXqZzxWiOAKHtUTI.jar Aug 18, 2020 12:07:53 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/sdks/java/extensions/google-cloud-platform-core/build/libs/beam-sdks-java-extensions-google-cloud-platform-core-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-extensions-google-cloud-platform-core-2.25.0-SNAPSHOT-VnyCx0GMD5AbZFiPUnE13PMvYnxqxK4SCfKbCfWbA8M.jar Aug 18, 2020 12:07:53 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /home/jenkins/.gradle/caches/modules-2/files-2.1/io.confluent/kafka-avro-serializer/5.3.2/8206e639ac28c3b7557e32ed6801c6b4abd039e5/kafka-avro-serializer-5.3.2.jar to gs://temp-storage-for-perf-tests/loadtests/staging/kafka-avro-serializer-5.3.2-7FAoV2nngjZLbCXD3IPBOus_R_2KrM_zffAaNlVOls4.jar Aug 18, 2020 12:07:53 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/sdks/java/io/kafka/build/libs/beam-sdks-java-io-kafka-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-io-kafka-2.25.0-SNAPSHOT-TYDwQxKQ-Q60hZcYOy3XjG-gIrr89GQs03GiwNtW8HA.jar Aug 18, 2020 12:07:53 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /home/jenkins/.gradle/caches/modules-2/files-2.1/org.apache.kafka/kafka-clients/1.0.0/98a12cfff3a5b7ca77e68aba7adccfd817b2d846/kafka-clients-1.0.0.jar to gs://temp-storage-for-perf-tests/loadtests/staging/kafka-clients-1.0.0-hkT7plonfEGDHIcExstJsUbIJ4-Dar2MLQTb_9wafEo.jar Aug 18, 2020 12:07:53 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/vendor/sdks-java-extensions-protobuf/build/libs/beam-vendor-sdks-java-extensions-protobuf-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-vendor-sdks-java-extensions-protobuf-2.25.0-SNAPSHOT-FxS9La90k-XG62kFGusnRnIz5ZCjMJkUI1G0KWtVClw.jar Aug 18, 2020 12:07:53 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/runners/core-construction-java/build/libs/beam-runners-core-construction-java-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-runners-core-construction-java-2.25.0-SNAPSHOT-teWKp1eeNsVhA9XssC9J3YQD6Ut9oG2u6zGSUKEmVW4.jar Aug 18, 2020 12:07:53 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/sdks/java/fn-execution/build/libs/beam-sdks-java-fn-execution-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-fn-execution-2.25.0-SNAPSHOT-6dPTGbsemyZ0icNE9Ary61i3mtNt1356U7u7RQOh98I.jar Aug 18, 2020 12:07:53 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/runners/direct-java/build/libs/beam-runners-direct-java-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-runners-direct-java-2.25.0-SNAPSHOT-wnBMFuw5lKGkjz5t3YHvRJM8bvrHpGAyBtNP_fGFLi4.jar Aug 18, 2020 12:07:53 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/model/pipeline/build/libs/beam-model-pipeline-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-model-pipeline-2.25.0-SNAPSHOT-LQffOuWxkMWIwWi1XoDmD6owdYwpuHhDFiAMX-UnQBk.jar Aug 18, 2020 12:07:53 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/sdks/java/io/kinesis/build/libs/beam-sdks-java-io-kinesis-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-io-kinesis-2.25.0-SNAPSHOT-jpX1iDuH10UAxBq-qCBnoOJug4ZqxVixDbrploRuXMk.jar Aug 18, 2020 12:07:53 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/sdks/java/io/synthetic/build/libs/beam-sdks-java-io-synthetic-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-io-synthetic-2.25.0-SNAPSHOT-tegIlXpc5YPcdT4bbszkBR9_HJotzelvdRkIfU7t3wg.jar Aug 18, 2020 12:07:53 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/sdks/java/core/build/libs/beam-sdks-java-core-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-core-2.25.0-SNAPSHOT--_BsGt99ZAIQjyywMjJdHlibtFVyrrcrdB3IJ2Ev0ZM.jar Aug 18, 2020 12:07:53 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/runners/google-cloud-dataflow-java/build/libs/beam-runners-google-cloud-dataflow-java-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-runners-google-cloud-dataflow-java-2.25.0-SNAPSHOT-QcaSdLorCKof4q2VVU4BA10-oOKNweRA2ZIyKwI6F88.jar Aug 18, 2020 12:07:53 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/sdks/java/extensions/protobuf/build/libs/beam-sdks-java-extensions-protobuf-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-extensions-protobuf-2.25.0-SNAPSHOT-Gd5Tu7xtVeA-StZ2ORJeVkf7jCeBfURlDN5FQxB5_pE.jar Aug 18, 2020 12:07:53 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-io-google-cloud-platform-2.25.0-SNAPSHOT-Eaj4PeSgmF3Xw5qS9OrvmwYuhekvdDobB-4cWoXxjEQ.jar Aug 18, 2020 12:07:53 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/runners/java-fn-execution/build/libs/beam-runners-java-fn-execution-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-runners-java-fn-execution-2.25.0-SNAPSHOT-Co1tbhYtEnd9WKIS181RM5k-cZgqkiTDAFHnmyjp5C0.jar Aug 18, 2020 12:07:53 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/sdks/java/expansion-service/build/libs/beam-sdks-java-expansion-service-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-expansion-service-2.25.0-SNAPSHOT-4C-CiF8dhcT614qwcPawQVdwr0VZ7jeNi3Ss0jBze78.jar Aug 18, 2020 12:07:53 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/runners/google-cloud-dataflow-java/****/legacy-****/build/libs/beam-runners-google-cloud-dataflow-java-legacy-****-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-runners-google-cloud-dataflow-java-legacy-****-2.25.0-SNAPSHOT-17jDKJCo1niyleelXT3gbXAQDKFn_WSRCfYkZmGDuKY.jar Aug 18, 2020 12:07:53 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /home/jenkins/.gradle/caches/modules-2/files-2.1/io.confluent/kafka-schema-registry-client/5.3.2/74bba24bc62eb9ade07e6a8ee0b71327f5132373/kafka-schema-registry-client-5.3.2.jar to gs://temp-storage-for-perf-tests/loadtests/staging/kafka-schema-registry-client-5.3.2-0mGQmXEF1E2hwRsR-O-mddgC_DwcNIU4Ba9SQcw2hKY.jar Aug 18, 2020 12:07:53 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /home/jenkins/.gradle/caches/modules-2/files-2.1/com.amazonaws/aws-java-sdk-kinesis/1.11.718/b1d58853fb7f79002c88409fde687a8b5acbc76d/aws-java-sdk-kinesis-1.11.718.jar to gs://temp-storage-for-perf-tests/loadtests/staging/aws-java-sdk-kinesis-1.11.718-USFZTKoh2eFkJqneoDnNRLfzHukvxfPGdphgPXziLGg.jar Aug 18, 2020 12:07:53 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /home/jenkins/.gradle/caches/modules-2/files-2.1/com.amazonaws/jmespath-java/1.11.718/42810912457b05b8acdfdfa8f52e52b7b4e419a4/jmespath-java-1.11.718.jar to gs://temp-storage-for-perf-tests/loadtests/staging/jmespath-java-1.11.718-QGFhpY1MtfXZpEVMihKnEBY-WHy_6zTiq_YSUjqHNOs.jar Aug 18, 2020 12:07:53 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /home/jenkins/.gradle/caches/modules-2/files-2.1/com.amazonaws/aws-java-sdk-cloudwatch/1.11.718/e1c29beaa15981f51fecb9dc5585df85b6e21b4f/aws-java-sdk-cloudwatch-1.11.718.jar to gs://temp-storage-for-perf-tests/loadtests/staging/aws-java-sdk-cloudwatch-1.11.718-l-hXHaoY80ODPGGrPvUcHEivzntTUucNx3wuro_kat0.jar Aug 18, 2020 12:07:53 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /home/jenkins/.gradle/caches/modules-2/files-2.1/com.amazonaws/aws-java-sdk-s3/1.11.718/ab90fe09d86080c678ee5dd0b0a4de657411f3c4/aws-java-sdk-s3-1.11.718.jar to gs://temp-storage-for-perf-tests/loadtests/staging/aws-java-sdk-s3-1.11.718-n7aKS9GlSvhE0B5pHMjn2fz4nngbW5J9cKxnwHYXZIs.jar Aug 18, 2020 12:07:53 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /home/jenkins/.gradle/caches/modules-2/files-2.1/com.amazonaws/amazon-kinesis-client/1.13.0/8a936932a1408a1b31129fee35b77f00fc7d65bd/amazon-kinesis-client-1.13.0.jar to gs://temp-storage-for-perf-tests/loadtests/staging/amazon-kinesis-client-1.13.0-qQFHBlQWtwYmFlxcWPYww_ZQMpWQLNTTHDwlHtNOWaM.jar Aug 18, 2020 12:07:54 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /home/jenkins/.gradle/caches/modules-2/files-2.1/io.confluent/common-utils/5.3.2/bbc8719cc79819b4af7443e887dc8b7d5633b6f5/common-utils-5.3.2.jar to gs://temp-storage-for-perf-tests/loadtests/staging/common-utils-5.3.2-4Rm-xdfQw0Yp9EH6ym0CZyPloNIGMAJ4b5DO_bByg9c.jar Aug 18, 2020 12:07:54 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/model/fn-execution/build/libs/beam-model-fn-execution-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-model-fn-execution-2.25.0-SNAPSHOT-nZx1jfQUsG60aVkNgRakUybD4rQIKfW026oEDdAsgS8.jar Aug 18, 2020 12:07:54 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /home/jenkins/.gradle/caches/modules-2/files-2.1/com.amazonaws/aws-java-sdk-kms/1.11.718/2932d8dfe6a374fdd3a3ee885ebb259073ea107e/aws-java-sdk-kms-1.11.718.jar to gs://temp-storage-for-perf-tests/loadtests/staging/aws-java-sdk-kms-1.11.718-XsLprXxJPp2ErLPPD5Pff4ol68yeox4sS2YOFC8eK-k.jar Aug 18, 2020 12:07:54 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /home/jenkins/.gradle/caches/modules-2/files-2.1/com.amazonaws/amazon-kinesis-producer/0.13.1/b728a3fd71b653ea9894619a953384917f6b9f71/amazon-kinesis-producer-0.13.1.jar to gs://temp-storage-for-perf-tests/loadtests/staging/amazon-kinesis-producer-0.13.1-bFcPYJXMRErKduQ4ounfteHsYsKjvRyr6qqx83EBM8k.jar Aug 18, 2020 12:07:54 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /home/jenkins/.gradle/caches/modules-2/files-2.1/com.amazonaws/aws-java-sdk-dynamodb/1.11.718/4985df59021becb6b291bf08f382b063b1c72c28/aws-java-sdk-dynamodb-1.11.718.jar to gs://temp-storage-for-perf-tests/loadtests/staging/aws-java-sdk-dynamodb-1.11.718-BIySvK_TPMMaZ-9GCTl0ZMbBoPbc_s_tSB572wx0oy8.jar Aug 18, 2020 12:07:54 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /home/jenkins/.gradle/caches/modules-2/files-2.1/com.101tec/zkclient/0.10/c54d4b5a5e89af75a80b6d5857400165ce5188d0/zkclient-0.10.jar to gs://temp-storage-for-perf-tests/loadtests/staging/zkclient-0.10-JumIuLuoOMck_YNQszHui1_8WcOpwHTfEVxMOmyEOHg.jar Aug 18, 2020 12:07:54 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /home/jenkins/.gradle/caches/modules-2/files-2.1/io.confluent/common-config/5.3.2/dd28d8981a590d136a3a5a8804dfc4f3355a6185/common-config-5.3.2.jar to gs://temp-storage-for-perf-tests/loadtests/staging/common-config-5.3.2-B_HQvj7POWJFHkM1OEwdiS2Y6zStZi7OXoiw-BJN7Nc.jar Aug 18, 2020 12:07:54 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /home/jenkins/.gradle/caches/modules-2/files-2.1/org.lz4/lz4-java/1.4/9bedb74f461a87ff2161bdf0778ad8ca6bad3e1c/lz4-java-1.4.jar to gs://temp-storage-for-perf-tests/loadtests/staging/lz4-java-1.4-ntUesjY0DKtYeA7X0gdB_4Eryzh1vrl0-nz53eonI1g.jar Aug 18, 2020 12:07:54 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/model/job-management/build/libs/beam-model-job-management-2.25.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-model-job-management-2.25.0-SNAPSHOT-esvZrgUp2M5e9NaqDbOuyt-XNflje1R2dDDV8E_j09c.jar Aug 18, 2020 12:07:54 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /home/jenkins/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-cbor/2.10.2/b86c6ef094fed25ebadd5ab9df8ffe0ba161434f/jackson-dataformat-cbor-2.10.2.jar to gs://temp-storage-for-perf-tests/loadtests/staging/jackson-dataformat-cbor-2.10.2-fB5excnYR4BBvQV4Ncjyhyz4OOe8w84bWUoX7pWdxTQ.jar Aug 18, 2020 12:07:54 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /home/jenkins/.gradle/caches/modules-2/files-2.1/com.amazonaws/aws-java-sdk-core/1.11.718/96c0394b894895ae8c7d0288fe29aba5cb2c01c7/aws-java-sdk-core-1.11.718.jar to gs://temp-storage-for-perf-tests/loadtests/staging/aws-java-sdk-core-1.11.718--xLTtGYrNOJQxP-n_qby8XR33XSXzPfhx9It1gKyYGc.jar Aug 18, 2020 12:07:54 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /home/jenkins/.gradle/caches/modules-2/files-2.1/org.springframework/spring-core/4.3.18.RELEASE/4acbce682c3dfe38181d57b7e0792e2cc21e4f77/spring-core-4.3.18.RELEASE.jar to gs://temp-storage-for-perf-tests/loadtests/staging/spring-core-4.3.18.RELEASE-qd-hKocv03STfOrJiihBIfzwVyBqE8q9UnPLPqutu2w.jar Aug 18, 2020 12:07:54 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /home/jenkins/.gradle/caches/modules-2/files-2.1/org.springframework/spring-expression/4.3.18.RELEASE/64f2270bfd9d615c8c3b9d94995867f39391ed30/spring-expression-4.3.18.RELEASE.jar to gs://temp-storage-for-perf-tests/loadtests/staging/spring-expression-4.3.18.RELEASE-aB3PS8pe5rtvJwR2BF3WrB6nO_rdEY2KpkFz7bel1_A.jar Aug 18, 2020 12:07:54 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /home/jenkins/.gradle/caches/modules-2/files-2.1/software.amazon.ion/ion-java/1.0.2/ee9dacea7726e495f8352b81c12c23834ffbc564/ion-java-1.0.2.jar to gs://temp-storage-for-perf-tests/loadtests/staging/ion-java-1.0.2-DRJ7IFofzgq8KjdXoEF0hlG8ZsFc9MBZusWDOyfUcaU.jar Aug 18, 2020 12:07:54 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /home/jenkins/.gradle/caches/modules-2/files-2.1/commons-io/commons-io/2.6/815893df5f31da2ece4040fe0a12fd44b577afaf/commons-io-2.6.jar to gs://temp-storage-for-perf-tests/loadtests/staging/commons-io-2.6--HfTBGYKwqFC84ZbrfyXHex-1zx0fH-NXS9ROcpzZRM.jar Aug 18, 2020 12:07:54 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /home/jenkins/.gradle/caches/modules-2/files-2.1/commons-lang/commons-lang/2.6/ce1edb914c94ebc388f086c6827e8bdeec71ac2/commons-lang-2.6.jar to gs://temp-storage-for-perf-tests/loadtests/staging/commons-lang-2.6-UPEbCfh3wpTVbyRGP0fSj5Kc9QRPZIZhwPDPuumi9Jw.jar Aug 18, 2020 12:07:57 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Staging files complete: 140 files cached, 42 files newly uploaded in 3 seconds Aug 18, 2020 12:07:57 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Read input/DataflowRunner.StreamingUnboundedRead.ReadWithIds as step s1 Aug 18, 2020 12:07:57 PM org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource split INFO: Split into 20 bundles of sizes: [org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5bdd5689, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2788d0fe, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@73ab3aac, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@54f66455, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@65ae095c, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@4c03a37, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2e140e59, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@49a71302, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5cd61783, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@59429fac, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@57aa341b, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@26e412ef, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@4ae263bf, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7103ab0, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@19ccca5, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@769d513, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5e4fa1da, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@588cd519, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3f92c349, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@769a58e5] Aug 18, 2020 12:07:57 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Read input/StripIds as step s2 Aug 18, 2020 12:07:57 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Collect start time metrics as step s3 Aug 18, 2020 12:07:57 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Total bytes monitor as step s4 Aug 18, 2020 12:07:57 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Window.Into()/Window.Assign as step s5 Aug 18, 2020 12:07:57 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Group by key (0) as step s6 Aug 18, 2020 12:07:57 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Ungroup and reiterate (0) as step s7 Aug 18, 2020 12:07:57 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Collect end time metrics (0) as step s8 Aug 18, 2020 12:07:57 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/loadtests/staging/ Aug 18, 2020 12:07:57 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <79495 bytes, hash 57e16c8dea0676ce98974fad1f3b1c303c26f1a85533ffae64fec2bfd0295ba0> to gs://temp-storage-for-perf-tests/loadtests/staging/pipeline-V-FsjeoGds6Yl0-tHzscMDwm8ahVM_-uZP7Cv9ApW6A.pb Aug 18, 2020 12:07:57 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Dataflow SDK version: 2.25.0-SNAPSHOT Aug 18, 2020 12:07:58 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-18_05_07_57-13954527050393233290?project=apache-beam-testing Aug 18, 2020 12:07:58 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Submitted job: 2020-08-18_05_07_57-13954527050393233290 Aug 18, 2020 12:07:58 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To cancel the job using the 'gcloud' tool, run: > gcloud dataflow jobs --project=apache-beam-testing cancel > --region=us-central1 2020-08-18_05_07_57-13954527050393233290 Aug 18, 2020 12:08:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process WARNING: 2020-08-18T12:08:00.490Z: The workflow name is not a valid Cloud Label. Labels applied to Cloud resources (such as GCE Instances) for monitoring will be labeled with this modified job name: load0tests0java0dataflow0streaming0gbk01-jenkins-081812075-1uys. For the best monitoring experience, please name your job with a valid Cloud Label. For details, see: https://cloud.google.com/compute/docs/labeling-resources#restrictions Aug 18, 2020 12:08:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2020-08-18T12:08:08.172Z: Workflow failed. Causes: The Compute Engine API has not fully initialized. Please try again in a few minutes., Unable to get machine type information for machine type n1-standard-4 in zone us-central1-f because of insufficient permissions. Please refer to https://cloud.google.com/dataflow/access-control#creating_jobs and make sure you have sufficient permissions. Aug 18, 2020 12:08:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-18T12:08:08.176Z: Cleaning up. Aug 18, 2020 12:08:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-18T12:08:08.198Z: Worker pool stopped. Aug 18, 2020 12:08:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-08-18T12:08:08.210Z: Stopping **** pool... Aug 18, 2020 12:08:11 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process WARNING: 2020-08-18T12:08:10.604Z: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete Aug 18, 2020 12:08:16 PM org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState INFO: Job 2020-08-18_05_07_57-13954527050393233290 failed with status FAILED. Aug 18, 2020 12:08:16 PM org.apache.beam.sdk.testutils.metrics.MetricsReader getCounterMetric SEVERE: Failed to get metric totalBytes.count, from namespace gbk Load test results for test (ID): 72765d53-fbef-4d41-bc8d-bc1afdcee16c and timestamp: 2020-08-18T12:07:50.307000000Z: Metric: Value: dataflow_runtime_sec 0.0 dataflow_total_bytes_count -1.0 Exception in thread "main" java.lang.RuntimeException: Invalid job state: FAILED. at org.apache.beam.sdk.loadtests.JobFailure.handleFailure(JobFailure.java:55) at org.apache.beam.sdk.loadtests.LoadTest.run(LoadTest.java:134) at org.apache.beam.sdk.loadtests.GroupByKeyLoadTest.run(GroupByKeyLoadTest.java:57) at org.apache.beam.sdk.loadtests.GroupByKeyLoadTest.main(GroupByKeyLoadTest.java:131) > Task :sdks:java:testing:load-tests:run FAILED FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':sdks:java:testing:load-tests:run'. > Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with > non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 1m 21s 73 actionable tasks: 48 executed, 25 from cache Publishing build scan... https://gradle.com/s/qndfp355ljis2 Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
