See <https://ci-beam.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/1574/display/redirect?page=changes>
Changes: [samuelw] [BEAM-11143] Ensure that AfterWatermarkStateMachine clears late trigger [noreply] [BEAM-11154] Check coder proto to avoid registering same coder under [noreply] [BEAM-9444] Use GCP BOM to set package versions (#13075) [noreply] [BEAM-11052] Memoize to_pcollection (#13066) [noreply] [BEAM-9547] Auto-populate any unimplemented methods/properties with ------------------------------------------ [...truncated 228.02 KB...] :sdks:java:io:bigquery-io-perf-tests:integrationTest (Thread[Execution **** for ':' Thread 9,5,main]) started. Gradle Test Executor 6 started executing tests. > Task :sdks:java:io:bigquery-io-perf-tests:integrationTest Custom actions are attached to task ':sdks:java:io:bigquery-io-perf-tests:integrationTest'. Build cache key for task ':sdks:java:io:bigquery-io-perf-tests:integrationTest' is b095a5da7de8fad57f122aca99218704 Task ':sdks:java:io:bigquery-io-perf-tests:integrationTest' is not up-to-date because: Task.upToDateWhen is false. Starting process 'Gradle Test Executor 6'. Working directory: <https://ci-beam.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/ws/src/sdks/java/io/bigquery-io-perf-tests> Command: /usr/lib/jvm/java-8-openjdk-amd64/bin/java -DbeamTestPipelineOptions=["--project=apache-beam-testing","--tempLocation=gs://temp-storage-for-perf-tests/loadtests","--tempRoot=gs://temp-storage-for-perf-tests/loadtests","--writeMethod=STREAMING_INSERTS","--writeFormat=JSON","--testBigQueryDataset=beam_performance","--testBigQueryTable=bqio_write_10GB_java_stream_1103000915","--metricsBigQueryDataset=beam_performance","--metricsBigQueryTable=bqio_10GB_results_java_stream","--influxMeasurement=bqio_10GB_results_java_stream","--sourceOptions={\"numRecords\":\"10485760\",\"keySizeBytes\":\"1\",\"valueSizeBytes\":\"1024\"}","--runner=DataflowRunner","--maxNumWorkers=5","--numWorkers=5","--autoscalingAlgorithm=NONE","--influxDatabase=beam_test_metrics","--influxHost=http://10.128.0.96:8086","--****HarnessContainerImage=","--dataflowWorkerJar=<https://ci-beam.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/ws/src/runners/google-cloud-dataflow-java/****/legacy-****/build/libs/beam-runners-google-cloud-dataflow-java-legacy-****-2.26.0-SNAPSHOT.jar","--region=us-central1";]> -Djava.security.manager=****.org.gradle.process.internal.****.child.BootstrapSecurityManager -Dorg.gradle.native=false -Xmx2g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant -ea -cp /home/jenkins/.gradle/caches/6.6.1/****Main/gradle-****.jar ****.org.gradle.process.internal.****.GradleWorkerMain 'Gradle Test Executor 6' Successfully started process 'Gradle Test Executor 6' org.apache.beam.sdk.bigqueryioperftests.BigQueryIOIT STANDARD_ERROR SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:<https://ci-beam.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/ws/src/runners/google-cloud-dataflow-java/****/legacy-****/build/libs/beam-runners-google-cloud-dataflow-java-legacy-****-2.26.0-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]> SLF4J: Found binding in [jar:file:/home/jenkins/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-jdk14/1.7.30/d35953dd2fe54ebe39fdf18cfd82fe6eb35b25ed/slf4j-jdk14-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.JDK14LoggerFactory] org.apache.beam.sdk.bigqueryioperftests.BigQueryIOIT > testWriteThenRead STANDARD_ERROR Nov 03, 2020 12:34:23 AM org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory create INFO: No stagingLocation provided, falling back to gcpTempLocation Nov 03, 2020 12:34:23 AM org.apache.beam.runners.dataflow.DataflowRunner fromOptions INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 202 files. Enable logging at DEBUG level to see which files will be staged. Nov 03, 2020 12:34:25 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services. Nov 03, 2020 12:34:26 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Uploading 203 files from PipelineOptions.filesToStage to staging location to prepare for execution. Nov 03, 2020 12:34:26 AM org.apache.beam.runners.dataflow.util.PackageUtil$PackageAttributes forFileToStage INFO: Staging custom dataflow-****.jar as beam-runners-google-cloud-dataflow-java-legacy-****-2.26.0-SNAPSHOT-_AwTHxGrrwGBpnwgZtkwwGRNI3M9XyFdxkzDZ-0ijk4.jar Nov 03, 2020 12:34:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/ws/src/sdks/java/io/common/build/libs/beam-sdks-java-io-common-2.26.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-io-common-2.26.0-SNAPSHOT-uZl9xMwpn8pxLvKdtlVetSqXkZW3CK7ZbF6BH15PQF8.jar Nov 03, 2020 12:34:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/ws/src/sdks/java/extensions/google-cloud-platform-core/build/libs/beam-sdks-java-extensions-google-cloud-platform-core-2.26.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-extensions-google-cloud-platform-core-2.26.0-SNAPSHOT-J3IfzoF18xdO3Q6mqsRtLPlgjWWVAsseJPcGswXfsMM.jar Nov 03, 2020 12:34:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/ws/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.26.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-io-google-cloud-platform-2.26.0-SNAPSHOT-xMfRKBtB4x6wfDW5H1uLIp9AiycD2lYDiIMYfew8LDE.jar Nov 03, 2020 12:34:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/ws/src/sdks/java/expansion-service/build/libs/beam-sdks-java-expansion-service-2.26.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-expansion-service-2.26.0-SNAPSHOT-GnbDsgHHLoD7y4q8MvdodqoRxve8gKc5hK66ETaJloo.jar Nov 03, 2020 12:34:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/ws/src/runners/google-cloud-dataflow-java/build/libs/beam-runners-google-cloud-dataflow-java-2.26.0-SNAPSHOT-tests.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-runners-google-cloud-dataflow-java-2.26.0-SNAPSHOT-tests-WTo3xiLsgPYE81Ro6smJmF9qLxs10mpgM7y2xWrBwek.jar Nov 03, 2020 12:34:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/ws/src/sdks/java/extensions/google-cloud-platform-core/build/libs/beam-sdks-java-extensions-google-cloud-platform-core-2.26.0-SNAPSHOT-tests.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-extensions-google-cloud-platform-core-2.26.0-SNAPSHOT-tests-OY-FsqjnbzM8OCUIORh8xNgnh0PHWfx8_GI2RgyDy64.jar Nov 03, 2020 12:34:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/ws/src/sdks/java/core/build/libs/beam-sdks-java-core-2.26.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-core-2.26.0-SNAPSHOT-qnS67NFaM0uI58lUcJMCJhSIijhkyLmnFqlM3jw4yIk.jar Nov 03, 2020 12:34:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/ws/src/runners/java-fn-execution/build/libs/beam-runners-java-fn-execution-2.26.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-runners-java-fn-execution-2.26.0-SNAPSHOT-1E4xU1Za2n4_vR_dDO-fzfB_t1x-uNbaLxJTrfCWCsw.jar Nov 03, 2020 12:34:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading /tmp/test4140640459516818805.zip to gs://temp-storage-for-perf-tests/loadtests/staging/test-pHzh9XRxR-e5K8euPj8_20zegqMi7LtOtlOQmxJDnck.jar Nov 03, 2020 12:34:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/ws/src/sdks/java/io/synthetic/build/libs/beam-sdks-java-io-synthetic-2.26.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-io-synthetic-2.26.0-SNAPSHOT-VYszCHZwWBgYjic64gv3bOhn-sM3kgUQz1XD-IYTOP0.jar Nov 03, 2020 12:34:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/ws/src/runners/google-cloud-dataflow-java/build/libs/beam-runners-google-cloud-dataflow-java-2.26.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-runners-google-cloud-dataflow-java-2.26.0-SNAPSHOT-FXyYBW75FikTJ-mWtsqFFqtBSNMu6h_Hc5Z0FQhlxaw.jar Nov 03, 2020 12:34:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/ws/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.26.0-SNAPSHOT-tests.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-io-google-cloud-platform-2.26.0-SNAPSHOT-tests-3f7Ot5ViE1CrnKJitjsnacM3lZdskbBOFwHNn3JKtz4.jar Nov 03, 2020 12:34:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/ws/src/sdks/java/testing/test-utils/build/libs/beam-sdks-java-test-utils-2.26.0-SNAPSHOT-tests.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-test-utils-2.26.0-SNAPSHOT-tests-nYItctYNRt-L73CrhmSJtdU16PG_ia7rG5QXWVSpVyY.jar Nov 03, 2020 12:34:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/ws/src/sdks/java/extensions/protobuf/build/libs/beam-sdks-java-extensions-protobuf-2.26.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-extensions-protobuf-2.26.0-SNAPSHOT-v4iMfXyQUVf1t9UKEt2QHRNJFLTMbPqXfb_e7ZU-YwI.jar Nov 03, 2020 12:34:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/ws/src/sdks/java/io/common/build/libs/beam-sdks-java-io-common-2.26.0-SNAPSHOT-tests.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-io-common-2.26.0-SNAPSHOT-tests-10jjBXmZH3sJ52CB4Uvb7_WzWM3xTD4YHMuCgT71RZE.jar Nov 03, 2020 12:34:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/ws/src/sdks/java/core/build/libs/beam-sdks-java-core-2.26.0-SNAPSHOT-tests.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-core-2.26.0-SNAPSHOT-tests-4opLhqj2Kj3yOd5rp_qsahGMYlN-3XPsAF1Va-pWwxM.jar Nov 03, 2020 12:34:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/ws/src/sdks/java/testing/test-utils/build/libs/beam-sdks-java-test-utils-2.26.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-test-utils-2.26.0-SNAPSHOT-cBqJrYG39-RLwUevRDulZgb_0NUGyXLerpjLI7QCcmU.jar Nov 03, 2020 12:34:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/ws/src/runners/direct-java/build/libs/beam-runners-direct-java-2.26.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-runners-direct-java-2.26.0-SNAPSHOT-JyymbSvcxzTS0eGp-eYvzfY5nxPgboK-ztE_wr2LdZk.jar Nov 03, 2020 12:34:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/ws/src/runners/google-cloud-dataflow-java/****/legacy-****/build/libs/beam-runners-google-cloud-dataflow-java-legacy-****-2.26.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-runners-google-cloud-dataflow-java-legacy-****-2.26.0-SNAPSHOT-_AwTHxGrrwGBpnwgZtkwwGRNI3M9XyFdxkzDZ-0ijk4.jar Nov 03, 2020 12:34:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/ws/src/sdks/java/fn-execution/build/libs/beam-sdks-java-fn-execution-2.26.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-sdks-java-fn-execution-2.26.0-SNAPSHOT-yPX9R7aQlOufcUxsRn1KnapQgsVA8CxWZH0GjNkXIrw.jar Nov 03, 2020 12:34:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/ws/src/runners/core-construction-java/build/libs/beam-runners-core-construction-java-2.26.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-runners-core-construction-java-2.26.0-SNAPSHOT-8C_tQa2UjHFnRfvQdN-YWQIdCbdV2ziU218QnTt0Oko.jar Nov 03, 2020 12:34:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/ws/src/vendor/sdks-java-extensions-protobuf/build/libs/beam-vendor-sdks-java-extensions-protobuf-2.26.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-vendor-sdks-java-extensions-protobuf-2.26.0-SNAPSHOT-eiVDZtO4H-2cFCsStAa6bYaCvcCmd9JiNVdJBoAhStU.jar Nov 03, 2020 12:34:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/ws/src/model/job-management/build/libs/beam-model-job-management-2.26.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-model-job-management-2.26.0-SNAPSHOT-hMTImCbSut85pAYmL2FzQMX7E1nl0Eyy93nz3VJ78qA.jar Nov 03, 2020 12:34:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/ws/src/model/fn-execution/build/libs/beam-model-fn-execution-2.26.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-model-fn-execution-2.26.0-SNAPSHOT-8qW5xqW-J-TR4y-F-IHozxEdudpsmjBzNhlZo1tPnzs.jar Nov 03, 2020 12:34:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <https://ci-beam.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/ws/src/model/pipeline/build/libs/beam-model-pipeline-2.26.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/loadtests/staging/beam-model-pipeline-2.26.0-SNAPSHOT-eaLxhlZpSB08k_zWuD_itcYr8BSoZyBlkDi3akSgnJ8.jar Nov 03, 2020 12:34:27 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Staging files complete: 178 files cached, 25 files newly uploaded in 1 seconds Nov 03, 2020 12:34:28 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Read from source as step s1 Nov 03, 2020 12:34:28 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Gather time as step s2 Nov 03, 2020 12:34:28 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Map records as step s3 Nov 03, 2020 12:34:28 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Write to BQ/PrepareWrite/ParDo(Anonymous) as step s4 Nov 03, 2020 12:34:28 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Write to BQ/StreamingInserts/CreateTables/ParDo(CreateTables) as step s5 Nov 03, 2020 12:34:28 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Write to BQ/StreamingInserts/StreamingWriteTables/ShardTableWrites as step s6 Nov 03, 2020 12:34:28 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Write to BQ/StreamingInserts/StreamingWriteTables/TagWithUniqueIds as step s7 Nov 03, 2020 12:34:28 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Write to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign as step s8 Nov 03, 2020 12:34:28 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Write to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey as step s9 Nov 03, 2020 12:34:28 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Write to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/ExpandIterable as step s10 Nov 03, 2020 12:34:28 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Write to BQ/StreamingInserts/StreamingWriteTables/GlobalWindow/Window.Assign as step s11 Nov 03, 2020 12:34:28 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Write to BQ/StreamingInserts/StreamingWriteTables/StreamingWrite as step s12 Nov 03, 2020 12:34:28 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/loadtests/staging/ Nov 03, 2020 12:34:28 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <106496 bytes, hash b5bc2e81326f22a6ee0de772b1c5594dd5cc822a97057aa33147ab99af67f871> to gs://temp-storage-for-perf-tests/loadtests/staging/pipeline-tbwugTJvIqbuDedyscVZTdXMgiqXBXqjMUerma9n-HE.pb Nov 03, 2020 12:34:28 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Dataflow SDK version: 2.26.0-SNAPSHOT Nov 03, 2020 12:34:30 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-11-02_16_34_28-14520997460096287554?project=apache-beam-testing Nov 03, 2020 12:34:30 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Submitted job: 2020-11-02_16_34_28-14520997460096287554 Nov 03, 2020 12:34:30 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To cancel the job using the 'gcloud' tool, run: > gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2020-11-02_16_34_28-14520997460096287554 Nov 03, 2020 12:34:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process WARNING: 2020-11-03T00:34:28.671Z: The requested max number of ****s (5) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE). Nov 03, 2020 12:34:37 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:34:35.700Z: Worker configuration: n1-standard-1 in us-central1-f. Nov 03, 2020 12:34:37 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:34:36.508Z: Expanding CoGroupByKey operations into optimizable parts. Nov 03, 2020 12:34:37 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:34:36.578Z: Expanding GroupByKey operations into optimizable parts. Nov 03, 2020 12:34:37 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:34:36.627Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns Nov 03, 2020 12:34:37 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:34:36.802Z: Fusing adjacent ParDo, Read, Write, and Flatten operations Nov 03, 2020 12:34:37 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:34:36.841Z: Fusing consumer Gather time into Read from source Nov 03, 2020 12:34:37 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:34:36.895Z: Fusing consumer Map records into Gather time Nov 03, 2020 12:34:37 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:34:36.957Z: Fusing consumer Write to BQ/PrepareWrite/ParDo(Anonymous) into Map records Nov 03, 2020 12:34:37 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:34:36.997Z: Fusing consumer Write to BQ/StreamingInserts/CreateTables/ParDo(CreateTables) into Write to BQ/PrepareWrite/ParDo(Anonymous) Nov 03, 2020 12:34:37 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:34:37.043Z: Fusing consumer Write to BQ/StreamingInserts/StreamingWriteTables/ShardTableWrites into Write to BQ/StreamingInserts/CreateTables/ParDo(CreateTables) Nov 03, 2020 12:34:37 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:34:37.078Z: Fusing consumer Write to BQ/StreamingInserts/StreamingWriteTables/TagWithUniqueIds into Write to BQ/StreamingInserts/StreamingWriteTables/ShardTableWrites Nov 03, 2020 12:34:37 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:34:37.107Z: Fusing consumer Write to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign into Write to BQ/StreamingInserts/StreamingWriteTables/TagWithUniqueIds Nov 03, 2020 12:34:37 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:34:37.147Z: Fusing consumer Write to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/Reify into Write to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign Nov 03, 2020 12:34:37 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:34:37.183Z: Fusing consumer Write to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/Write into Write to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/Reify Nov 03, 2020 12:34:37 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:34:37.218Z: Fusing consumer Write to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/GroupByWindow into Write to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/Read Nov 03, 2020 12:34:37 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:34:37.254Z: Fusing consumer Write to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/ExpandIterable into Write to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/GroupByWindow Nov 03, 2020 12:34:37 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:34:37.287Z: Fusing consumer Write to BQ/StreamingInserts/StreamingWriteTables/GlobalWindow/Window.Assign into Write to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/ExpandIterable Nov 03, 2020 12:34:37 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:34:37.328Z: Fusing consumer Write to BQ/StreamingInserts/StreamingWriteTables/StreamingWrite into Write to BQ/StreamingInserts/StreamingWriteTables/GlobalWindow/Window.Assign Nov 03, 2020 12:34:37 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:34:37.727Z: Executing operation Write to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/Create Nov 03, 2020 12:34:37 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:34:37.835Z: Starting 5 ****s in us-central1-f... Nov 03, 2020 12:34:37 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:34:37.888Z: Finished operation Write to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/Create Nov 03, 2020 12:34:40 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:34:38.050Z: Executing operation Read from source+Gather time+Map records+Write to BQ/PrepareWrite/ParDo(Anonymous)+Write to BQ/StreamingInserts/CreateTables/ParDo(CreateTables)+Write to BQ/StreamingInserts/StreamingWriteTables/ShardTableWrites+Write to BQ/StreamingInserts/StreamingWriteTables/TagWithUniqueIds+Write to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign+Write to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/Reify+Write to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/Write Nov 03, 2020 12:34:49 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:34:48.508Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete Nov 03, 2020 12:35:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:35:03.252Z: Autoscaling: Raised the number of ****s to 1 based on the rate of progress in the currently running stage(s). Nov 03, 2020 12:35:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:35:03.287Z: Resized **** pool to 1, though goal was 5. This could be a quota issue. Nov 03, 2020 12:35:10 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:35:08.514Z: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s). Nov 03, 2020 12:35:25 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:35:25.123Z: Workers have started successfully. Nov 03, 2020 12:35:25 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:35:25.162Z: Workers have started successfully. Nov 03, 2020 12:37:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:37:33.663Z: Finished operation Read from source+Gather time+Map records+Write to BQ/PrepareWrite/ParDo(Anonymous)+Write to BQ/StreamingInserts/CreateTables/ParDo(CreateTables)+Write to BQ/StreamingInserts/StreamingWriteTables/ShardTableWrites+Write to BQ/StreamingInserts/StreamingWriteTables/TagWithUniqueIds+Write to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign+Write to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/Reify+Write to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/Write Nov 03, 2020 12:37:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:37:33.756Z: Executing operation Write to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/Close Nov 03, 2020 12:37:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:37:33.811Z: Finished operation Write to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/Close Nov 03, 2020 12:37:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-11-03T00:37:33.874Z: Executing operation Write to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/Read+Write to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/GroupByWindow+Write to BQ/StreamingInserts/StreamingWriteTables/Reshuffle/ExpandIterable+Write to BQ/StreamingInserts/StreamingWriteTables/GlobalWindow/Window.Assign+Write to BQ/StreamingInserts/StreamingWriteTables/StreamingWrite org.apache.beam.sdk.bigqueryioperftests.BigQueryIOIT > testWriteThenRead SKIPPED > Task :sdks:java:io:bigquery-io-perf-tests:integrationTest FAILED :sdks:java:io:bigquery-io-perf-tests:integrationTest (Thread[Execution **** for ':' Thread 9,5,main]) completed. Took 14 mins 7.398 secs. FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':sdks:java:io:bigquery-io-perf-tests:integrationTest'. > Process 'Gradle Test Executor 6' finished with non-zero exit value 143 This problem might be caused by incorrect test process configuration. Please refer to the test execution section in the User Manual at https://docs.gradle.org/6.6.1/userguide/java_testing.html#sec:test_execution * Try: Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.6.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 15m 8s 84 actionable tasks: 53 executed, 31 from cache The message received from the daemon indicates that the daemon has disappeared. Build request sent: Build{id=438c5c57-2824-4e64-856b-3b0cc021e6e8, currentDir=<https://ci-beam.apache.org/job/beam_BiqQueryIO_Streaming_Performance_Test_Java/ws/src}> Attempting to read last messages from the daemon log... Daemon pid: 9323 log file: /home/jenkins/.gradle/daemon/6.6.1/daemon-9323.out.log ----- Last 20 lines from daemon log file - daemon-9323.out.log ----- FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':sdks:java:io:bigquery-io-perf-tests:integrationTest'. > Process 'Gradle Test Executor 6' finished with non-zero exit value 143 This problem might be caused by incorrect test process configuration. Please refer to the test execution section in the User Manual at https://docs.gradle.org/6.6.1/userguide/java_testing.html#sec:test_execution * Try: Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.6.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 15m 8s 84 actionable tasks: 53 executed, 31 from cache Daemon vm is shutting down... The daemon has exited normally or was terminated in response to a user interrupt. ----- End of the daemon log ----- FAILURE: Build failed with an exception. * What went wrong: Gradle build daemon disappeared unexpectedly (it may have been killed or may have crashed) * Try: Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
