See <https://builds.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/301/display/redirect?page=changes>
Changes: [boyuanz] Update verify_release_build script to run python tests with dev version. [filiperegadas] Add BigQuery useAvroLogicalTypes option [filiperegadas] fixup! Add BigQuery useAvroLogicalTypes option [jvilcek] [BEAM-9360] Fix equivalence check for FieldType [rohde.samuel] ReverseTestStream Implementation [github] Update lostluck's info on the Go SDK roadmap [github] [BEAM-9432] Move expansion service into its own project. (#11035) [ehudm] [BEAM-3713] Remove nosetests from tox.ini [github] Merge pull request #11025: [BEAM-6428] Improve select performance with [github] Switch contact email to apache.org. [github] [BEAM-6374] Emit PCollection metrics from GoSDK (#10942) [github] Update container image tags used by Dataflow runner for Beam master [github] [BEAM-8328] Disable community metrics integration test in 'test' task [github] Add integration test for AnnotateVideoWithContext transform (#10986) ------------------------------------------ [...truncated 73.77 KB...] > Task :model:job-management:generateProto UP-TO-DATE > Task :model:fn-execution:generateProto UP-TO-DATE > Task :model:job-management:compileJava UP-TO-DATE > Task :model:job-management:classes UP-TO-DATE > Task :model:fn-execution:compileJava UP-TO-DATE > Task :model:job-management:shadowJar UP-TO-DATE > Task :model:fn-execution:classes UP-TO-DATE > Task :model:fn-execution:shadowJar UP-TO-DATE > Task :sdks:java:core:compileJava UP-TO-DATE > Task :sdks:java:core:classes UP-TO-DATE > Task :sdks:java:core:shadowJar UP-TO-DATE > Task :sdks:java:extensions:protobuf:extractIncludeProto UP-TO-DATE > Task :sdks:java:extensions:protobuf:generateProto NO-SOURCE > Task :sdks:java:io:synthetic:compileJava UP-TO-DATE > Task :sdks:java:io:synthetic:classes UP-TO-DATE > Task :sdks:java:io:synthetic:jar UP-TO-DATE > Task :runners:core-construction-java:compileJava UP-TO-DATE > Task :runners:core-construction-java:classes UP-TO-DATE > Task :sdks:java:io:kinesis:compileJava UP-TO-DATE > Task :sdks:java:io:kinesis:classes UP-TO-DATE > Task :runners:core-construction-java:jar UP-TO-DATE > Task :sdks:java:io:kinesis:jar UP-TO-DATE > Task :vendor:sdks-java-extensions-protobuf:compileJava UP-TO-DATE > Task :vendor:sdks-java-extensions-protobuf:classes UP-TO-DATE > Task :runners:local-java:compileJava UP-TO-DATE > Task :runners:local-java:classes UP-TO-DATE > Task :runners:local-java:jar UP-TO-DATE > Task :sdks:java:extensions:protobuf:compileJava UP-TO-DATE > Task :sdks:java:extensions:protobuf:classes UP-TO-DATE > Task :sdks:java:fn-execution:compileJava UP-TO-DATE > Task :sdks:java:fn-execution:classes UP-TO-DATE > Task :vendor:sdks-java-extensions-protobuf:shadowJar UP-TO-DATE > Task :sdks:java:extensions:protobuf:jar UP-TO-DATE > Task :sdks:java:fn-execution:jar UP-TO-DATE > Task :sdks:java:extensions:google-cloud-platform-core:compileJava UP-TO-DATE > Task :sdks:java:extensions:google-cloud-platform-core:classes UP-TO-DATE > Task :sdks:java:expansion-service:compileJava UP-TO-DATE > Task :sdks:java:extensions:google-cloud-platform-core:jar UP-TO-DATE > Task :sdks:java:expansion-service:classes UP-TO-DATE > Task :sdks:java:expansion-service:jar UP-TO-DATE > Task :sdks:java:io:kafka:compileJava UP-TO-DATE > Task :sdks:java:io:kafka:classes UP-TO-DATE > Task :sdks:java:io:kafka:jar UP-TO-DATE > Task :sdks:java:testing:test-utils:compileJava UP-TO-DATE > Task :sdks:java:testing:test-utils:classes UP-TO-DATE > Task :runners:core-java:compileJava UP-TO-DATE > Task :runners:core-java:classes UP-TO-DATE > Task :sdks:java:testing:test-utils:jar UP-TO-DATE > Task :sdks:java:io:google-cloud-platform:compileJava UP-TO-DATE > Task :sdks:java:io:google-cloud-platform:classes UP-TO-DATE > Task :runners:core-java:jar UP-TO-DATE > Task :sdks:java:io:google-cloud-platform:jar UP-TO-DATE > Task :sdks:java:harness:compileJava UP-TO-DATE > Task :sdks:java:harness:classes UP-TO-DATE > Task :sdks:java:harness:jar UP-TO-DATE > Task :sdks:java:harness:shadowJar UP-TO-DATE > Task :runners:google-cloud-dataflow-java:compileJava UP-TO-DATE > Task :runners:google-cloud-dataflow-java:classes UP-TO-DATE > Task :runners:google-cloud-dataflow-java:jar UP-TO-DATE > Task :runners:java-fn-execution:compileJava UP-TO-DATE > Task :runners:java-fn-execution:classes UP-TO-DATE > Task :runners:java-fn-execution:jar UP-TO-DATE > Task :runners:direct-java:compileJava UP-TO-DATE > Task :runners:direct-java:classes UP-TO-DATE > Task :runners:direct-java:shadowJar UP-TO-DATE > Task :sdks:java:testing:load-tests:compileJava UP-TO-DATE > Task :sdks:java:testing:load-tests:classes UP-TO-DATE > Task :sdks:java:testing:load-tests:jar UP-TO-DATE > Task :runners:google-cloud-dataflow-java:worker:legacy-worker:compileJava > UP-TO-DATE > Task :runners:google-cloud-dataflow-java:worker:legacy-worker:classes > UP-TO-DATE > Task :runners:google-cloud-dataflow-java:worker:legacy-worker:shadowJar > UP-TO-DATE > Task :sdks:java:testing:load-tests:run Mar 05, 2020 12:34:50 PM org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory create INFO: No stagingLocation provided, falling back to gcpTempLocation Mar 05, 2020 12:34:50 PM org.apache.beam.runners.dataflow.DataflowRunner fromOptions INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 164 files. Enable logging at DEBUG level to see which files will be staged. Mar 05, 2020 12:34:51 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services. Mar 05, 2020 12:34:51 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Uploading 165 files from PipelineOptions.filesToStage to staging location to prepare for execution. Mar 05, 2020 12:34:52 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements INFO: Staging files complete: 165 files cached, 0 files newly uploaded in 0 seconds Mar 05, 2020 12:34:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Read input/DataflowRunner.StreamingUnboundedRead.ReadWithIds as step s1 Mar 05, 2020 12:34:52 PM org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource split INFO: Split into 20 bundles of sizes: [org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@75b3673, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@1bcb79c2, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@d1a10ac, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@6fca2a8f, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@31fc71ab, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@71391b3f, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2cfbeac4, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@12db3386, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@4078695f, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@79a1728c, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@a7f0ab6, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@41f35f7c, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@42c2f48c, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3005db4a, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@425d5d46, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@198ef2ce, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@4cbd03e7, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@52fc5eb1, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7a639ec5, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@14151bc5] Mar 05, 2020 12:34:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Read input/StripIds as step s2 Mar 05, 2020 12:34:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Collect start time metrics as step s3 Mar 05, 2020 12:34:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Total bytes monitor as step s4 Mar 05, 2020 12:34:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Window.Into()/Window.Assign as step s5 Mar 05, 2020 12:34:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Group by key (0) as step s6 Mar 05, 2020 12:34:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Ungroup and reiterate (0) as step s7 Mar 05, 2020 12:34:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding Collect end time metrics (0) as step s8 Mar 05, 2020 12:34:52 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/loadtests/staging/ Mar 05, 2020 12:34:52 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Dataflow SDK version: 2.21.0-SNAPSHOT Mar 05, 2020 12:34:53 PM org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$DefaultGcpRegionFactory create WARNING: Region will default to us-central1. Future releases of Beam will require the user to set the region explicitly. https://cloud.google.com/compute/docs/regions-zones/regions-zones Mar 05, 2020 12:34:54 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-03-05_04_34_53-9062355887300871786?project=apache-beam-testing Mar 05, 2020 12:34:54 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Submitted job: 2020-03-05_04_34_53-9062355887300871786 Mar 05, 2020 12:34:54 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To cancel the job using the 'gcloud' tool, run: > gcloud dataflow jobs --project=apache-beam-testing cancel > --region=us-central1 2020-03-05_04_34_53-9062355887300871786 Mar 05, 2020 12:34:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process WARNING: 2020-03-05T12:34:56.596Z: The workflow name is not a valid Cloud Label. Labels applied to Cloud resources (such as GCE Instances) for monitoring will be labeled with this modified job name: load0tests0java0dataflow0streaming0gbk02-jenkins-030512345-hidt. For the best monitoring experience, please name your job with a valid Cloud Label. For details, see: https://cloud.google.com/compute/docs/labeling-resources#restrictions Mar 05, 2020 12:34:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T12:34:56.711Z: Checking permissions granted to controller Service Account. Mar 05, 2020 12:35:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T12:35:00.125Z: Worker configuration: n1-standard-4 in us-central1-f. Mar 05, 2020 12:35:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T12:35:00.810Z: Expanding CoGroupByKey operations into optimizable parts. Mar 05, 2020 12:35:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T12:35:00.821Z: Expanding SplittableProcessKeyed operations into optimizable parts. Mar 05, 2020 12:35:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T12:35:00.823Z: Expanding GroupByKey operations into streaming Read/Write steps Mar 05, 2020 12:35:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T12:35:00.827Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns Mar 05, 2020 12:35:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T12:35:00.857Z: Fusing adjacent ParDo, Read, Write, and Flatten operations Mar 05, 2020 12:35:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T12:35:00.859Z: Fusing consumer Read input/StripIds into Read input/DataflowRunner.StreamingUnboundedRead.ReadWithIds Mar 05, 2020 12:35:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T12:35:00.861Z: Fusing consumer Collect start time metrics into Read input/StripIds Mar 05, 2020 12:35:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T12:35:00.863Z: Fusing consumer Total bytes monitor into Collect start time metrics Mar 05, 2020 12:35:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T12:35:00.865Z: Fusing consumer Window.Into()/Window.Assign into Total bytes monitor Mar 05, 2020 12:35:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T12:35:00.867Z: Fusing consumer Group by key (0)/WriteStream into Window.Into()/Window.Assign Mar 05, 2020 12:35:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T12:35:00.870Z: Fusing consumer Group by key (0)/MergeBuckets into Group by key (0)/ReadStream Mar 05, 2020 12:35:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T12:35:00.872Z: Fusing consumer Ungroup and reiterate (0) into Group by key (0)/MergeBuckets Mar 05, 2020 12:35:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T12:35:00.874Z: Fusing consumer Collect end time metrics (0) into Ungroup and reiterate (0) Mar 05, 2020 12:35:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T12:35:01.085Z: Starting 5 workers... Mar 05, 2020 12:35:04 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T12:35:03.546Z: Executing operation Group by key (0)/ReadStream+Group by key (0)/MergeBuckets+Ungroup and reiterate (0)+Collect end time metrics (0) Mar 05, 2020 12:35:04 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T12:35:03.549Z: Executing operation Read input/DataflowRunner.StreamingUnboundedRead.ReadWithIds+Read input/StripIds+Collect start time metrics+Total bytes monitor+Window.Into()/Window.Assign+Group by key (0)/WriteStream Mar 05, 2020 12:35:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process WARNING: 2020-03-05T12:35:16.045Z: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete Mar 05, 2020 12:35:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T12:35:31.787Z: Checking permissions granted to controller Service Account. Mar 05, 2020 12:35:36 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T12:35:35.747Z: Worker configuration: n1-standard-4 in us-central1-f. Mar 05, 2020 12:35:44 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T12:35:43.403Z: Workers have started successfully. Mar 05, 2020 12:41:32 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T12:41:31.785Z: Checking permissions granted to controller Service Account. Mar 05, 2020 12:47:33 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T12:47:31.786Z: Checking permissions granted to controller Service Account. Mar 05, 2020 12:53:32 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T12:53:31.786Z: Checking permissions granted to controller Service Account. Mar 05, 2020 12:59:34 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T12:59:31.787Z: Checking permissions granted to controller Service Account. Mar 05, 2020 1:04:33 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T13:04:32.202Z: Finished operation Read input/DataflowRunner.StreamingUnboundedRead.ReadWithIds+Read input/StripIds+Collect start time metrics+Total bytes monitor+Window.Into()/Window.Assign+Group by key (0)/WriteStream Mar 05, 2020 1:04:33 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T13:04:32.202Z: Finished operation Group by key (0)/ReadStream+Group by key (0)/MergeBuckets+Ungroup and reiterate (0)+Collect end time metrics (0) Mar 05, 2020 1:04:33 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T13:04:32.364Z: Cleaning up. Mar 05, 2020 1:04:33 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T13:04:32.392Z: Worker pool stopped. Mar 05, 2020 1:04:33 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T13:04:32.403Z: Stopping worker pool... Mar 05, 2020 1:05:32 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T13:05:31.788Z: Checking permissions granted to controller Service Account. Mar 05, 2020 1:06:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2020-03-05T13:06:28.542Z: Worker pool stopped. Mar 05, 2020 1:06:35 PM org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState INFO: Job 2020-03-05_04_34_53-9062355887300871786 finished with status DONE. Load test results for test (ID): eb6cebfc-7209-4534-b2e6-dd7545e542c9 and timestamp: 2020-03-05T12:34:50.905000000Z: Metric: Value: runtime_sec 1676.42 total_bytes_count 3.369998E9 Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 31m 51s 72 actionable tasks: 1 executed, 71 up-to-date A build scan cannot be produced as an error occurred spooling the build data. Please report this problem via https://gradle.com/scans/help/plugin and include the following via copy/paste: ---------- Gradle version: 5.2.1 Plugin version: 2.3 java.lang.IllegalStateException: java.io.IOException: No space left on device at com.gradle.scan.a.e.b.flush(SourceFile:233) at com.gradle.scan.a.e.b.close(SourceFile:244) at com.gradle.scan.a.e.c.close(SourceFile:66) at com.gradle.scan.plugin.internal.n.a(SourceFile:78) at com.gradle.scan.plugin.internal.e.c.c.b(SourceFile:125) at com.gradle.scan.plugin.internal.e.c.c.a(SourceFile:117) at com.gradle.scan.plugin.internal.o.a$a.a(SourceFile:30) at com.gradle.scan.plugin.internal.o.a$a.a(SourceFile:19) at com.gradle.scan.plugin.internal.o.a.c(SourceFile:60) Caused by: java.io.IOException: No space left on device at com.gradle.scan.a.e.b.flush(SourceFile:231) ... 8 more ---------- FAILURE: Build failed with an exception. * What went wrong: Failed to notify root build lifecycle listener. > Could not update > /home/jenkins/.gradle/caches/5.2.1/file-changes/last-build.bin > Could not update > <https://builds.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/ws/src/.gradle/5.2.1/fileChanges/last-build.bin> * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 31m 51s FAILURE: Build failed with an exception. * What went wrong: Could not write cache value to '/home/jenkins/.gradle/daemon/5.2.1/registry.bin'. * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
