See <https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/58/display/redirect?page=changes>
Changes: [kedin] [SQL] Infer Row type from input elements ------------------------------------------ [...truncated 18.65 MB...] Apr 12, 2018 11:14:41 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Extract keys as step s15 Apr 12, 2018 11:14:41 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey as step s16 Apr 12, 2018 11:14:41 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/Combine.GroupedValues as step s17 Apr 12, 2018 11:14:41 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow) as step s18 Apr 12, 2018 11:14:41 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s19 Apr 12, 2018 11:14:41 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/ParMultiDo(ToIsmRecordForMapLike) as step s20 Apr 12, 2018 11:14:41 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForSize as step s21 Apr 12, 2018 11:14:41 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/ParDo(ToIsmMetadataRecordForSize) as step s22 Apr 12, 2018 11:14:41 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForKeys as step s23 Apr 12, 2018 11:14:41 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/ParDo(ToIsmMetadataRecordForKey) as step s24 Apr 12, 2018 11:14:41 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/Flatten.PCollections as step s25 Apr 12, 2018 11:14:41 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/CreateDataflowView as step s26 Apr 12, 2018 11:14:41 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Partition input as step s27 Apr 12, 2018 11:14:41 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition as step s28 Apr 12, 2018 11:14:41 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Batch mutations together as step s29 Apr 12, 2018 11:14:41 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Write mutations to Spanner as step s30 Apr 12, 2018 11:14:41 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Staging pipeline description to gs://temp-storage-for-end-to-end-tests/spannerwriteit0testwrite-jenkins-0412231426-3b70b920/output/results/staging/ Apr 12, 2018 11:14:41 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <80355 bytes, hash maM2DFrrTwC0SWeilGLwUw> to gs://temp-storage-for-end-to-end-tests/spannerwriteit0testwrite-jenkins-0412231426-3b70b920/output/results/staging/pipeline-maM2DFrrTwC0SWeilGLwUw.pb org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testWrite STANDARD_OUT Dataflow SDK version: 2.5.0-SNAPSHOT org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testWrite STANDARD_ERROR Apr 12, 2018 11:14:42 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-04-12_16_14_41-11638957145981735383?project=apache-beam-testing org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testWrite STANDARD_OUT Submitted job: 2018-04-12_16_14_41-11638957145981735383 org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testWrite STANDARD_ERROR Apr 12, 2018 11:14:42 PM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To cancel the job using the 'gcloud' tool, run: > gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-04-12_16_14_41-11638957145981735383 Apr 12, 2018 11:14:42 PM org.apache.beam.runners.dataflow.TestDataflowRunner run INFO: Running Dataflow job 2018-04-12_16_14_41-11638957145981735383 with 0 expected assertions. Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:41.717Z: Autoscaling is enabled for job 2018-04-12_16_14_41-11638957145981735383. The number of workers will be between 1 and 1000. Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:41.749Z: Autoscaling was automatically enabled for job 2018-04-12_16_14_41-11638957145981735383. Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:44.559Z: Checking required Cloud APIs are enabled. Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:44.671Z: Checking permissions granted to controller Service Account. Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:47.928Z: Worker configuration: n1-standard-1 in us-central1-f. Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:48.393Z: Expanding CoGroupByKey operations into optimizable parts. Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:48.600Z: Expanding GroupByKey operations into optimizable parts. Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:48.638Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:48.806Z: Fusing adjacent ParDo, Read, Write, and Flatten operations Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:48.847Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write mutations to Cloud Spanner/Create.Values/Read(CreateSource) Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:48.878Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial into SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:48.904Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract into SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:48.940Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues into SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:48.971Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:48.993Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write into SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:49.021Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) into SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:49.049Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:49.084Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify into SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:49.126Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) into SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:49.154Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map into SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:49.180Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Extract keys into SpannerIO.Write/Write mutations to Cloud Spanner/Serialize mutations Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:49.210Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Serialize mutations into SpannerIO.Write/To mutation group Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:49.245Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/ParDo(ToIsmMetadataRecordForKey) into SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForKeys/Read Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:49.269Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Write mutations to Spanner into SpannerIO.Write/Write mutations to Cloud Spanner/Batch mutations together Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:49.302Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/ParMultiDo(ToIsmRecordForMapLike) into SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:49.338Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForSize/Write into SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/ParMultiDo(ToIsmRecordForMapLike) Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:49.374Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow) into SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/Combine.GroupedValues/Extract Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:49.405Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/Reify into SpannerIO.Write/Write mutations to Cloud Spanner/Partition input Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:49.428Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/Combine.GroupedValues into SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Read Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:49.461Z: Fusing consumer SpannerIO.Write/To mutation group into ParDo(GenerateMutations) Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:49.501Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/ParDo(ToIsmMetadataRecordForSize) into SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForSize/Read Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:49.523Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/GroupByWindow into SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/Read Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:49.549Z: Fusing consumer ParDo(GenerateMutations) into GenerateSequence/Read(BoundedCountingSource) Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:49.575Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForKeys/Write into SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/ParMultiDo(ToIsmRecordForMapLike) Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:49.596Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Batch mutations together into SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/GroupByWindow Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:49.633Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/Combine.GroupedValues/Extract into SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/Combine.GroupedValues Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:49.694Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Reify into SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey+SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/Combine.GroupedValues/Partial Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:49.734Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey+SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/Combine.GroupedValues/Partial into SpannerIO.Write/Write mutations to Cloud Spanner/Extract keys Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:49.768Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow) Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:49.803Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Write into SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Reify Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:49.844Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/Write into SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/Reify Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:50.224Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:50.267Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Create Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:50.307Z: Starting 1 workers in us-central1-f... Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:50.307Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:50.338Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForSize/Create Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:50.373Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForKeys/Create Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:50.410Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/Create Apr 12, 2018 11:14:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:14:50.448Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Create Apr 12, 2018 11:15:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:15:00.231Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s). Apr 12, 2018 11:15:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:15:00.677Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Create.Values/Read(CreateSource)+SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write Apr 12, 2018 11:15:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:15:05.990Z: Autoscaling: Resizing worker pool from 1 to 7. Apr 12, 2018 11:15:13 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:15:12.687Z: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running step(s). Apr 12, 2018 11:15:13 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:15:12.715Z: Resized worker pool to 1, though goal was 7. This could be a quota issue. Apr 12, 2018 11:15:47 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:15:46.757Z: Workers have started successfully. Apr 12, 2018 11:15:55 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:15:54.846Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s). Apr 12, 2018 11:15:55 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:15:54.896Z: Resized worker pool to 4, though goal was 7. This could be a quota issue. Apr 12, 2018 11:16:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:16:00.221Z: Autoscaling: Raised the number of workers to 7 based on the rate of progress in the currently running step(s). Apr 12, 2018 11:16:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:16:26.381Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Close Apr 12, 2018 11:16:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:16:26.447Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write Apr 12, 2018 11:16:35 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:16:32.436Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close Apr 12, 2018 11:16:35 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:16:32.504Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) Apr 12, 2018 11:16:43 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:16:42.645Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/CreateDataflowView Apr 12, 2018 11:16:43 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:16:42.834Z: Executing operation GenerateSequence/Read(BoundedCountingSource)+ParDo(GenerateMutations)+SpannerIO.Write/To mutation group+SpannerIO.Write/Write mutations to Cloud Spanner/Serialize mutations+SpannerIO.Write/Write mutations to Cloud Spanner/Extract keys+SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey+SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/Combine.GroupedValues/Partial+SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Reify+SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Write Apr 12, 2018 11:16:55 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:16:54.995Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Close Apr 12, 2018 11:16:55 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:16:55.066Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Read+SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/Combine.GroupedValues+SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/Combine.GroupedValues/Extract+SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow)+SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write Apr 12, 2018 11:17:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:17:01.239Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close Apr 12, 2018 11:17:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:17:01.307Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/ParMultiDo(ToIsmRecordForMapLike)+SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForSize/Write+SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForKeys/Write Apr 12, 2018 11:17:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:17:05.078Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForSize/Close Apr 12, 2018 11:17:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:17:05.114Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForKeys/Close Apr 12, 2018 11:17:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:17:05.147Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForSize/Read+SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/ParDo(ToIsmMetadataRecordForSize) Apr 12, 2018 11:17:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:17:05.179Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForKeys/Read+SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/ParDo(ToIsmMetadataRecordForKey) Apr 12, 2018 11:17:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:17:08.759Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/Flatten.PCollections Apr 12, 2018 11:17:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:17:09.018Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/CreateDataflowView Apr 12, 2018 11:17:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:17:09.235Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Partition input+SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/Reify+SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/Write Apr 12, 2018 11:17:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:17:12.626Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/Close Apr 12, 2018 11:17:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:17:12.738Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/Read+SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/GroupByWindow+SpannerIO.Write/Write mutations to Cloud Spanner/Batch mutations together+SpannerIO.Write/Write mutations to Cloud Spanner/Write mutations to Spanner Apr 12, 2018 11:17:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:17:19.059Z: Cleaning up. Apr 12, 2018 11:17:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:17:19.136Z: Stopping worker pool... Apr 12, 2018 11:18:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:18:56.300Z: Autoscaling: Resized worker pool from 7 to 0. Apr 12, 2018 11:18:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:18:56.334Z: Autoscaling: Would further reduce the number of workers but reached the minimum number allowed for the job. Apr 12, 2018 11:18:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-04-12T23:18:56.375Z: Worker pool stopped. Apr 12, 2018 11:19:04 PM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish INFO: Job 2018-04-12_16_14_41-11638957145981735383 finished with status DONE. Apr 12, 2018 11:19:04 PM org.apache.beam.runners.dataflow.TestDataflowRunner checkForPAssertSuccess INFO: Success result for Dataflow job 2018-04-12_16_14_41-11638957145981735383. Found 0 success, 0 failures out of 0 expected assertions. Gradle Test Executor 127 finished executing tests. :beam-runners-google-cloud-dataflow-java:googleCloudPlatformIntegrationTest FAILED :beam-runners-google-cloud-dataflow-java:googleCloudPlatformIntegrationTest (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 24 mins 36.72 secs. FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':beam-runners-google-cloud-dataflow-java:googleCloudPlatformIntegrationTest'. > Process 'Gradle Test Executor 128' finished with non-zero exit value 137 * Try: Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0. See https://docs.gradle.org/4.5.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 1h 19m 19s 528 actionable tasks: 528 executed Publishing build scan... https://gradle.com/s/56afd5g35h4yk Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure Recording test results Not sending mail to unregistered user ke...@google.com