See <https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1333/display/redirect?page=changes>
Changes: [jiangkai] Support datetime type minus time interval [jiangkai] address comments [jiangkai] add unit test for DatetimeMinusIntervalExpression [jiangkai] add license header ------------------------------------------ [...truncated 20.27 MB...] INFO: 2018-08-24T00:30:16.257Z: Autoscaling was automatically enabled for job 2018-08-23_17_30_16-7298639393089505239. Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:19.630Z: Checking required Cloud APIs are enabled. Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:19.938Z: Checking permissions granted to controller Service Account. Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:23.919Z: Worker configuration: n1-standard-1 in us-central1-f. Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:24.352Z: Expanding CoGroupByKey operations into optimizable parts. Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:24.578Z: Expanding GroupByKey operations into optimizable parts. Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:24.611Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:24.845Z: Fusing adjacent ParDo, Read, Write, and Flatten operations Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:24.898Z: Elided trivial flatten Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:24.932Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write mutations to Cloud Spanner/Create seed/Read(CreateSource) Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:24.963Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/Wait/Map Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:24.999Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:25.032Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) into SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:25.071Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:25.108Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues into SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:25.148Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map into SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:25.179Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) into SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:25.213Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract into SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:25.257Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial into SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:25.307Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify into SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:25.350Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write into SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:25.399Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/ParDo(CollectWindows) into SpannerIO.Write/To mutation group Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:25.434Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Extract keys into SpannerIO.Write/Write mutations to Cloud Spanner/Serialize mutations Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:25.477Z: Fusing consumer ParDo(GenerateMutations) into GenerateSequence/Read(BoundedCountingSource) Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:25.521Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/Combine.GroupedValues into SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Read Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:25.561Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Batch mutations together into SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/GroupByWindow Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:25.604Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/ParMultiDo(ToIsmRecordForMapLike) into SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:25.649Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow) Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:25.688Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow) into SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/Combine.GroupedValues/Extract Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:25.736Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Write into SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Reify Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:25.780Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/Combine.GroupedValues/Extract into SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/Combine.GroupedValues Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:25.824Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Reify into SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey+SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/Combine.GroupedValues/Partial Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:25.868Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey+SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/Combine.GroupedValues/Partial into SpannerIO.Write/Write mutations to Cloud Spanner/Extract keys Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:25.910Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Write into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Reify Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:25.955Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForSize/Write into SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/ParMultiDo(ToIsmRecordForMapLike) Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:25.998Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/View.AsList/ParDo(ToIsmRecordForGlobalWindow) into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Flatten.Iterables/FlattenIterables/FlatMap Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:26.039Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForKeys/Write into SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/ParMultiDo(ToIsmRecordForMapLike) Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:26.080Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Flatten.Iterables/FlattenIterables/FlatMap into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Values/Values/Map Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:26.131Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Values/Values/Map into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues/Extract Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:26.171Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/ParDo(ToIsmMetadataRecordForKey) into SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForKeys/Read Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:26.208Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues/Extract into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:26.256Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/WithKeys/AddKeys/Map into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/ParDo(CollectWindows) Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:26.303Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Read Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:26.349Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Write mutations to Spanner into SpannerIO.Write/Write mutations to Cloud Spanner/Batch mutations together Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:26.407Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Reify into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey+SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues/Partial Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:26.448Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey+SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues/Partial into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/WithKeys/AddKeys/Map Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:26.490Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/Reify into SpannerIO.Write/Write mutations to Cloud Spanner/Partition input Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:26.536Z: Fusing consumer SpannerIO.Write/To mutation group into ParDo(GenerateMutations) Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:26.576Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/GroupByWindow into SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/Read Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:26.622Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/Write into SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/Reify Aug 24, 2018 12:30:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:26.668Z: Fusing consumer SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/ParDo(ToIsmMetadataRecordForSize) into SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForSize/Read Aug 24, 2018 12:30:28 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:27.184Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Create Aug 24, 2018 12:30:28 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:27.230Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create Aug 24, 2018 12:30:28 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:27.279Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Create Aug 24, 2018 12:30:28 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:27.289Z: Starting 1 workers in us-central1-f... Aug 24, 2018 12:30:28 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:27.325Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create Aug 24, 2018 12:30:28 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:27.380Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForSize/Create Aug 24, 2018 12:30:28 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:27.423Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForKeys/Create Aug 24, 2018 12:30:28 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:27.471Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/Create Aug 24, 2018 12:30:28 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:27.514Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Create Aug 24, 2018 12:30:28 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:27.993Z: Executing operation GenerateSequence/Read(BoundedCountingSource)+ParDo(GenerateMutations)+SpannerIO.Write/To mutation group+SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/ParDo(CollectWindows)+SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/WithKeys/AddKeys/Map+SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey+SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues/Partial+SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Reify+SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Write Aug 24, 2018 12:30:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:39.642Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s). Aug 24, 2018 12:30:50 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:50.161Z: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running step(s). Aug 24, 2018 12:30:50 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:30:50.205Z: Autoscaling: Would further reduce the number of workers but reached the minimum number allowed for the job. Aug 24, 2018 12:31:15 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:31:14.393Z: Workers have started successfully. Aug 24, 2018 12:31:15 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:31:14.452Z: Workers have started successfully. Aug 24, 2018 12:31:35 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:31:33.821Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Close Aug 24, 2018 12:31:35 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:31:34.090Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Read+SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues+SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues/Extract+SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Combine.globally(SampleAny)/Values/Values/Map+SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/Sample.Any/Flatten.Iterables/FlattenIterables/FlatMap+SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/View.AsList/ParDo(ToIsmRecordForGlobalWindow) Aug 24, 2018 12:31:45 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:31:43.784Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/View.AsList/CreateDataflowView Aug 24, 2018 12:31:45 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:31:44.062Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Create seed/Read(CreateSource)+SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/Wait/Map+SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write Aug 24, 2018 12:31:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:31:51.559Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Close Aug 24, 2018 12:31:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:31:51.714Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write Aug 24, 2018 12:31:59 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:31:57.577Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close Aug 24, 2018 12:31:59 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:31:57.877Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) Aug 24, 2018 12:32:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:32:04.674Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/CreateDataflowView Aug 24, 2018 12:32:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:32:04.893Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Serialize mutations+SpannerIO.Write/Write mutations to Cloud Spanner/Extract keys+SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey+SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/Combine.GroupedValues/Partial+SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Reify+SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Write Aug 24, 2018 12:32:16 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:32:15.245Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Close Aug 24, 2018 12:32:16 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:32:15.621Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Read+SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/Combine.GroupedValues+SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/Combine.GroupedValues/Extract+SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow)+SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write Aug 24, 2018 12:32:23 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:32:21.122Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close Aug 24, 2018 12:32:23 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:32:21.491Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/ParMultiDo(ToIsmRecordForMapLike)+SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForSize/Write+SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForKeys/Write Aug 24, 2018 12:32:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:32:30.990Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForSize/Close Aug 24, 2018 12:32:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:32:31.146Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForKeys/Close Aug 24, 2018 12:32:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:32:31.693Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForSize/Read+SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/ParDo(ToIsmMetadataRecordForSize) Aug 24, 2018 12:32:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:32:31.874Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/GBKaSVForKeys/Read+SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/ParDo(ToIsmMetadataRecordForKey) Aug 24, 2018 12:32:45 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:32:45.398Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/Flatten.PCollections Aug 24, 2018 12:32:48 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:32:45.651Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as view/CreateDataflowView Aug 24, 2018 12:32:48 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:32:46.367Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Partition input+SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/Reify+SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/Write Aug 24, 2018 12:32:56 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:32:54.444Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/Close Aug 24, 2018 12:32:56 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:32:54.544Z: Executing operation SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/Read+SpannerIO.Write/Write mutations to Cloud Spanner/Group by partition/GroupByWindow+SpannerIO.Write/Write mutations to Cloud Spanner/Batch mutations together+SpannerIO.Write/Write mutations to Cloud Spanner/Write mutations to Spanner Aug 24, 2018 12:33:02 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:33:01.162Z: Cleaning up. Aug 24, 2018 12:33:02 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:33:01.341Z: Stopping worker pool... Aug 24, 2018 12:34:15 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:34:14.701Z: Autoscaling: Resized worker pool from 1 to 0. Aug 24, 2018 12:34:15 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-08-24T00:34:14.739Z: Autoscaling: Would further reduce the number of workers but reached the minimum number allowed for the job. Aug 24, 2018 12:34:23 AM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish INFO: Job 2018-08-23_17_30_16-7298639393089505239 finished with status DONE. Aug 24, 2018 12:34:23 AM org.apache.beam.runners.dataflow.TestDataflowRunner checkForPAssertSuccess INFO: Success result for Dataflow job 2018-08-23_17_30_16-7298639393089505239. Found 0 success, 0 failures out of 0 expected assertions. Aug 24, 2018 12:34:23 AM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish INFO: Job 2018-08-23_17_30_16-7298639393089505239 finished with status DONE. Gradle Test Executor 147 finished executing tests. > Task > :beam-runners-google-cloud-dataflow-java:googleCloudPlatformIntegrationTest Finished generating test XML results (0.01 secs) into: <https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/ws/src/runners/google-cloud-dataflow-java/build/test-results/googleCloudPlatformIntegrationTest> Generating HTML test report... Finished generating test html results (0.012 secs) into: <https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/ws/src/runners/google-cloud-dataflow-java/build/reports/tests/googleCloudPlatformIntegrationTest> Packing task ':beam-runners-google-cloud-dataflow-java:googleCloudPlatformIntegrationTest' :beam-runners-google-cloud-dataflow-java:googleCloudPlatformIntegrationTest (Thread[Daemon worker,5,main]) completed. Took 29 mins 45.959 secs. :beam-runners-google-cloud-dataflow-java:buildDependents (Thread[Daemon worker,5,main]) started. > Task :beam-runners-google-cloud-dataflow-java:buildDependents Caching disabled for task ':beam-runners-google-cloud-dataflow-java:buildDependents': Caching has not been enabled for the task Task ':beam-runners-google-cloud-dataflow-java:buildDependents' is not up-to-date because: Task has not declared any outputs despite executing actions. :beam-runners-google-cloud-dataflow-java:buildDependents (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :beam-sdks-java-io-google-cloud-platform:buildDependents (Thread[Daemon worker,5,main]) started. :beam-runners-google-cloud-dataflow-java:postCommit (Thread[Task worker for ':' Thread 6,5,main]) started. > Task :beam-runners-google-cloud-dataflow-java:postCommit Skipping task ':beam-runners-google-cloud-dataflow-java:postCommit' as it has no actions. > Task :beam-sdks-java-io-google-cloud-platform:buildDependents Caching disabled for task ':beam-sdks-java-io-google-cloud-platform:buildDependents': Caching has not been enabled for the task Task ':beam-sdks-java-io-google-cloud-platform:buildDependents' is not up-to-date because: Task has not declared any outputs despite executing actions. :beam-runners-google-cloud-dataflow-java:postCommit (Thread[Task worker for ':' Thread 6,5,main]) completed. Took 0.0 secs. :beam-sdks-java-io-google-cloud-platform:buildDependents (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. :beam-sdks-java-extensions-protobuf:buildDependents (Thread[Daemon worker,5,main]) started. > Task :beam-sdks-java-extensions-protobuf:buildDependents Caching disabled for task ':beam-sdks-java-extensions-protobuf:buildDependents': Caching has not been enabled for the task Task ':beam-sdks-java-extensions-protobuf:buildDependents' is not up-to-date because: Task has not declared any outputs despite executing actions. :beam-sdks-java-extensions-protobuf:buildDependents (Thread[Daemon worker,5,main]) completed. Took 0.0 secs. FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':beam-runners-spark:test'. > There were failing tests. See the report at: > file://<https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/ws/src/runners/spark/build/reports/tests/test/index.html> * Try: Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0. See https://docs.gradle.org/4.8/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 45m 29s 804 actionable tasks: 799 executed, 5 from cache Publishing build scan... https://gradle.com/s/lgc7jov55qbaw Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure