See
<https://ci-beam.apache.org/job/beam_PostCommit_Python36/3850/display/redirect>
Changes:
------------------------------------------
[...truncated 45.74 MB...]
}
],
"is_pair_like": true,
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_3"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "out",
"user_name": "write/BigQueryBatchFileLoads/Flatten.out"
}
],
"user_name": "write/BigQueryBatchFileLoads/Flatten"
}
}
],
"type": "JOB_TYPE_BATCH"
}
apache_beam.runners.dataflow.internal.apiclient: INFO: Create job: <Job
createTime: '2021-05-03T12:24:32.354473Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-05-03_05_24_31-8984118972030335048'
location: 'us-central1'
name: 'beamapp-jenkins-0503122422-492799'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-05-03T12:24:32.354473Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
apache_beam.runners.dataflow.internal.apiclient: INFO: Created job with id:
[2021-05-03_05_24_31-8984118972030335048]
apache_beam.runners.dataflow.internal.apiclient: INFO: Submitted job:
2021-05-03_05_24_31-8984118972030335048
apache_beam.runners.dataflow.internal.apiclient: INFO: To access the Dataflow
monitoring console, please navigate to
https://console.cloud.google.com/dataflow/jobs/us-central1/2021-05-03_05_24_31-8984118972030335048?project=apache-beam-testing
apache_beam.runners.dataflow.dataflow_runner: INFO: Job
2021-05-03_05_24_31-8984118972030335048 is in state JOB_STATE_RUNNING
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:34.653Z:
JOB_MESSAGE_DETAILED: Autoscaling is enabled for job
2021-05-03_05_24_31-8984118972030335048. The number of workers will be between
1 and 1000.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:34.765Z:
JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job
2021-05-03_05_24_31-8984118972030335048.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:38.352Z:
JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:39.010Z:
JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:39.044Z:
JOB_MESSAGE_DEBUG: Combiner lifting skipped for step
write/BigQueryBatchFileLoads/RemoveTempTables/DeduplicateTables: GroupByKey not
followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:39.074Z:
JOB_MESSAGE_DEBUG: Combiner lifting skipped for step
write/BigQueryBatchFileLoads/GroupFilesByTableDestinations: GroupByKey not
followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:39.107Z:
JOB_MESSAGE_DEBUG: Combiner lifting skipped for step
write/BigQueryBatchFileLoads/GroupShardedRows: GroupByKey not followed by a
combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:39.143Z:
JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:39.173Z:
JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into
MergeBucketsMappingFns
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:39.302Z:
JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:39.344Z:
JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:39.369Z:
JOB_MESSAGE_DETAILED: Unzipping flatten s24 for input s18.WrittenFiles
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:39.400Z:
JOB_MESSAGE_DETAILED: Fusing unzipped copy of
write/BigQueryBatchFileLoads/IdentityWorkaround, through flatten
write/BigQueryBatchFileLoads/DestinationFilesUnion, into producer
write/BigQueryBatchFileLoads/ParDo(WriteRecordsToFile)/ParDo(WriteRecordsToFile)/ParDo(WriteRecordsToFile)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:39.423Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/GroupFilesByTableDestinations/Write into
write/BigQueryBatchFileLoads/GroupFilesByTableDestinations/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:39.455Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/GroupFilesByTableDestinations/GroupByWindow into
write/BigQueryBatchFileLoads/GroupFilesByTableDestinations/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:39.511Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/ParDo(PartitionFiles)/ParDo(PartitionFiles) into
write/BigQueryBatchFileLoads/GroupFilesByTableDestinations/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:39.559Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/TriggerLoadJobsWithTempTables/ParDo(TriggerLoadJobs)/ParDo(TriggerLoadJobs)
into write/BigQueryBatchFileLoads/ParDo(PartitionFiles)/ParDo(PartitionFiles)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:39.598Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/TriggerLoadJobsWithoutTempTables/TriggerLoadJobsWithoutTempTables
into write/BigQueryBatchFileLoads/ParDo(PartitionFiles)/ParDo(PartitionFiles)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:39.623Z:
JOB_MESSAGE_DETAILED: Unzipping flatten s24-u32 for input s25.None-c30
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:39.644Z:
JOB_MESSAGE_DETAILED: Fusing unzipped copy of
write/BigQueryBatchFileLoads/GroupFilesByTableDestinations/Reify, through
flatten write/BigQueryBatchFileLoads/DestinationFilesUnion/Unzipped-1, into
producer write/BigQueryBatchFileLoads/IdentityWorkaround
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:39.672Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/IdentityWorkaround into
write/BigQueryBatchFileLoads/WriteGroupedRecordsToFile/WriteGroupedRecordsToFile
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:39.697Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/GroupFilesByTableDestinations/Reify into
write/BigQueryBatchFileLoads/IdentityWorkaround
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:39.722Z:
JOB_MESSAGE_DETAILED: Fusing consumer
read/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough) into
read/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:39.744Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/RewindowIntoGlobal into
read/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:39.767Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/AppendDestination into
write/BigQueryBatchFileLoads/RewindowIntoGlobal
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:39.793Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/ParDo(WriteRecordsToFile)/ParDo(WriteRecordsToFile)/ParDo(WriteRecordsToFile)
into write/BigQueryBatchFileLoads/AppendDestination
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:39.825Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/ParDo(_ShardDestinations) into
write/BigQueryBatchFileLoads/ParDo(WriteRecordsToFile)/ParDo(WriteRecordsToFile)/ParDo(WriteRecordsToFile)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:39.860Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/GroupShardedRows/Reify into
write/BigQueryBatchFileLoads/ParDo(_ShardDestinations)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:39.894Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/GroupShardedRows/Write into
write/BigQueryBatchFileLoads/GroupShardedRows/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:39.926Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/GroupShardedRows/GroupByWindow into
write/BigQueryBatchFileLoads/GroupShardedRows/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:39.969Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/DropShardNumber into
write/BigQueryBatchFileLoads/GroupShardedRows/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:39.991Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/WriteGroupedRecordsToFile/WriteGroupedRecordsToFile
into write/BigQueryBatchFileLoads/DropShardNumber
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:40.025Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/LoadJobNamePrefix into
write/BigQueryBatchFileLoads/ImpulseSingleElementPC/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:40.056Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/SchemaModJobNamePrefix into
write/BigQueryBatchFileLoads/ImpulseSingleElementPC/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:40.079Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/CopyJobNamePrefix into
write/BigQueryBatchFileLoads/ImpulseSingleElementPC/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:40.111Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/GenerateFilePrefix into
write/BigQueryBatchFileLoads/ImpulseSingleElementPC/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:40.155Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/ParDo(UpdateDestinationSchema)/ParDo(UpdateDestinationSchema)
into
write/BigQueryBatchFileLoads/WaitForTempTableLoadJobs/WaitForTempTableLoadJobs
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:40.187Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/WaitForTempTableLoadJobs/WaitForTempTableLoadJobs
into write/BigQueryBatchFileLoads/ImpulseMonitorLoadJobs/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:40.220Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/WaitForSchemaModJobs/WaitForSchemaModJobs into
write/BigQueryBatchFileLoads/ImpulseMonitorSchemaModJobs/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:40.242Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/WaitForCopyJobs/WaitForCopyJobs into
write/BigQueryBatchFileLoads/ImpulseMonitorCopyJobs/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:40.274Z:
JOB_MESSAGE_DETAILED: Fusing consumer read/MapFilesToRemove into
read/FilesToRemoveImpulse/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:40.307Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/RemoveTempTables/PassTables/PassTables into
write/BigQueryBatchFileLoads/RemoveTempTables/Impulse/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:40.340Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/RemoveTempTables/AddUselessValue into
write/BigQueryBatchFileLoads/RemoveTempTables/PassTables/PassTables
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:40.371Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/RemoveTempTables/DeduplicateTables/Reify into
write/BigQueryBatchFileLoads/RemoveTempTables/AddUselessValue
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:40.392Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/RemoveTempTables/DeduplicateTables/Write into
write/BigQueryBatchFileLoads/RemoveTempTables/DeduplicateTables/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:40.418Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/RemoveTempTables/DeduplicateTables/GroupByWindow
into write/BigQueryBatchFileLoads/RemoveTempTables/DeduplicateTables/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:40.440Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/RemoveTempTables/GetTableNames/Keys into
write/BigQueryBatchFileLoads/RemoveTempTables/DeduplicateTables/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:40.494Z:
JOB_MESSAGE_DETAILED: Fusing consumer
read/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)/ParDo(RemoveExtractedFiles)
into read/_PassThroughThenCleanup/Create/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:40.543Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/RemoveTempTables/Delete into
write/BigQueryBatchFileLoads/RemoveTempTables/GetTableNames/Keys
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:40.565Z:
JOB_MESSAGE_DETAILED: Fusing consumer
write/BigQueryBatchFileLoads/WaitForDestinationLoadJobs/WaitForDestinationLoadJobs
into write/BigQueryBatchFileLoads/ImpulseMonitorDestinationLoadJobs/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:40.604Z:
JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:40.634Z:
JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:40.656Z:
JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:40.689Z:
JOB_MESSAGE_DEBUG: Assigning stage ids.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:40.828Z:
JOB_MESSAGE_DEBUG: Executing wait step start52
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:40.883Z:
JOB_MESSAGE_BASIC: Executing operation
read/FilesToRemoveImpulse/Read+read/MapFilesToRemove
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:40.905Z:
JOB_MESSAGE_BASIC: Executing operation
write/BigQueryBatchFileLoads/ImpulseSingleElementPC/Read+write/BigQueryBatchFileLoads/LoadJobNamePrefix+write/BigQueryBatchFileLoads/SchemaModJobNamePrefix+write/BigQueryBatchFileLoads/CopyJobNamePrefix+write/BigQueryBatchFileLoads/GenerateFilePrefix
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:40.929Z:
JOB_MESSAGE_DEBUG: Starting worker pool setup.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:40.938Z:
JOB_MESSAGE_BASIC: Executing operation
write/BigQueryBatchFileLoads/ImpulseEmptyPC/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:40.970Z:
JOB_MESSAGE_BASIC: Starting 1 workers in us-central1-f...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:40.972Z:
JOB_MESSAGE_BASIC: Finished operation
write/BigQueryBatchFileLoads/ImpulseEmptyPC/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:41.027Z:
JOB_MESSAGE_DEBUG: Value "write/BigQueryBatchFileLoads/ImpulseEmptyPC/Read.out"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:24:52.173Z:
JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric
descriptors, so new user metrics of the form custom.googleapis.com/* will not
be created. However, all user metrics are also available in the metric
dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics,
you can delete old / unused metric descriptors. See
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list
and
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:25:34.829Z:
JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 1 based on
the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:25:57.730Z:
JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:25:57.762Z:
JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:17.885Z:
JOB_MESSAGE_BASIC: Finished operation
read/FilesToRemoveImpulse/Read+read/MapFilesToRemove
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:18.170Z:
JOB_MESSAGE_DEBUG: Value "read/MapFilesToRemove.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:18.252Z:
JOB_MESSAGE_BASIC: Executing operation
read/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)/_UnpickledSideInput(MapFilesToRemove.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:18.310Z:
JOB_MESSAGE_BASIC: Finished operation
read/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)/_UnpickledSideInput(MapFilesToRemove.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:18.382Z:
JOB_MESSAGE_DEBUG: Value
"read/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)/_UnpickledSideInput(MapFilesToRemove.out.0).output"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:21.746Z:
JOB_MESSAGE_BASIC: Finished operation
write/BigQueryBatchFileLoads/ImpulseSingleElementPC/Read+write/BigQueryBatchFileLoads/LoadJobNamePrefix+write/BigQueryBatchFileLoads/SchemaModJobNamePrefix+write/BigQueryBatchFileLoads/CopyJobNamePrefix+write/BigQueryBatchFileLoads/GenerateFilePrefix
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:21.851Z:
JOB_MESSAGE_DEBUG: Value "write/BigQueryBatchFileLoads/LoadJobNamePrefix.out"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:21.884Z:
JOB_MESSAGE_DEBUG: Value
"write/BigQueryBatchFileLoads/SchemaModJobNamePrefix.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:21.924Z:
JOB_MESSAGE_DEBUG: Value "write/BigQueryBatchFileLoads/CopyJobNamePrefix.out"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:21.960Z:
JOB_MESSAGE_DEBUG: Value "write/BigQueryBatchFileLoads/GenerateFilePrefix.out"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:22.006Z:
JOB_MESSAGE_BASIC: Executing operation
write/BigQueryBatchFileLoads/TriggerLoadJobsWithTempTables/ParDo(TriggerLoadJobs)/_UnpickledSideInput(LoadJobNamePrefix.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:22.050Z:
JOB_MESSAGE_BASIC: Executing operation
write/BigQueryBatchFileLoads/TriggerLoadJobsWithoutTempTables/_UnpickledSideInput(LoadJobNamePrefix.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:22.066Z:
JOB_MESSAGE_BASIC: Finished operation
write/BigQueryBatchFileLoads/TriggerLoadJobsWithTempTables/ParDo(TriggerLoadJobs)/_UnpickledSideInput(LoadJobNamePrefix.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:22.094Z:
JOB_MESSAGE_BASIC: Executing operation
write/BigQueryBatchFileLoads/ParDo(UpdateDestinationSchema)/_UnpickledSideInput(SchemaModJobNamePrefix.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:22.099Z:
JOB_MESSAGE_BASIC: Finished operation
write/BigQueryBatchFileLoads/TriggerLoadJobsWithoutTempTables/_UnpickledSideInput(LoadJobNamePrefix.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:22.125Z:
JOB_MESSAGE_BASIC: Executing operation
write/BigQueryBatchFileLoads/ParDo(TriggerCopyJobs)/_UnpickledSideInput(CopyJobNamePrefix.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:22.150Z:
JOB_MESSAGE_BASIC: Finished operation
write/BigQueryBatchFileLoads/ParDo(UpdateDestinationSchema)/_UnpickledSideInput(SchemaModJobNamePrefix.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:22.171Z:
JOB_MESSAGE_BASIC: Executing operation
write/BigQueryBatchFileLoads/WriteGroupedRecordsToFile/_UnpickledSideInput(GenerateFilePrefix.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:22.202Z:
JOB_MESSAGE_BASIC: Executing operation
write/BigQueryBatchFileLoads/ParDo(WriteRecordsToFile)/ParDo(WriteRecordsToFile)/_UnpickledSideInput(GenerateFilePrefix.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:22.215Z:
JOB_MESSAGE_BASIC: Finished operation
write/BigQueryBatchFileLoads/ParDo(TriggerCopyJobs)/_UnpickledSideInput(CopyJobNamePrefix.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:22.229Z:
JOB_MESSAGE_BASIC: Finished operation
write/BigQueryBatchFileLoads/WriteGroupedRecordsToFile/_UnpickledSideInput(GenerateFilePrefix.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:22.246Z:
JOB_MESSAGE_DEBUG: Value
"write/BigQueryBatchFileLoads/TriggerLoadJobsWithTempTables/ParDo(TriggerLoadJobs)/_UnpickledSideInput(LoadJobNamePrefix.out.0).output"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:22.280Z:
JOB_MESSAGE_DEBUG: Value
"write/BigQueryBatchFileLoads/TriggerLoadJobsWithoutTempTables/_UnpickledSideInput(LoadJobNamePrefix.out.0).output"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:22.286Z:
JOB_MESSAGE_BASIC: Finished operation
write/BigQueryBatchFileLoads/ParDo(WriteRecordsToFile)/ParDo(WriteRecordsToFile)/_UnpickledSideInput(GenerateFilePrefix.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:22.312Z:
JOB_MESSAGE_DEBUG: Value
"write/BigQueryBatchFileLoads/ParDo(UpdateDestinationSchema)/_UnpickledSideInput(SchemaModJobNamePrefix.out.0).output"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:22.356Z:
JOB_MESSAGE_DEBUG: Value
"write/BigQueryBatchFileLoads/ParDo(TriggerCopyJobs)/_UnpickledSideInput(CopyJobNamePrefix.out.0).output"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:22.398Z:
JOB_MESSAGE_DEBUG: Value
"write/BigQueryBatchFileLoads/WriteGroupedRecordsToFile/_UnpickledSideInput(GenerateFilePrefix.out.0).output"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:22.421Z:
JOB_MESSAGE_DEBUG: Value
"write/BigQueryBatchFileLoads/ParDo(WriteRecordsToFile)/ParDo(WriteRecordsToFile)/_UnpickledSideInput(GenerateFilePrefix.out.0).output"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:22.499Z:
JOB_MESSAGE_BASIC: Executing operation
write/BigQueryBatchFileLoads/GroupShardedRows/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:22.974Z:
JOB_MESSAGE_BASIC: Finished operation
write/BigQueryBatchFileLoads/GroupShardedRows/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:23.091Z:
JOB_MESSAGE_DEBUG: Value
"write/BigQueryBatchFileLoads/GroupShardedRows/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:23.200Z:
JOB_MESSAGE_BASIC: Executing operation
write/BigQueryBatchFileLoads/GroupFilesByTableDestinations/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:23.354Z:
JOB_MESSAGE_BASIC: Finished operation
write/BigQueryBatchFileLoads/GroupFilesByTableDestinations/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:23.435Z:
JOB_MESSAGE_DEBUG: Value
"write/BigQueryBatchFileLoads/GroupFilesByTableDestinations/Session"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:23.510Z:
JOB_MESSAGE_BASIC: Executing operation
read/Read+read/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough)+write/BigQueryBatchFileLoads/RewindowIntoGlobal+write/BigQueryBatchFileLoads/AppendDestination+write/BigQueryBatchFileLoads/ParDo(WriteRecordsToFile)/ParDo(WriteRecordsToFile)/ParDo(WriteRecordsToFile)+write/BigQueryBatchFileLoads/IdentityWorkaround+write/BigQueryBatchFileLoads/GroupFilesByTableDestinations/Reify+write/BigQueryBatchFileLoads/GroupFilesByTableDestinations/Write+write/BigQueryBatchFileLoads/ParDo(_ShardDestinations)+write/BigQueryBatchFileLoads/GroupShardedRows/Reify+write/BigQueryBatchFileLoads/GroupShardedRows/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:43.336Z:
JOB_MESSAGE_BASIC: Finished operation
read/Read+read/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough)+write/BigQueryBatchFileLoads/RewindowIntoGlobal+write/BigQueryBatchFileLoads/AppendDestination+write/BigQueryBatchFileLoads/ParDo(WriteRecordsToFile)/ParDo(WriteRecordsToFile)/ParDo(WriteRecordsToFile)+write/BigQueryBatchFileLoads/IdentityWorkaround+write/BigQueryBatchFileLoads/GroupFilesByTableDestinations/Reify+write/BigQueryBatchFileLoads/GroupFilesByTableDestinations/Write+write/BigQueryBatchFileLoads/ParDo(_ShardDestinations)+write/BigQueryBatchFileLoads/GroupShardedRows/Reify+write/BigQueryBatchFileLoads/GroupShardedRows/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:43.415Z:
JOB_MESSAGE_DEBUG: Value
"read/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough).cleanup_signal"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:43.457Z:
JOB_MESSAGE_BASIC: Executing operation
write/BigQueryBatchFileLoads/GroupShardedRows/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:43.500Z:
JOB_MESSAGE_BASIC: Executing operation
read/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)/_UnpickledSideInput(ParDo(PassThrough).cleanup_signal.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:43.510Z:
JOB_MESSAGE_BASIC: Finished operation
write/BigQueryBatchFileLoads/GroupShardedRows/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:43.586Z:
JOB_MESSAGE_BASIC: Finished operation
read/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)/_UnpickledSideInput(ParDo(PassThrough).cleanup_signal.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:43.609Z:
JOB_MESSAGE_BASIC: Executing operation
write/BigQueryBatchFileLoads/GroupShardedRows/Read+write/BigQueryBatchFileLoads/GroupShardedRows/GroupByWindow+write/BigQueryBatchFileLoads/DropShardNumber+write/BigQueryBatchFileLoads/WriteGroupedRecordsToFile/WriteGroupedRecordsToFile+write/BigQueryBatchFileLoads/IdentityWorkaround+write/BigQueryBatchFileLoads/GroupFilesByTableDestinations/Reify+write/BigQueryBatchFileLoads/GroupFilesByTableDestinations/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:43.674Z:
JOB_MESSAGE_DEBUG: Value
"read/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)/_UnpickledSideInput(ParDo(PassThrough).cleanup_signal.0).output"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:43.773Z:
JOB_MESSAGE_BASIC: Executing operation
read/_PassThroughThenCleanup/Create/Read+read/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)/ParDo(RemoveExtractedFiles)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:46.285Z:
JOB_MESSAGE_BASIC: Finished operation
write/BigQueryBatchFileLoads/GroupShardedRows/Read+write/BigQueryBatchFileLoads/GroupShardedRows/GroupByWindow+write/BigQueryBatchFileLoads/DropShardNumber+write/BigQueryBatchFileLoads/WriteGroupedRecordsToFile/WriteGroupedRecordsToFile+write/BigQueryBatchFileLoads/IdentityWorkaround+write/BigQueryBatchFileLoads/GroupFilesByTableDestinations/Reify+write/BigQueryBatchFileLoads/GroupFilesByTableDestinations/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:46.367Z:
JOB_MESSAGE_BASIC: Executing operation
write/BigQueryBatchFileLoads/GroupFilesByTableDestinations/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:46.429Z:
JOB_MESSAGE_BASIC: Finished operation
write/BigQueryBatchFileLoads/GroupFilesByTableDestinations/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:38:46.498Z:
JOB_MESSAGE_BASIC: Executing operation
write/BigQueryBatchFileLoads/GroupFilesByTableDestinations/Read+write/BigQueryBatchFileLoads/GroupFilesByTableDestinations/GroupByWindow+write/BigQueryBatchFileLoads/ParDo(PartitionFiles)/ParDo(PartitionFiles)+write/BigQueryBatchFileLoads/TriggerLoadJobsWithTempTables/ParDo(TriggerLoadJobs)/ParDo(TriggerLoadJobs)+write/BigQueryBatchFileLoads/TriggerLoadJobsWithoutTempTables/TriggerLoadJobsWithoutTempTables
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:01.278Z:
JOB_MESSAGE_BASIC: Finished operation
read/_PassThroughThenCleanup/Create/Read+read/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)/ParDo(RemoveExtractedFiles)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:05.583Z:
JOB_MESSAGE_BASIC: Finished operation
write/BigQueryBatchFileLoads/GroupFilesByTableDestinations/Read+write/BigQueryBatchFileLoads/GroupFilesByTableDestinations/GroupByWindow+write/BigQueryBatchFileLoads/ParDo(PartitionFiles)/ParDo(PartitionFiles)+write/BigQueryBatchFileLoads/TriggerLoadJobsWithTempTables/ParDo(TriggerLoadJobs)/ParDo(TriggerLoadJobs)+write/BigQueryBatchFileLoads/TriggerLoadJobsWithoutTempTables/TriggerLoadJobsWithoutTempTables
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:05.667Z:
JOB_MESSAGE_DEBUG: Value
"write/BigQueryBatchFileLoads/TriggerLoadJobsWithTempTables/ParDo(TriggerLoadJobs).out"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:05.729Z:
JOB_MESSAGE_DEBUG: Value
"write/BigQueryBatchFileLoads/TriggerLoadJobsWithTempTables/ParDo(TriggerLoadJobs).TemporaryTables"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:05.777Z:
JOB_MESSAGE_DEBUG: Value
"write/BigQueryBatchFileLoads/TriggerLoadJobsWithoutTempTables.out"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:05.820Z:
JOB_MESSAGE_BASIC: Executing operation
write/BigQueryBatchFileLoads/WaitForTempTableLoadJobs/_UnpickledSideInput(ParDo(TriggerLoadJobs).out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:05.853Z:
JOB_MESSAGE_BASIC: Executing operation
write/BigQueryBatchFileLoads/RemoveTempTables/PassTables/_UnpickledSideInput(ParDo(TriggerLoadJobs).TemporaryTables.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:05.867Z:
JOB_MESSAGE_BASIC: Finished operation
write/BigQueryBatchFileLoads/WaitForTempTableLoadJobs/_UnpickledSideInput(ParDo(TriggerLoadJobs).out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:05.885Z:
JOB_MESSAGE_BASIC: Executing operation
write/BigQueryBatchFileLoads/WaitForDestinationLoadJobs/_UnpickledSideInput(TriggerLoadJobsWithoutTempTables.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:05.894Z:
JOB_MESSAGE_BASIC: Finished operation
write/BigQueryBatchFileLoads/RemoveTempTables/PassTables/_UnpickledSideInput(ParDo(TriggerLoadJobs).TemporaryTables.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:05.918Z:
JOB_MESSAGE_BASIC: Executing operation write/BigQueryBatchFileLoads/Flatten
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:05.930Z:
JOB_MESSAGE_BASIC: Finished operation
write/BigQueryBatchFileLoads/WaitForDestinationLoadJobs/_UnpickledSideInput(TriggerLoadJobsWithoutTempTables.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:05.952Z:
JOB_MESSAGE_DEBUG: Value
"write/BigQueryBatchFileLoads/WaitForTempTableLoadJobs/_UnpickledSideInput(ParDo(TriggerLoadJobs).out.0).output"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:05.984Z:
JOB_MESSAGE_BASIC: Finished operation write/BigQueryBatchFileLoads/Flatten
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:05.995Z:
JOB_MESSAGE_DEBUG: Value
"write/BigQueryBatchFileLoads/RemoveTempTables/PassTables/_UnpickledSideInput(ParDo(TriggerLoadJobs).TemporaryTables.0).output"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:06.017Z:
JOB_MESSAGE_DEBUG: Value
"write/BigQueryBatchFileLoads/WaitForDestinationLoadJobs/_UnpickledSideInput(TriggerLoadJobsWithoutTempTables.out.0).output"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:06.049Z:
JOB_MESSAGE_BASIC: Executing operation
write/BigQueryBatchFileLoads/ImpulseMonitorLoadJobs/Read+write/BigQueryBatchFileLoads/WaitForTempTableLoadJobs/WaitForTempTableLoadJobs+write/BigQueryBatchFileLoads/ParDo(UpdateDestinationSchema)/ParDo(UpdateDestinationSchema)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:06.081Z:
JOB_MESSAGE_DEBUG: Value "write/BigQueryBatchFileLoads/Flatten.out"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:06.116Z:
JOB_MESSAGE_BASIC: Executing operation
write/BigQueryBatchFileLoads/ImpulseMonitorDestinationLoadJobs/Read+write/BigQueryBatchFileLoads/WaitForDestinationLoadJobs/WaitForDestinationLoadJobs
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:14.945Z:
JOB_MESSAGE_BASIC: Finished operation
write/BigQueryBatchFileLoads/ImpulseMonitorDestinationLoadJobs/Read+write/BigQueryBatchFileLoads/WaitForDestinationLoadJobs/WaitForDestinationLoadJobs
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:19.257Z:
JOB_MESSAGE_BASIC: Finished operation
write/BigQueryBatchFileLoads/ImpulseMonitorLoadJobs/Read+write/BigQueryBatchFileLoads/WaitForTempTableLoadJobs/WaitForTempTableLoadJobs+write/BigQueryBatchFileLoads/ParDo(UpdateDestinationSchema)/ParDo(UpdateDestinationSchema)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:19.341Z:
JOB_MESSAGE_DEBUG: Value
"write/BigQueryBatchFileLoads/WaitForTempTableLoadJobs.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:19.366Z:
JOB_MESSAGE_DEBUG: Value
"write/BigQueryBatchFileLoads/ParDo(UpdateDestinationSchema).out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:19.450Z:
JOB_MESSAGE_BASIC: Executing operation
write/BigQueryBatchFileLoads/WaitForSchemaModJobs/_UnpickledSideInput(ParDo(UpdateDestinationSchema).out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:19.497Z:
JOB_MESSAGE_BASIC: Finished operation
write/BigQueryBatchFileLoads/WaitForSchemaModJobs/_UnpickledSideInput(ParDo(UpdateDestinationSchema).out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:19.569Z:
JOB_MESSAGE_DEBUG: Value
"write/BigQueryBatchFileLoads/WaitForSchemaModJobs/_UnpickledSideInput(ParDo(UpdateDestinationSchema).out.0).output"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:19.637Z:
JOB_MESSAGE_BASIC: Executing operation
write/BigQueryBatchFileLoads/ImpulseMonitorSchemaModJobs/Read+write/BigQueryBatchFileLoads/WaitForSchemaModJobs/WaitForSchemaModJobs
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:26.068Z:
JOB_MESSAGE_BASIC: Finished operation
write/BigQueryBatchFileLoads/ImpulseMonitorSchemaModJobs/Read+write/BigQueryBatchFileLoads/WaitForSchemaModJobs/WaitForSchemaModJobs
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:26.144Z:
JOB_MESSAGE_DEBUG: Value
"write/BigQueryBatchFileLoads/WaitForSchemaModJobs.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:26.228Z:
JOB_MESSAGE_BASIC: Executing operation
write/BigQueryBatchFileLoads/ParDo(TriggerCopyJobs)/_UnpickledSideInput(WaitForSchemaModJobs.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:26.273Z:
JOB_MESSAGE_BASIC: Finished operation
write/BigQueryBatchFileLoads/ParDo(TriggerCopyJobs)/_UnpickledSideInput(WaitForSchemaModJobs.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:26.351Z:
JOB_MESSAGE_DEBUG: Value
"write/BigQueryBatchFileLoads/ParDo(TriggerCopyJobs)/_UnpickledSideInput(WaitForSchemaModJobs.out.0).output"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:26.430Z:
JOB_MESSAGE_BASIC: Executing operation
write/BigQueryBatchFileLoads/ParDo(TriggerCopyJobs)/ParDo(TriggerCopyJobs)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:30.052Z:
JOB_MESSAGE_BASIC: Finished operation
write/BigQueryBatchFileLoads/ParDo(TriggerCopyJobs)/ParDo(TriggerCopyJobs)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:30.123Z:
JOB_MESSAGE_DEBUG: Value
"write/BigQueryBatchFileLoads/ParDo(TriggerCopyJobs).out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:30.226Z:
JOB_MESSAGE_BASIC: Executing operation
write/BigQueryBatchFileLoads/WaitForCopyJobs/_UnpickledSideInput(ParDo(TriggerCopyJobs).out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:30.272Z:
JOB_MESSAGE_BASIC: Finished operation
write/BigQueryBatchFileLoads/WaitForCopyJobs/_UnpickledSideInput(ParDo(TriggerCopyJobs).out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:30.346Z:
JOB_MESSAGE_DEBUG: Value
"write/BigQueryBatchFileLoads/WaitForCopyJobs/_UnpickledSideInput(ParDo(TriggerCopyJobs).out.0).output"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:30.435Z:
JOB_MESSAGE_BASIC: Executing operation
write/BigQueryBatchFileLoads/ImpulseMonitorCopyJobs/Read+write/BigQueryBatchFileLoads/WaitForCopyJobs/WaitForCopyJobs
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:35.012Z:
JOB_MESSAGE_BASIC: Finished operation
write/BigQueryBatchFileLoads/ImpulseMonitorCopyJobs/Read+write/BigQueryBatchFileLoads/WaitForCopyJobs/WaitForCopyJobs
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:35.083Z:
JOB_MESSAGE_DEBUG: Value "write/BigQueryBatchFileLoads/WaitForCopyJobs.out"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:35.145Z:
JOB_MESSAGE_BASIC: Executing operation
write/BigQueryBatchFileLoads/RemoveTempTables/PassTables/_UnpickledSideInput(WaitForCopyJobs.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:35.202Z:
JOB_MESSAGE_BASIC: Finished operation
write/BigQueryBatchFileLoads/RemoveTempTables/PassTables/_UnpickledSideInput(WaitForCopyJobs.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:35.285Z:
JOB_MESSAGE_DEBUG: Value
"write/BigQueryBatchFileLoads/RemoveTempTables/PassTables/_UnpickledSideInput(WaitForCopyJobs.out.0).output"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:35.352Z:
JOB_MESSAGE_BASIC: Executing operation
write/BigQueryBatchFileLoads/RemoveTempTables/DeduplicateTables/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:35.506Z:
JOB_MESSAGE_BASIC: Finished operation
write/BigQueryBatchFileLoads/RemoveTempTables/DeduplicateTables/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:35.585Z:
JOB_MESSAGE_DEBUG: Value
"write/BigQueryBatchFileLoads/RemoveTempTables/DeduplicateTables/Session"
materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:35.663Z:
JOB_MESSAGE_BASIC: Executing operation
write/BigQueryBatchFileLoads/RemoveTempTables/Impulse/Read+write/BigQueryBatchFileLoads/RemoveTempTables/PassTables/PassTables+write/BigQueryBatchFileLoads/RemoveTempTables/AddUselessValue+write/BigQueryBatchFileLoads/RemoveTempTables/DeduplicateTables/Reify+write/BigQueryBatchFileLoads/RemoveTempTables/DeduplicateTables/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:40.384Z:
JOB_MESSAGE_BASIC: Finished operation
write/BigQueryBatchFileLoads/RemoveTempTables/Impulse/Read+write/BigQueryBatchFileLoads/RemoveTempTables/PassTables/PassTables+write/BigQueryBatchFileLoads/RemoveTempTables/AddUselessValue+write/BigQueryBatchFileLoads/RemoveTempTables/DeduplicateTables/Reify+write/BigQueryBatchFileLoads/RemoveTempTables/DeduplicateTables/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:40.450Z:
JOB_MESSAGE_BASIC: Executing operation
write/BigQueryBatchFileLoads/RemoveTempTables/DeduplicateTables/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:40.496Z:
JOB_MESSAGE_BASIC: Finished operation
write/BigQueryBatchFileLoads/RemoveTempTables/DeduplicateTables/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:40.560Z:
JOB_MESSAGE_BASIC: Executing operation
write/BigQueryBatchFileLoads/RemoveTempTables/DeduplicateTables/Read+write/BigQueryBatchFileLoads/RemoveTempTables/DeduplicateTables/GroupByWindow+write/BigQueryBatchFileLoads/RemoveTempTables/GetTableNames/Keys+write/BigQueryBatchFileLoads/RemoveTempTables/Delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:43.573Z:
JOB_MESSAGE_BASIC: Finished operation
write/BigQueryBatchFileLoads/RemoveTempTables/DeduplicateTables/Read+write/BigQueryBatchFileLoads/RemoveTempTables/DeduplicateTables/GroupByWindow+write/BigQueryBatchFileLoads/RemoveTempTables/GetTableNames/Keys+write/BigQueryBatchFileLoads/RemoveTempTables/Delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:43.646Z:
JOB_MESSAGE_DEBUG: Executing success step success50
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:43.737Z:
JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:43.795Z:
JOB_MESSAGE_DEBUG: Starting worker pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2021-05-03T12:39:43.836Z:
JOB_MESSAGE_BASIC: Stopping worker pool...
apache_beam.runners.dataflow.dataflow_runner: WARNING: Timing out on waiting
for job 2021-05-03_05_24_31-8984118972030335048 after 902 seconds
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: nosetests-postCommitIT-df-py36.xml
----------------------------------------------------------------------
XML:
<https://ci-beam.apache.org/job/beam_PostCommit_Python36/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 72 tests in 5072.014s
FAILED (SKIP=6, errors=3, failures=1)
> Task :sdks:python:test-suites:dataflow:py36:postCommitIT FAILED
FAILURE: Build failed with an exception.
* Where:
Script
'<https://ci-beam.apache.org/job/beam_PostCommit_Python36/ws/src/sdks/python/test-suites/dataflow/common.gradle'>
line: 118
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py36:postCommitIT'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 28m 37s
208 actionable tasks: 148 executed, 56 from cache, 4 up-to-date
Gradle was unable to watch the file system for changes. The inotify watches
limit is too low.
Publishing build scan...
https://gradle.com/s/4e6bw33cdeaww
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
