See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/24/display/redirect?page=changes>
Changes: [david.prieto.rivera] Missing contribution [noreply] Merge pull request #15848 from [BEAM-13835] An any-type implementation [Valentyn Tymofieiev] [BEAM-12920] Assume that bare generators types define simple generators. [noreply] [release-2.36.0][website] Fix github release notes script, header for [noreply] Use shell to run python for setupVirtualenv (#16796) [Daniel Oliveira] [BEAM-13830] Properly shut down Debezium expansion service in IT script. [noreply] Merge pull request #16659 from [BEAM-13774][Playground] Add user to [noreply] [BEAM-13776][Playground] (#16731) ------------------------------------------ [...truncated 79.35 KB...] [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:19.355Z: JOB_MESSAGE_DEBUG: Inserted coder converter after flatten ref_AppliedPTransform_ToPCollection-df---gs-temp-storage-for-end-to-end-tests-temp-it-flight_delays__44 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:19.411Z: JOB_MESSAGE_DETAILED: Elided trivial flatten [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:19.440Z: JOB_MESSAGE_DETAILED: Unzipping flatten ref_AppliedPTransform_WriteToPandas-df---gs-temp-storage-for-end-to-end-tests-temp-it-flight_delays__58 for input ref_AppliedPTransform_WriteToPandas-df---gs-temp-storage-for-end-to-end-tests-temp-it-flight_delays__54.written_files [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:19.480Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/Map(<lambda at fileio.py:584>), through flatten WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/Flatten, into producer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/ParDo(_WriteUnshardedRecordsFn)/ParDo(_WriteUnshardedRecordsFn) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:19.508Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupTempFilesByDestination/Write into WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupTempFilesByDestination/Reify [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:19.545Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupTempFilesByDestination/GroupByWindow into WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupTempFilesByDestination/Read [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:19.576Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/ParDo(_MoveTempFilesIntoFinalDestinationFn) into WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupTempFilesByDestination/GroupByWindow [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:19.603Z: JOB_MESSAGE_DETAILED: Unzipping flatten ref_AppliedPTransform_WriteToPandas-df---gs-temp-storage-for-end-to-end-tests-temp-it-flight_delays__58-u39 for input ref_AppliedPTransform_WriteToPandas-df---gs-temp-storage-for-end-to-end-tests-temp-it-flight_delays__59.None-c37 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:19.628Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupTempFilesByDestination/Reify, through flatten WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/Flatten/Unzipped-1, into producer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/Map(<lambda at fileio.py:584>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:19.655Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/Map(<lambda at fileio.py:584>) into WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/ParDo(_WriteShardedRecordsFn) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:19.681Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupTempFilesByDestination/Reify into WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/Map(<lambda at fileio.py:584>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:19.707Z: JOB_MESSAGE_DETAILED: Fusing consumer read table/Read/Map(<lambda at iobase.py:898>) into read table/Read/Impulse [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:19.732Z: JOB_MESSAGE_DETAILED: Fusing consumer ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13/PairWithRestriction into read table/Read/Map(<lambda at iobase.py:898>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:19.790Z: JOB_MESSAGE_DETAILED: Fusing consumer ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13/SplitWithSizing into ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13/PairWithRestriction [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:19.817Z: JOB_MESSAGE_DETAILED: Fusing consumer read table/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough) into ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13/ProcessElementAndRestrictionWithSizing [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:19.852Z: JOB_MESSAGE_DETAILED: Fusing consumer read table/FilesToRemoveImpulse/FlatMap(<lambda at core.py:3228>) into read table/FilesToRemoveImpulse/Impulse [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:19.885Z: JOB_MESSAGE_DETAILED: Fusing consumer read table/FilesToRemoveImpulse/Map(decode) into read table/FilesToRemoveImpulse/FlatMap(<lambda at core.py:3228>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:19.920Z: JOB_MESSAGE_DETAILED: Fusing consumer read table/MapFilesToRemove into read table/FilesToRemoveImpulse/Map(decode) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:19.945Z: JOB_MESSAGE_DETAILED: Fusing consumer assign timestamp into read table/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:19.979Z: JOB_MESSAGE_DETAILED: Fusing consumer set schema/Map(<lambda at core.py:2836>) into assign timestamp [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:20.015Z: JOB_MESSAGE_DETAILED: Fusing consumer daily windows into set schema/Map(<lambda at core.py:2836>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:20.051Z: JOB_MESSAGE_DETAILED: Fusing consumer BatchElements(daily)/BatchElements/ParDo(_WindowAwareBatchingDoFn) into daily windows [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:20.089Z: JOB_MESSAGE_DETAILED: Fusing consumer BatchElements(daily)/Map(<lambda at schemas.py:140>) into BatchElements(daily)/BatchElements/ParDo(_WindowAwareBatchingDoFn) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:20.121Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[move_grouped_columns_to_index_DataFrame_140226962656176]]:140226958734288/Map(<lambda at transforms.py:244>) into BatchElements(daily)/Map(<lambda at schemas.py:140>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:20.155Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[move_grouped_columns_to_index_DataFrame_140226962656176]]:140226958734288/FlatMap(evaluate)/FlatMap(evaluate) into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[move_grouped_columns_to_index_DataFrame_140226962656176]]:140226958734288/Map(<lambda at transforms.py:244>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:20.188Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_140226962656176/ParDo(_PreBatch) into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[move_grouped_columns_to_index_DataFrame_140226962656176]]:140226958734288/FlatMap(evaluate)/FlatMap(evaluate) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:20.224Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_140226962656176/FlatMap(partition_fn) into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_140226962656176/ParDo(_PreBatch) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:20.257Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_140226962656176/Filter(<lambda at transforms.py:230>) into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_140226962656176/FlatMap(partition_fn) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:20.292Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/Tag[0] into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_140226962656176/Filter(<lambda at transforms.py:230>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:20.315Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/Flatten/OutputIdentity into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/Tag[0] [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:20.348Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Reify into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/Flatten/OutputIdentity [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:20.394Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Write into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Reify [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:20.428Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/GroupByWindow into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Read [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:20.453Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/MapTuple(collect_values) into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/GroupByWindow [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:20.484Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/RestoreTags into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/MapTuple(collect_values) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:20.509Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/ParDo(_ReBatch) into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/RestoreTags [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:20.539Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/FlatMap(evaluate)/FlatMap(evaluate) into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/ParDo(_ReBatch) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:20.572Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/ParDo(_WriteUnshardedRecordsFn)/ParDo(_WriteUnshardedRecordsFn) into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/FlatMap(evaluate)/FlatMap(evaluate) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:20.604Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/ParDo(_AppendShardedDestination) into WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/ParDo(_WriteUnshardedRecordsFn)/ParDo(_WriteUnshardedRecordsFn) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:20.640Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Reify into WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/ParDo(_AppendShardedDestination) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:20.696Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Write into WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Reify [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:20.729Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/GroupByWindow into WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Read [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:20.786Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/ParDo(_WriteShardedRecordsFn) into WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/GroupByWindow [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:20.821Z: JOB_MESSAGE_DETAILED: Fusing consumer read table/_PassThroughThenCleanup/Create/FlatMap(<lambda at core.py:3228>) into read table/_PassThroughThenCleanup/Create/Impulse [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:20.855Z: JOB_MESSAGE_DETAILED: Fusing consumer read table/_PassThroughThenCleanup/Create/Map(decode) into read table/_PassThroughThenCleanup/Create/FlatMap(<lambda at core.py:3228>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:20.888Z: JOB_MESSAGE_DETAILED: Fusing consumer read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles) into read table/_PassThroughThenCleanup/Create/Map(decode) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:20.923Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:20.957Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:20.992Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:21.023Z: JOB_MESSAGE_DEBUG: Assigning stage ids. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:21.178Z: JOB_MESSAGE_DEBUG: Executing wait step start53 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:21.262Z: JOB_MESSAGE_BASIC: Executing operation read table/FilesToRemoveImpulse/Impulse+read table/FilesToRemoveImpulse/FlatMap(<lambda at core.py:3228>)+read table/FilesToRemoveImpulse/Map(decode)+read table/MapFilesToRemove [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:21.284Z: JOB_MESSAGE_BASIC: Executing operation read table/Read/Impulse+read table/Read/Map(<lambda at iobase.py:898>)+ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13/PairWithRestriction+ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13/SplitWithSizing [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:21.309Z: JOB_MESSAGE_DEBUG: Starting worker pool setup. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:21.315Z: JOB_MESSAGE_BASIC: Executing operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:21.338Z: JOB_MESSAGE_BASIC: Starting 1 workers in us-central1-b... [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:21.349Z: JOB_MESSAGE_BASIC: Executing operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:21.376Z: JOB_MESSAGE_BASIC: Executing operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupTempFilesByDestination/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:21.413Z: JOB_MESSAGE_BASIC: Finished operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:21.413Z: JOB_MESSAGE_BASIC: Finished operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:21.433Z: JOB_MESSAGE_BASIC: Finished operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupTempFilesByDestination/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:21.487Z: JOB_MESSAGE_DEBUG: Value "ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Session" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:21.520Z: JOB_MESSAGE_DEBUG: Value "WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Session" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:21.551Z: JOB_MESSAGE_DEBUG: Value "WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupTempFilesByDestination/Session" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:27:29.788Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:28:04.982Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running stage(s). [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:28:20.098Z: JOB_MESSAGE_DETAILED: Workers have started successfully. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:28:20.123Z: JOB_MESSAGE_DETAILED: Workers have started successfully. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:02.309Z: JOB_MESSAGE_BASIC: Finished operation read table/FilesToRemoveImpulse/Impulse+read table/FilesToRemoveImpulse/FlatMap(<lambda at core.py:3228>)+read table/FilesToRemoveImpulse/Map(decode)+read table/MapFilesToRemove [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:02.364Z: JOB_MESSAGE_DEBUG: Value "read table/MapFilesToRemove.None" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:02.428Z: JOB_MESSAGE_BASIC: Executing operation read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)/View-python_side_input1-read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:02.486Z: JOB_MESSAGE_BASIC: Finished operation read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)/View-python_side_input1-read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:02.554Z: JOB_MESSAGE_DEBUG: Value "read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)/View-python_side_input1-read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles).out" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:15.586Z: JOB_MESSAGE_BASIC: Finished operation read table/Read/Impulse+read table/Read/Map(<lambda at iobase.py:898>)+ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13/PairWithRestriction+ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13/SplitWithSizing [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:15.648Z: JOB_MESSAGE_DEBUG: Value "ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13-split-with-sizing-out3" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:15.724Z: JOB_MESSAGE_BASIC: Executing operation ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13/ProcessElementAndRestrictionWithSizing+read table/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough)+assign timestamp+set schema/Map(<lambda at core.py:2836>)+daily windows+BatchElements(daily)/BatchElements/ParDo(_WindowAwareBatchingDoFn)+BatchElements(daily)/Map(<lambda at schemas.py:140>)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[move_grouped_columns_to_index_DataFrame_140226962656176]]:140226958734288/Map(<lambda at transforms.py:244>)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[move_grouped_columns_to_index_DataFrame_140226962656176]]:140226958734288/FlatMap(evaluate)/FlatMap(evaluate)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_140226962656176/ParDo(_PreBatch)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_140226962656176/FlatMap(partition_fn)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_140226962656176/Filter(<lambda at transforms.py:230>)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/Tag[0]+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/Flatten/OutputIdentity+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Reify+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:28.132Z: JOB_MESSAGE_BASIC: Finished operation ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13/ProcessElementAndRestrictionWithSizing+read table/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough)+assign timestamp+set schema/Map(<lambda at core.py:2836>)+daily windows+BatchElements(daily)/BatchElements/ParDo(_WindowAwareBatchingDoFn)+BatchElements(daily)/Map(<lambda at schemas.py:140>)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[move_grouped_columns_to_index_DataFrame_140226962656176]]:140226958734288/Map(<lambda at transforms.py:244>)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[move_grouped_columns_to_index_DataFrame_140226962656176]]:140226958734288/FlatMap(evaluate)/FlatMap(evaluate)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_140226962656176/ParDo(_PreBatch)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_140226962656176/FlatMap(partition_fn)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_140226962656176/Filter(<lambda at transforms.py:230>)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/Tag[0]+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/Flatten/OutputIdentity+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Reify+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:28.203Z: JOB_MESSAGE_DEBUG: Value "read table/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough).cleanup_signal" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:28.237Z: JOB_MESSAGE_BASIC: Executing operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:28.271Z: JOB_MESSAGE_BASIC: Executing operation read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)/View-python_side_input0-read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:28.289Z: JOB_MESSAGE_BASIC: Finished operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:28.340Z: JOB_MESSAGE_BASIC: Finished operation read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)/View-python_side_input0-read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:28.353Z: JOB_MESSAGE_BASIC: Executing operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Read+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/GroupByWindow+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/MapTuple(collect_values)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/RestoreTags+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/ParDo(_ReBatch)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/FlatMap(evaluate)/FlatMap(evaluate)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/ParDo(_WriteUnshardedRecordsFn)/ParDo(_WriteUnshardedRecordsFn)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/Map(<lambda at fileio.py:584>)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupTempFilesByDestination/Reify+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupTempFilesByDestination/Write+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/ParDo(_AppendShardedDestination)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Reify+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:28.411Z: JOB_MESSAGE_DEBUG: Value "read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)/View-python_side_input0-read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles).out" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:28.479Z: JOB_MESSAGE_BASIC: Executing operation read table/_PassThroughThenCleanup/Create/Impulse+read table/_PassThroughThenCleanup/Create/FlatMap(<lambda at core.py:3228>)+read table/_PassThroughThenCleanup/Create/Map(decode)+read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:31.167Z: JOB_MESSAGE_BASIC: Finished operation read table/_PassThroughThenCleanup/Create/Impulse+read table/_PassThroughThenCleanup/Create/FlatMap(<lambda at core.py:3228>)+read table/_PassThroughThenCleanup/Create/Map(decode)+read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:31.167Z: JOB_MESSAGE_BASIC: Finished operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Read+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/GroupByWindow+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/CoGroupByKeyImpl/MapTuple(collect_values)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/CoGroupByKey/RestoreTags+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/ParDo(_ReBatch)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/[ComputedExpression[apply_DataFrame_140226962552096]]:140226962522848/FlatMap(evaluate)/FlatMap(evaluate)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/ParDo(_WriteUnshardedRecordsFn)/ParDo(_WriteUnshardedRecordsFn)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/Map(<lambda at fileio.py:584>)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupTempFilesByDestination/Reify+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupTempFilesByDestination/Write+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/ParDo(_AppendShardedDestination)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Reify+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:31.257Z: JOB_MESSAGE_BASIC: Executing operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:31.306Z: JOB_MESSAGE_BASIC: Finished operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:31.421Z: JOB_MESSAGE_BASIC: Executing operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Read+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/GroupByWindow+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/ParDo(_WriteShardedRecordsFn)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/Map(<lambda at fileio.py:584>)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupTempFilesByDestination/Reify+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupTempFilesByDestination/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:34.195Z: JOB_MESSAGE_BASIC: Finished operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Read+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/GroupByWindow+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/ParDo(_WriteShardedRecordsFn)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/Map(<lambda at fileio.py:584>)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupTempFilesByDestination/Reify+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupTempFilesByDestination/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:34.262Z: JOB_MESSAGE_BASIC: Executing operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupTempFilesByDestination/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:34.318Z: JOB_MESSAGE_BASIC: Finished operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupTempFilesByDestination/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:34.386Z: JOB_MESSAGE_BASIC: Executing operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupTempFilesByDestination/Read+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupTempFilesByDestination/GroupByWindow+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/ParDo(_MoveTempFilesIntoFinalDestinationFn) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:36.445Z: JOB_MESSAGE_BASIC: Finished operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupTempFilesByDestination/Read+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/GroupTempFilesByDestination/GroupByWindow+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-91188eee-81db-498d-804f-6a63a4e1b5af/output.csv/WriteToFiles/ParDo(_MoveTempFilesIntoFinalDestinationFn) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:36.516Z: JOB_MESSAGE_DEBUG: Executing success step success51 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:36.599Z: JOB_MESSAGE_DETAILED: Cleaning up. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:36.646Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:35:36.677Z: JOB_MESSAGE_BASIC: Stopping worker pool... [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:37:57.798Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized worker pool from 1 to 0. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:37:57.839Z: JOB_MESSAGE_BASIC: Worker pool stopped. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-10T00:37:57.890Z: JOB_MESSAGE_DEBUG: Tearing down pending resources... [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:191 Job 2022-02-09_16_27_07-15647133683677275877 is in state JOB_STATE_DONE [32mINFO [0m apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input [32mINFO [0m apache_beam.io.gcp.gcsio:gcsio.py:572 Finished listing 1 files in 0.07295346260070801 seconds. [32mINFO [0m apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input [32mINFO [0m apache_beam.io.gcp.gcsio:gcsio.py:572 Finished listing 3 files in 0.08525347709655762 seconds. [33m=============================== warnings summary ===============================[0m apache_beam/io/filesystems_test.py:54 apache_beam/io/filesystems_test.py:54 apache_beam/io/filesystems_test.py:54 apache_beam/io/filesystems_test.py:54 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:54: DeprecationWarning: invalid escape sequence \c self.assertIsNone(FileSystems.get_scheme('c:\\abc\cdf')) # pylint: disable=anomalous-backslash-in-string apache_beam/io/filesystems_test.py:62 apache_beam/io/filesystems_test.py:62 apache_beam/io/filesystems_test.py:62 apache_beam/io/filesystems_test.py:62 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:62: DeprecationWarning: invalid escape sequence \d self.assertTrue(isinstance(FileSystems.get_filesystem('c:\\abc\def'), # pylint: disable=anomalous-backslash-in-string <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead def call(self, fn, *args, **kwargs): apache_beam/io/gcp/bigquery.py:2437 apache_beam/io/gcp/bigquery.py:2437 apache_beam/io/gcp/bigquery.py:2437 apache_beam/io/gcp/bigquery.py:2437 apache_beam/io/gcp/bigquery.py:2437 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2437: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported temp_location = pcoll.pipeline.options.view_as( apache_beam/io/gcp/bigquery.py:2439 apache_beam/io/gcp/bigquery.py:2439 apache_beam/io/gcp/bigquery.py:2439 apache_beam/io/gcp/bigquery.py:2439 apache_beam/io/gcp/bigquery.py:2439 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2439: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name apache_beam/io/gcp/bigquery.py:2463 apache_beam/io/gcp/bigquery.py:2463 apache_beam/io/gcp/bigquery.py:2463 apache_beam/io/gcp/bigquery.py:2463 apache_beam/io/gcp/bigquery.py:2463 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2463: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported pipeline_options=pcoll.pipeline.options, apache_beam/io/gcp/tests/utils.py:63 apache_beam/io/gcp/tests/utils.py:63 apache_beam/io/gcp/tests/utils.py:63 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead. dataset_ref = client.dataset(unique_dataset_name, project=project) apache_beam/io/gcp/bigquery.py:2138 apache_beam/io/gcp/bigquery.py:2138 apache_beam/io/gcp/bigquery.py:2138 apache_beam/io/gcp/bigquery.py:2138 apache_beam/io/gcp/bigquery.py:2138 apache_beam/io/gcp/bigquery.py:2138 apache_beam/io/gcp/bigquery.py:2138 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2138: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported is_streaming_pipeline = p.options.view_as(StandardOptions).streaming apache_beam/io/gcp/bigquery.py:2144 apache_beam/io/gcp/bigquery.py:2144 apache_beam/io/gcp/bigquery.py:2144 apache_beam/io/gcp/bigquery.py:2144 apache_beam/io/gcp/bigquery.py:2144 apache_beam/io/gcp/bigquery.py:2144 apache_beam/io/gcp/bigquery.py:2144 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2144: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported experiments = p.options.view_as(DebugOptions).experiments or [] apache_beam/io/gcp/bigquery_file_loads.py:1128 apache_beam/io/gcp/bigquery_file_loads.py:1128 apache_beam/io/gcp/bigquery_file_loads.py:1128 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1128: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported temp_location = p.options.view_as(GoogleCloudOptions).temp_location apache_beam/io/gcp/bigquery_file_loads.py:1130 apache_beam/io/gcp/bigquery_file_loads.py:1130 apache_beam/io/gcp/bigquery_file_loads.py:1130 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1130: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME') apache_beam/io/gcp/bigquery.py:2134 apache_beam/io/gcp/bigquery.py:2134 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2134: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported self.table_reference.projectId = pcoll.pipeline.options.view_as( apache_beam/io/gcp/tests/utils.py:100 apache_beam/io/gcp/tests/utils.py:100 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:100: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead. table_ref = client.dataset(dataset_id).table(table_id) apache_beam/dataframe/io.py:629 apache_beam/dataframe/io.py:629 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/io.py>:629: FutureWarning: WriteToFiles is experimental. return pcoll | fileio.WriteToFiles( apache_beam/io/fileio.py:550 apache_beam/io/fileio.py:550 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/fileio.py>:550: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported p.options.view_as(GoogleCloudOptions).temp_location or apache_beam/examples/dataframe/flight_delays.py:45 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/examples/dataframe/flight_delays.py>:45: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError. Select only valid columns before calling the reduction. return airline_df[at_top_airports].mean() -- Docs: https://docs.pytest.org/en/latest/warnings.html - generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py38.xml> - [31m[1m======== 8 failed, 16 passed, 1 skipped, 63 warnings in 1448.47 seconds ========[0m > Task :sdks:python:test-suites:dataflow:py38:examples FAILED FAILURE: Build failed with an exception. * Where: Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 165 * What went wrong: Execution failed for task ':sdks:python:test-suites:dataflow:py38:examples'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Run with --scan to get full insights. * Get more help at https://help.gradle.org BUILD FAILED in 26m 11s 15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date Publishing build scan... https://gradle.com/s/iyumtq5dno6za Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
