See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/23/display/redirect?page=changes>
Changes: [noreply] [BEAM-13803] Add support for native iterable side inputs to the Go SDK [noreply] [BEAM-11095] Better error handling for illegal emit functions (#16776) [noreply] Merge pull request #16613 from Supporting JdbcIO driver in classpath for ------------------------------------------ [...truncated 79.19 KB...] [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:15.253Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:15.287Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:15.308Z: JOB_MESSAGE_DEBUG: Inserted coder converter after flatten ref_AppliedPTransform_ToPCollection-df---gs-temp-storage-for-end-to-end-tests-temp-it-flight_delays__44 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:15.330Z: JOB_MESSAGE_DETAILED: Elided trivial flatten [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:15.354Z: JOB_MESSAGE_DETAILED: Unzipping flatten ref_AppliedPTransform_WriteToPandas-df---gs-temp-storage-for-end-to-end-tests-temp-it-flight_delays__58 for input ref_AppliedPTransform_WriteToPandas-df---gs-temp-storage-for-end-to-end-tests-temp-it-flight_delays__54.written_files [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:15.389Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/Map(<lambda at fileio.py:584>), through flatten WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/Flatten, into producer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/ParDo(_WriteUnshardedRecordsFn)/ParDo(_WriteUnshardedRecordsFn) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:15.420Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupTempFilesByDestination/Write into WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupTempFilesByDestination/Reify [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:15.454Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupTempFilesByDestination/GroupByWindow into WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupTempFilesByDestination/Read [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:15.504Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/ParDo(_MoveTempFilesIntoFinalDestinationFn) into WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupTempFilesByDestination/GroupByWindow [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:15.547Z: JOB_MESSAGE_DETAILED: Unzipping flatten ref_AppliedPTransform_WriteToPandas-df---gs-temp-storage-for-end-to-end-tests-temp-it-flight_delays__58-u39 for input ref_AppliedPTransform_WriteToPandas-df---gs-temp-storage-for-end-to-end-tests-temp-it-flight_delays__59.None-c37 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:15.590Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupTempFilesByDestination/Reify, through flatten WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/Flatten/Unzipped-1, into producer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/Map(<lambda at fileio.py:584>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:15.611Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/Map(<lambda at fileio.py:584>) into WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/ParDo(_WriteShardedRecordsFn) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:15.643Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupTempFilesByDestination/Reify into WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/Map(<lambda at fileio.py:584>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:15.681Z: JOB_MESSAGE_DETAILED: Fusing consumer read table/Read/Map(<lambda at iobase.py:898>) into read table/Read/Impulse [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:15.720Z: JOB_MESSAGE_DETAILED: Fusing consumer ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13/PairWithRestriction into read table/Read/Map(<lambda at iobase.py:898>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:15.766Z: JOB_MESSAGE_DETAILED: Fusing consumer ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13/SplitWithSizing into ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13/PairWithRestriction [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:15.799Z: JOB_MESSAGE_DETAILED: Fusing consumer read table/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough) into ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13/ProcessElementAndRestrictionWithSizing [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:15.831Z: JOB_MESSAGE_DETAILED: Fusing consumer read table/FilesToRemoveImpulse/FlatMap(<lambda at core.py:3228>) into read table/FilesToRemoveImpulse/Impulse [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:15.863Z: JOB_MESSAGE_DETAILED: Fusing consumer read table/FilesToRemoveImpulse/Map(decode) into read table/FilesToRemoveImpulse/FlatMap(<lambda at core.py:3228>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:15.894Z: JOB_MESSAGE_DETAILED: Fusing consumer read table/MapFilesToRemove into read table/FilesToRemoveImpulse/Map(decode) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:15.922Z: JOB_MESSAGE_DETAILED: Fusing consumer assign timestamp into read table/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:15.959Z: JOB_MESSAGE_DETAILED: Fusing consumer set schema/Map(<lambda at core.py:2836>) into assign timestamp [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:15.986Z: JOB_MESSAGE_DETAILED: Fusing consumer daily windows into set schema/Map(<lambda at core.py:2836>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:16.015Z: JOB_MESSAGE_DETAILED: Fusing consumer BatchElements(daily)/BatchElements/ParDo(_WindowAwareBatchingDoFn) into daily windows [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:16.049Z: JOB_MESSAGE_DETAILED: Fusing consumer BatchElements(daily)/Map(<lambda at schemas.py:140>) into BatchElements(daily)/BatchElements/ParDo(_WindowAwareBatchingDoFn) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:16.072Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[move_grouped_columns_to_index_DataFrame_139794751338768]]:139794746983376/Map(<lambda at transforms.py:244>) into BatchElements(daily)/Map(<lambda at schemas.py:140>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:16.098Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[move_grouped_columns_to_index_DataFrame_139794751338768]]:139794746983376/FlatMap(evaluate)/FlatMap(evaluate) into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[move_grouped_columns_to_index_DataFrame_139794751338768]]:139794746983376/Map(<lambda at transforms.py:244>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:16.121Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_139794751338768/ParDo(_PreBatch) into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[move_grouped_columns_to_index_DataFrame_139794751338768]]:139794746983376/FlatMap(evaluate)/FlatMap(evaluate) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:16.163Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_139794751338768/FlatMap(partition_fn) into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_139794751338768/ParDo(_PreBatch) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:16.194Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_139794751338768/Filter(<lambda at transforms.py:230>) into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_139794751338768/FlatMap(partition_fn) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:16.226Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/Tag[0] into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_139794751338768/Filter(<lambda at transforms.py:230>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:16.260Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/Flatten/OutputIdentity into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/Tag[0] [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:16.294Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Reify into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/Flatten/OutputIdentity [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:16.326Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Write into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Reify [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:16.360Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/GroupByWindow into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Read [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:16.395Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/MapTuple(collect_values) into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/GroupByWindow [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:16.433Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/RestoreTags into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/MapTuple(collect_values) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:16.534Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/ParDo(_ReBatch) into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/RestoreTags [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:16.684Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/FlatMap(evaluate)/FlatMap(evaluate) into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/ParDo(_ReBatch) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:16.791Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/ParDo(_WriteUnshardedRecordsFn)/ParDo(_WriteUnshardedRecordsFn) into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/FlatMap(evaluate)/FlatMap(evaluate) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:16.899Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/ParDo(_AppendShardedDestination) into WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/ParDo(_WriteUnshardedRecordsFn)/ParDo(_WriteUnshardedRecordsFn) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:17.015Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Reify into WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/ParDo(_AppendShardedDestination) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:17.116Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Write into WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Reify [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:17.188Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/GroupByWindow into WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Read [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:17.248Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/ParDo(_WriteShardedRecordsFn) into WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/GroupByWindow [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:17.368Z: JOB_MESSAGE_DETAILED: Fusing consumer read table/_PassThroughThenCleanup/Create/FlatMap(<lambda at core.py:3228>) into read table/_PassThroughThenCleanup/Create/Impulse [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:17.507Z: JOB_MESSAGE_DETAILED: Fusing consumer read table/_PassThroughThenCleanup/Create/Map(decode) into read table/_PassThroughThenCleanup/Create/FlatMap(<lambda at core.py:3228>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:17.561Z: JOB_MESSAGE_DETAILED: Fusing consumer read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles) into read table/_PassThroughThenCleanup/Create/Map(decode) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:17.668Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:17.761Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:17.997Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:18.134Z: JOB_MESSAGE_DEBUG: Assigning stage ids. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:19.129Z: JOB_MESSAGE_DEBUG: Executing wait step start53 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:19.321Z: JOB_MESSAGE_BASIC: Executing operation read table/FilesToRemoveImpulse/Impulse+read table/FilesToRemoveImpulse/FlatMap(<lambda at core.py:3228>)+read table/FilesToRemoveImpulse/Map(decode)+read table/MapFilesToRemove [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:19.405Z: JOB_MESSAGE_BASIC: Executing operation read table/Read/Impulse+read table/Read/Map(<lambda at iobase.py:898>)+ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13/PairWithRestriction+ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13/SplitWithSizing [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:19.463Z: JOB_MESSAGE_DEBUG: Starting worker pool setup. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:19.509Z: JOB_MESSAGE_BASIC: Executing operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:19.558Z: JOB_MESSAGE_BASIC: Executing operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:19.570Z: JOB_MESSAGE_BASIC: Starting 1 workers in us-central1-b... [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:19.639Z: JOB_MESSAGE_BASIC: Executing operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupTempFilesByDestination/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:22.271Z: JOB_MESSAGE_BASIC: Finished operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:22.271Z: JOB_MESSAGE_BASIC: Finished operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:22.272Z: JOB_MESSAGE_BASIC: Finished operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupTempFilesByDestination/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:22.464Z: JOB_MESSAGE_DEBUG: Value "WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Session" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:22.508Z: JOB_MESSAGE_DEBUG: Value "ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Session" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:22.541Z: JOB_MESSAGE_DEBUG: Value "WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupTempFilesByDestination/Session" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:16:40.343Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:17:06.637Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running stage(s). [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:17:22.454Z: JOB_MESSAGE_DETAILED: Workers have started successfully. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:17:22.519Z: JOB_MESSAGE_DETAILED: Workers have started successfully. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:09.102Z: JOB_MESSAGE_BASIC: Finished operation read table/FilesToRemoveImpulse/Impulse+read table/FilesToRemoveImpulse/FlatMap(<lambda at core.py:3228>)+read table/FilesToRemoveImpulse/Map(decode)+read table/MapFilesToRemove [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:09.186Z: JOB_MESSAGE_DEBUG: Value "read table/MapFilesToRemove.None" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:09.243Z: JOB_MESSAGE_BASIC: Executing operation read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)/View-python_side_input1-read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:09.304Z: JOB_MESSAGE_BASIC: Finished operation read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)/View-python_side_input1-read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:09.367Z: JOB_MESSAGE_DEBUG: Value "read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)/View-python_side_input1-read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles).out" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:22.347Z: JOB_MESSAGE_BASIC: Finished operation read table/Read/Impulse+read table/Read/Map(<lambda at iobase.py:898>)+ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13/PairWithRestriction+ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13/SplitWithSizing [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:22.406Z: JOB_MESSAGE_DEBUG: Value "ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13-split-with-sizing-out3" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:22.472Z: JOB_MESSAGE_BASIC: Executing operation ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13/ProcessElementAndRestrictionWithSizing+read table/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough)+assign timestamp+set schema/Map(<lambda at core.py:2836>)+daily windows+BatchElements(daily)/BatchElements/ParDo(_WindowAwareBatchingDoFn)+BatchElements(daily)/Map(<lambda at schemas.py:140>)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[move_grouped_columns_to_index_DataFrame_139794751338768]]:139794746983376/Map(<lambda at transforms.py:244>)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[move_grouped_columns_to_index_DataFrame_139794751338768]]:139794746983376/FlatMap(evaluate)/FlatMap(evaluate)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_139794751338768/ParDo(_PreBatch)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_139794751338768/FlatMap(partition_fn)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_139794751338768/Filter(<lambda at transforms.py:230>)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/Tag[0]+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/Flatten/OutputIdentity+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Reify+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:34.282Z: JOB_MESSAGE_BASIC: Finished operation ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13/ProcessElementAndRestrictionWithSizing+read table/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough)+assign timestamp+set schema/Map(<lambda at core.py:2836>)+daily windows+BatchElements(daily)/BatchElements/ParDo(_WindowAwareBatchingDoFn)+BatchElements(daily)/Map(<lambda at schemas.py:140>)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[move_grouped_columns_to_index_DataFrame_139794751338768]]:139794746983376/Map(<lambda at transforms.py:244>)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[move_grouped_columns_to_index_DataFrame_139794751338768]]:139794746983376/FlatMap(evaluate)/FlatMap(evaluate)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_139794751338768/ParDo(_PreBatch)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_139794751338768/FlatMap(partition_fn)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_139794751338768/Filter(<lambda at transforms.py:230>)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/Tag[0]+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/Flatten/OutputIdentity+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Reify+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:34.333Z: JOB_MESSAGE_DEBUG: Value "read table/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough).cleanup_signal" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:34.368Z: JOB_MESSAGE_BASIC: Executing operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:34.384Z: JOB_MESSAGE_BASIC: Executing operation read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)/View-python_side_input0-read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:34.412Z: JOB_MESSAGE_BASIC: Finished operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:34.421Z: JOB_MESSAGE_BASIC: Finished operation read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)/View-python_side_input0-read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:34.465Z: JOB_MESSAGE_BASIC: Executing operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Read+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/GroupByWindow+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/MapTuple(collect_values)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/RestoreTags+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/ParDo(_ReBatch)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/FlatMap(evaluate)/FlatMap(evaluate)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/ParDo(_WriteUnshardedRecordsFn)/ParDo(_WriteUnshardedRecordsFn)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/Map(<lambda at fileio.py:584>)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupTempFilesByDestination/Reify+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupTempFilesByDestination/Write+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/ParDo(_AppendShardedDestination)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Reify+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:34.489Z: JOB_MESSAGE_DEBUG: Value "read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)/View-python_side_input0-read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles).out" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:34.546Z: JOB_MESSAGE_BASIC: Executing operation read table/_PassThroughThenCleanup/Create/Impulse+read table/_PassThroughThenCleanup/Create/FlatMap(<lambda at core.py:3228>)+read table/_PassThroughThenCleanup/Create/Map(decode)+read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:36.502Z: JOB_MESSAGE_BASIC: Finished operation read table/_PassThroughThenCleanup/Create/Impulse+read table/_PassThroughThenCleanup/Create/FlatMap(<lambda at core.py:3228>)+read table/_PassThroughThenCleanup/Create/Map(decode)+read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:37.586Z: JOB_MESSAGE_BASIC: Finished operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Read+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/GroupByWindow+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/CoGroupByKeyImpl/MapTuple(collect_values)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/CoGroupByKey/RestoreTags+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/ParDo(_ReBatch)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/[ComputedExpression[apply_DataFrame_139794751448016]]:139794750335968/FlatMap(evaluate)/FlatMap(evaluate)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/ParDo(_WriteUnshardedRecordsFn)/ParDo(_WriteUnshardedRecordsFn)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/Map(<lambda at fileio.py:584>)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupTempFilesByDestination/Reify+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupTempFilesByDestination/Write+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/ParDo(_AppendShardedDestination)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Reify+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:37.684Z: JOB_MESSAGE_BASIC: Executing operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:37.722Z: JOB_MESSAGE_BASIC: Finished operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:37.782Z: JOB_MESSAGE_BASIC: Executing operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Read+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/GroupByWindow+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/ParDo(_WriteShardedRecordsFn)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/Map(<lambda at fileio.py:584>)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupTempFilesByDestination/Reify+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupTempFilesByDestination/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:39.561Z: JOB_MESSAGE_BASIC: Finished operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Read+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/GroupByWindow+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/ParDo(_WriteShardedRecordsFn)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/Map(<lambda at fileio.py:584>)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupTempFilesByDestination/Reify+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupTempFilesByDestination/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:39.658Z: JOB_MESSAGE_BASIC: Executing operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupTempFilesByDestination/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:39.709Z: JOB_MESSAGE_BASIC: Finished operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupTempFilesByDestination/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:39.759Z: JOB_MESSAGE_BASIC: Executing operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupTempFilesByDestination/Read+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupTempFilesByDestination/GroupByWindow+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/ParDo(_MoveTempFilesIntoFinalDestinationFn) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:40.827Z: JOB_MESSAGE_BASIC: Finished operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupTempFilesByDestination/Read+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/GroupTempFilesByDestination/GroupByWindow+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-802fe321-d4c3-4880-81ec-9d0ced6fa1ef/output.csv/WriteToFiles/ParDo(_MoveTempFilesIntoFinalDestinationFn) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:41.034Z: JOB_MESSAGE_DEBUG: Executing success step success51 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:41.256Z: JOB_MESSAGE_DETAILED: Cleaning up. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:41.385Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:24:41.412Z: JOB_MESSAGE_BASIC: Stopping worker pool... [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:26:59.863Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized worker pool from 1 to 0. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:26:59.917Z: JOB_MESSAGE_BASIC: Worker pool stopped. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T18:26:59.939Z: JOB_MESSAGE_DEBUG: Tearing down pending resources... [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:191 Job 2022-02-09_10_15_56-15573241249444342865 is in state JOB_STATE_DONE [32mINFO [0m apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input [32mINFO [0m apache_beam.io.gcp.gcsio:gcsio.py:572 Finished listing 1 files in 0.03798389434814453 seconds. [32mINFO [0m apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input [32mINFO [0m apache_beam.io.gcp.gcsio:gcsio.py:572 Finished listing 3 files in 0.02964615821838379 seconds. [33m=============================== warnings summary ===============================[0m apache_beam/io/filesystems_test.py:54 apache_beam/io/filesystems_test.py:54 apache_beam/io/filesystems_test.py:54 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:54: DeprecationWarning: invalid escape sequence \c self.assertIsNone(FileSystems.get_scheme('c:\\abc\cdf')) # pylint: disable=anomalous-backslash-in-string apache_beam/io/filesystems_test.py:62 apache_beam/io/filesystems_test.py:62 apache_beam/io/filesystems_test.py:62 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:62: DeprecationWarning: invalid escape sequence \d self.assertTrue(isinstance(FileSystems.get_filesystem('c:\\abc\def'), # pylint: disable=anomalous-backslash-in-string <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead def call(self, fn, *args, **kwargs): apache_beam/io/gcp/bigquery.py:2437 apache_beam/io/gcp/bigquery.py:2437 apache_beam/io/gcp/bigquery.py:2437 apache_beam/io/gcp/bigquery.py:2437 apache_beam/io/gcp/bigquery.py:2437 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2437: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported temp_location = pcoll.pipeline.options.view_as( apache_beam/io/gcp/bigquery.py:2439 apache_beam/io/gcp/bigquery.py:2439 apache_beam/io/gcp/bigquery.py:2439 apache_beam/io/gcp/bigquery.py:2439 apache_beam/io/gcp/bigquery.py:2439 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2439: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name apache_beam/io/gcp/bigquery.py:2463 apache_beam/io/gcp/bigquery.py:2463 apache_beam/io/gcp/bigquery.py:2463 apache_beam/io/gcp/bigquery.py:2463 apache_beam/io/gcp/bigquery.py:2463 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2463: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported pipeline_options=pcoll.pipeline.options, apache_beam/io/gcp/tests/utils.py:63 apache_beam/io/gcp/tests/utils.py:63 apache_beam/io/gcp/tests/utils.py:63 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead. dataset_ref = client.dataset(unique_dataset_name, project=project) apache_beam/io/gcp/bigquery.py:2138 apache_beam/io/gcp/bigquery.py:2138 apache_beam/io/gcp/bigquery.py:2138 apache_beam/io/gcp/bigquery.py:2138 apache_beam/io/gcp/bigquery.py:2138 apache_beam/io/gcp/bigquery.py:2138 apache_beam/io/gcp/bigquery.py:2138 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2138: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported is_streaming_pipeline = p.options.view_as(StandardOptions).streaming apache_beam/io/gcp/bigquery.py:2144 apache_beam/io/gcp/bigquery.py:2144 apache_beam/io/gcp/bigquery.py:2144 apache_beam/io/gcp/bigquery.py:2144 apache_beam/io/gcp/bigquery.py:2144 apache_beam/io/gcp/bigquery.py:2144 apache_beam/io/gcp/bigquery.py:2144 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2144: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported experiments = p.options.view_as(DebugOptions).experiments or [] apache_beam/io/gcp/bigquery_file_loads.py:1128 apache_beam/io/gcp/bigquery_file_loads.py:1128 apache_beam/io/gcp/bigquery_file_loads.py:1128 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1128: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported temp_location = p.options.view_as(GoogleCloudOptions).temp_location apache_beam/io/gcp/bigquery_file_loads.py:1130 apache_beam/io/gcp/bigquery_file_loads.py:1130 apache_beam/io/gcp/bigquery_file_loads.py:1130 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1130: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME') apache_beam/dataframe/io.py:629 apache_beam/dataframe/io.py:629 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/io.py>:629: FutureWarning: WriteToFiles is experimental. return pcoll | fileio.WriteToFiles( apache_beam/io/fileio.py:550 apache_beam/io/fileio.py:550 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/fileio.py>:550: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported p.options.view_as(GoogleCloudOptions).temp_location or apache_beam/io/gcp/bigquery.py:2134 apache_beam/io/gcp/bigquery.py:2134 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2134: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported self.table_reference.projectId = pcoll.pipeline.options.view_as( apache_beam/io/gcp/tests/utils.py:100 apache_beam/io/gcp/tests/utils.py:100 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:100: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead. table_ref = client.dataset(dataset_id).table(table_id) apache_beam/examples/dataframe/flight_delays.py:45 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/examples/dataframe/flight_delays.py>:45: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError. Select only valid columns before calling the reduction. return airline_df[at_top_airports].mean() -- Docs: https://docs.pytest.org/en/latest/warnings.html - generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py38.xml> - [31m[1m======== 8 failed, 16 passed, 1 skipped, 61 warnings in 1430.15 seconds ========[0m > Task :sdks:python:test-suites:dataflow:py38:examples FAILED FAILURE: Build failed with an exception. * Where: Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 165 * What went wrong: Execution failed for task ':sdks:python:test-suites:dataflow:py38:examples'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Run with --scan to get full insights. * Get more help at https://help.gradle.org BUILD FAILED in 26m 17s 15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date Publishing build scan... https://gradle.com/s/mfxcysjnlvbqy Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
