See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/21/display/redirect?page=changes>
Changes: [Kyle Weaver] Update Dataflow Python dev container images. [Kiley Sok] Add java 17 to changes [Daniel Oliveira] [BEAM-13732] Switch x-lang BigQueryIO expansion service to GCP one. [noreply] [BEAM-13858] Fix broken github action on :sdks:go:examples:wordCount [Kiley Sok] add jira for runner v2 [noreply] [BEAM-13732] Go SDK BigQuery IO wrapper. Initial implementation. [noreply] [BEAM-13732] Add example for Go BigQuery IO wrapper. (#16786) [noreply] Update CHANGES.md with Go SDK milestones. (#16787) [noreply] [BEAM-13193] Allow BeamFnDataOutboundObserver to flush elements. ------------------------------------------ [...truncated 78.52 KB...] [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:19.600Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/GroupByKey: GroupByKey not followed by a combiner. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:19.634Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:19.661Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:19.704Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:19.727Z: JOB_MESSAGE_DEBUG: Inserted coder converter after flatten ref_AppliedPTransform_ToPCollection-df---gs-temp-storage-for-end-to-end-tests-temp-it-flight_delays__44 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:19.752Z: JOB_MESSAGE_DETAILED: Elided trivial flatten [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:19.777Z: JOB_MESSAGE_DETAILED: Unzipping flatten ref_AppliedPTransform_WriteToPandas-df---gs-temp-storage-for-end-to-end-tests-temp-it-flight_delays__58 for input ref_AppliedPTransform_WriteToPandas-df---gs-temp-storage-for-end-to-end-tests-temp-it-flight_delays__54.written_files [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:19.810Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/Map(<lambda at fileio.py:584>), through flatten WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/Flatten, into producer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/ParDo(_WriteUnshardedRecordsFn)/ParDo(_WriteUnshardedRecordsFn) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:19.843Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupTempFilesByDestination/Write into WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupTempFilesByDestination/Reify [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:19.874Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupTempFilesByDestination/GroupByWindow into WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupTempFilesByDestination/Read [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:19.908Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/ParDo(_MoveTempFilesIntoFinalDestinationFn) into WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupTempFilesByDestination/GroupByWindow [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:19.940Z: JOB_MESSAGE_DETAILED: Unzipping flatten ref_AppliedPTransform_WriteToPandas-df---gs-temp-storage-for-end-to-end-tests-temp-it-flight_delays__58-u39 for input ref_AppliedPTransform_WriteToPandas-df---gs-temp-storage-for-end-to-end-tests-temp-it-flight_delays__59.None-c37 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:19.971Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupTempFilesByDestination/Reify, through flatten WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/Flatten/Unzipped-1, into producer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/Map(<lambda at fileio.py:584>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.003Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/Map(<lambda at fileio.py:584>) into WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/ParDo(_WriteShardedRecordsFn) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.025Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupTempFilesByDestination/Reify into WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/Map(<lambda at fileio.py:584>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.058Z: JOB_MESSAGE_DETAILED: Fusing consumer read table/Read/Map(<lambda at iobase.py:898>) into read table/Read/Impulse [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.081Z: JOB_MESSAGE_DETAILED: Fusing consumer ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13/PairWithRestriction into read table/Read/Map(<lambda at iobase.py:898>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.110Z: JOB_MESSAGE_DETAILED: Fusing consumer ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13/SplitWithSizing into ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13/PairWithRestriction [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.132Z: JOB_MESSAGE_DETAILED: Fusing consumer read table/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough) into ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13/ProcessElementAndRestrictionWithSizing [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.164Z: JOB_MESSAGE_DETAILED: Fusing consumer read table/FilesToRemoveImpulse/FlatMap(<lambda at core.py:3228>) into read table/FilesToRemoveImpulse/Impulse [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.199Z: JOB_MESSAGE_DETAILED: Fusing consumer read table/FilesToRemoveImpulse/Map(decode) into read table/FilesToRemoveImpulse/FlatMap(<lambda at core.py:3228>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.233Z: JOB_MESSAGE_DETAILED: Fusing consumer read table/MapFilesToRemove into read table/FilesToRemoveImpulse/Map(decode) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.252Z: JOB_MESSAGE_DETAILED: Fusing consumer assign timestamp into read table/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.275Z: JOB_MESSAGE_DETAILED: Fusing consumer set schema/Map(<lambda at core.py:2836>) into assign timestamp [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.306Z: JOB_MESSAGE_DETAILED: Fusing consumer daily windows into set schema/Map(<lambda at core.py:2836>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.340Z: JOB_MESSAGE_DETAILED: Fusing consumer BatchElements(daily)/BatchElements/ParDo(_WindowAwareBatchingDoFn) into daily windows [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.373Z: JOB_MESSAGE_DETAILED: Fusing consumer BatchElements(daily)/Map(<lambda at schemas.py:140>) into BatchElements(daily)/BatchElements/ParDo(_WindowAwareBatchingDoFn) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.411Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[move_grouped_columns_to_index_DataFrame_139896156867984]]:139896156738608/Map(<lambda at transforms.py:244>) into BatchElements(daily)/Map(<lambda at schemas.py:140>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.436Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[move_grouped_columns_to_index_DataFrame_139896156867984]]:139896156738608/FlatMap(evaluate)/FlatMap(evaluate) into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[move_grouped_columns_to_index_DataFrame_139896156867984]]:139896156738608/Map(<lambda at transforms.py:244>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.469Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_139896156867984/ParDo(_PreBatch) into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[move_grouped_columns_to_index_DataFrame_139896156867984]]:139896156738608/FlatMap(evaluate)/FlatMap(evaluate) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.503Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_139896156867984/FlatMap(partition_fn) into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_139896156867984/ParDo(_PreBatch) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.539Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_139896156867984/Filter(<lambda at transforms.py:230>) into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_139896156867984/FlatMap(partition_fn) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.602Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/Tag[0] into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_139896156867984/Filter(<lambda at transforms.py:230>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.631Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/Flatten/OutputIdentity into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/Tag[0] [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.666Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Reify into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/Flatten/OutputIdentity [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.699Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Write into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Reify [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.733Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/GroupByWindow into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Read [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.764Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/MapTuple(collect_values) into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/GroupByWindow [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.797Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/RestoreTags into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/MapTuple(collect_values) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.830Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/ParDo(_ReBatch) into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/RestoreTags [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.854Z: JOB_MESSAGE_DETAILED: Fusing consumer ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/FlatMap(evaluate)/FlatMap(evaluate) into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/ParDo(_ReBatch) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.886Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/ParDo(_WriteUnshardedRecordsFn)/ParDo(_WriteUnshardedRecordsFn) into ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/FlatMap(evaluate)/FlatMap(evaluate) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.917Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/ParDo(_AppendShardedDestination) into WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/ParDo(_WriteUnshardedRecordsFn)/ParDo(_WriteUnshardedRecordsFn) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.960Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Reify into WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/ParDo(_AppendShardedDestination) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:20.984Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Write into WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Reify [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:21.015Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/GroupByWindow into WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Read [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:21.048Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/ParDo(_WriteShardedRecordsFn) into WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/GroupByWindow [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:21.081Z: JOB_MESSAGE_DETAILED: Fusing consumer read table/_PassThroughThenCleanup/Create/FlatMap(<lambda at core.py:3228>) into read table/_PassThroughThenCleanup/Create/Impulse [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:21.113Z: JOB_MESSAGE_DETAILED: Fusing consumer read table/_PassThroughThenCleanup/Create/Map(decode) into read table/_PassThroughThenCleanup/Create/FlatMap(<lambda at core.py:3228>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:21.138Z: JOB_MESSAGE_DETAILED: Fusing consumer read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles) into read table/_PassThroughThenCleanup/Create/Map(decode) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:21.166Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:21.203Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:21.234Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:21.266Z: JOB_MESSAGE_DEBUG: Assigning stage ids. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:21.405Z: JOB_MESSAGE_DEBUG: Executing wait step start53 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:21.471Z: JOB_MESSAGE_BASIC: Executing operation read table/FilesToRemoveImpulse/Impulse+read table/FilesToRemoveImpulse/FlatMap(<lambda at core.py:3228>)+read table/FilesToRemoveImpulse/Map(decode)+read table/MapFilesToRemove [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:21.504Z: JOB_MESSAGE_BASIC: Executing operation read table/Read/Impulse+read table/Read/Map(<lambda at iobase.py:898>)+ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13/PairWithRestriction+ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13/SplitWithSizing [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:21.516Z: JOB_MESSAGE_DEBUG: Starting worker pool setup. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:21.527Z: JOB_MESSAGE_BASIC: Executing operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:21.538Z: JOB_MESSAGE_BASIC: Starting 1 workers in us-central1-b... [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:21.560Z: JOB_MESSAGE_BASIC: Executing operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:21.590Z: JOB_MESSAGE_BASIC: Finished operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:21.613Z: JOB_MESSAGE_BASIC: Executing operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupTempFilesByDestination/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:21.632Z: JOB_MESSAGE_BASIC: Finished operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:21.661Z: JOB_MESSAGE_BASIC: Finished operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupTempFilesByDestination/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:21.665Z: JOB_MESSAGE_DEBUG: Value "ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Session" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:21.700Z: JOB_MESSAGE_DEBUG: Value "WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Session" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:21.732Z: JOB_MESSAGE_DEBUG: Value "WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupTempFilesByDestination/Session" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:03:40.077Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:06:52.783Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running stage(s). [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:07:03.965Z: JOB_MESSAGE_DETAILED: Workers have started successfully. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:07:03.989Z: JOB_MESSAGE_DETAILED: Workers have started successfully. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:13:34.243Z: JOB_MESSAGE_BASIC: Finished operation read table/FilesToRemoveImpulse/Impulse+read table/FilesToRemoveImpulse/FlatMap(<lambda at core.py:3228>)+read table/FilesToRemoveImpulse/Map(decode)+read table/MapFilesToRemove [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:13:34.332Z: JOB_MESSAGE_DEBUG: Value "read table/MapFilesToRemove.None" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:13:34.429Z: JOB_MESSAGE_BASIC: Executing operation read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)/View-python_side_input1-read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:13:34.491Z: JOB_MESSAGE_BASIC: Finished operation read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)/View-python_side_input1-read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:13:34.540Z: JOB_MESSAGE_DEBUG: Value "read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)/View-python_side_input1-read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles).out" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:13:47.080Z: JOB_MESSAGE_BASIC: Finished operation read table/Read/Impulse+read table/Read/Map(<lambda at iobase.py:898>)+ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13/PairWithRestriction+ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13/SplitWithSizing [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:13:47.135Z: JOB_MESSAGE_DEBUG: Value "ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13-split-with-sizing-out3" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:13:47.193Z: JOB_MESSAGE_BASIC: Executing operation ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13/ProcessElementAndRestrictionWithSizing+read table/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough)+assign timestamp+set schema/Map(<lambda at core.py:2836>)+daily windows+BatchElements(daily)/BatchElements/ParDo(_WindowAwareBatchingDoFn)+BatchElements(daily)/Map(<lambda at schemas.py:140>)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[move_grouped_columns_to_index_DataFrame_139896156867984]]:139896156738608/Map(<lambda at transforms.py:244>)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[move_grouped_columns_to_index_DataFrame_139896156867984]]:139896156738608/FlatMap(evaluate)/FlatMap(evaluate)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_139896156867984/ParDo(_PreBatch)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_139896156867984/FlatMap(partition_fn)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_139896156867984/Filter(<lambda at transforms.py:230>)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/Tag[0]+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/Flatten/OutputIdentity+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Reify+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:13:59.636Z: JOB_MESSAGE_BASIC: Finished operation ref_AppliedPTransform_read-table-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_13/ProcessElementAndRestrictionWithSizing+read table/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough)+assign timestamp+set schema/Map(<lambda at core.py:2836>)+daily windows+BatchElements(daily)/BatchElements/ParDo(_WindowAwareBatchingDoFn)+BatchElements(daily)/Map(<lambda at schemas.py:140>)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[move_grouped_columns_to_index_DataFrame_139896156867984]]:139896156738608/Map(<lambda at transforms.py:244>)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[move_grouped_columns_to_index_DataFrame_139896156867984]]:139896156738608/FlatMap(evaluate)/FlatMap(evaluate)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_139896156867984/ParDo(_PreBatch)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_139896156867984/FlatMap(partition_fn)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/Partition_Index['airline']_move_grouped_columns_to_index_DataFrame_139896156867984/Filter(<lambda at transforms.py:230>)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/Tag[0]+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/Flatten/OutputIdentity+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Reify+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:13:59.716Z: JOB_MESSAGE_DEBUG: Value "read table/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough).cleanup_signal" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:13:59.741Z: JOB_MESSAGE_BASIC: Executing operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:13:59.775Z: JOB_MESSAGE_BASIC: Executing operation read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)/View-python_side_input0-read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:13:59.792Z: JOB_MESSAGE_BASIC: Finished operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:13:59.829Z: JOB_MESSAGE_BASIC: Finished operation read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)/View-python_side_input0-read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:13:59.854Z: JOB_MESSAGE_BASIC: Executing operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Read+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/GroupByWindow+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/MapTuple(collect_values)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/RestoreTags+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/ParDo(_ReBatch)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/FlatMap(evaluate)/FlatMap(evaluate)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/ParDo(_WriteUnshardedRecordsFn)/ParDo(_WriteUnshardedRecordsFn)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/Map(<lambda at fileio.py:584>)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupTempFilesByDestination/Reify+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupTempFilesByDestination/Write+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/ParDo(_AppendShardedDestination)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Reify+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:13:59.887Z: JOB_MESSAGE_DEBUG: Value "read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles)/View-python_side_input0-read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles).out" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:13:59.949Z: JOB_MESSAGE_BASIC: Executing operation read table/_PassThroughThenCleanup/Create/Impulse+read table/_PassThroughThenCleanup/Create/FlatMap(<lambda at core.py:3228>)+read table/_PassThroughThenCleanup/Create/Map(decode)+read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:14:02.731Z: JOB_MESSAGE_BASIC: Finished operation read table/_PassThroughThenCleanup/Create/Impulse+read table/_PassThroughThenCleanup/Create/FlatMap(<lambda at core.py:3228>)+read table/_PassThroughThenCleanup/Create/Map(decode)+read table/_PassThroughThenCleanup/ParDo(RemoveExtractedFiles) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:14:02.827Z: JOB_MESSAGE_BASIC: Finished operation ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/Read+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/GroupByKey/GroupByWindow+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/CoGroupByKeyImpl/MapTuple(collect_values)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/CoGroupByKey/RestoreTags+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/ParDo(_ReBatch)+ToPCollection(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/[ComputedExpression[apply_DataFrame_139896156829632]]:139896156738464/FlatMap(evaluate)/FlatMap(evaluate)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/ParDo(_WriteUnshardedRecordsFn)/ParDo(_WriteUnshardedRecordsFn)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/Map(<lambda at fileio.py:584>)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupTempFilesByDestination/Reify+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupTempFilesByDestination/Write+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/ParDo(_AppendShardedDestination)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Reify+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:14:02.874Z: JOB_MESSAGE_BASIC: Executing operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:14:02.915Z: JOB_MESSAGE_BASIC: Finished operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:14:02.969Z: JOB_MESSAGE_BASIC: Executing operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Read+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/GroupByWindow+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/ParDo(_WriteShardedRecordsFn)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/Map(<lambda at fileio.py:584>)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupTempFilesByDestination/Reify+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupTempFilesByDestination/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:14:05.773Z: JOB_MESSAGE_BASIC: Finished operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/Read+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupRecordsByDestinationAndShard/GroupByWindow+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/ParDo(_WriteShardedRecordsFn)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/Map(<lambda at fileio.py:584>)+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupTempFilesByDestination/Reify+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupTempFilesByDestination/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:14:05.841Z: JOB_MESSAGE_BASIC: Executing operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupTempFilesByDestination/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:14:05.883Z: JOB_MESSAGE_BASIC: Finished operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupTempFilesByDestination/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:14:05.955Z: JOB_MESSAGE_BASIC: Executing operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupTempFilesByDestination/Read+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupTempFilesByDestination/GroupByWindow+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/ParDo(_MoveTempFilesIntoFinalDestinationFn) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:14:08.001Z: JOB_MESSAGE_BASIC: Finished operation WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupTempFilesByDestination/Read+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/GroupTempFilesByDestination/GroupByWindow+WriteToPandas(df) - gs://temp-storage-for-end-to-end-tests/temp-it/flight_delays_it-ff1cfd14-fc4f-486c-86c7-25efc28273dd/output.csv/WriteToFiles/ParDo(_MoveTempFilesIntoFinalDestinationFn) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:14:08.063Z: JOB_MESSAGE_DEBUG: Executing success step success51 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:14:08.141Z: JOB_MESSAGE_DETAILED: Cleaning up. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:14:08.186Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:14:08.218Z: JOB_MESSAGE_BASIC: Stopping worker pool... [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:18:20.515Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized worker pool from 1 to 0. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:18:20.562Z: JOB_MESSAGE_BASIC: Worker pool stopped. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2022-02-09T06:18:20.593Z: JOB_MESSAGE_DEBUG: Tearing down pending resources... [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:191 Job 2022-02-08_22_03_08-5394103083123072525 is in state JOB_STATE_DONE [32mINFO [0m apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input [32mINFO [0m apache_beam.io.gcp.gcsio:gcsio.py:572 Finished listing 1 files in 0.04675722122192383 seconds. [32mINFO [0m apache_beam.io.gcp.gcsio:gcsio.py:559 Starting the size estimation of the input [32mINFO [0m apache_beam.io.gcp.gcsio:gcsio.py:572 Finished listing 3 files in 0.03428149223327637 seconds. [33m=============================== warnings summary ===============================[0m <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead def call(self, fn, *args, **kwargs): apache_beam/io/filesystems_test.py:54 apache_beam/io/filesystems_test.py:54 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:54: DeprecationWarning: invalid escape sequence \c self.assertIsNone(FileSystems.get_scheme('c:\\abc\cdf')) # pylint: disable=anomalous-backslash-in-string apache_beam/io/filesystems_test.py:62 apache_beam/io/filesystems_test.py:62 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:62: DeprecationWarning: invalid escape sequence \d self.assertTrue(isinstance(FileSystems.get_filesystem('c:\\abc\def'), # pylint: disable=anomalous-backslash-in-string apache_beam/io/gcp/bigquery.py:2437 apache_beam/io/gcp/bigquery.py:2437 apache_beam/io/gcp/bigquery.py:2437 apache_beam/io/gcp/bigquery.py:2437 apache_beam/io/gcp/bigquery.py:2437 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2437: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported temp_location = pcoll.pipeline.options.view_as( apache_beam/io/gcp/bigquery.py:2439 apache_beam/io/gcp/bigquery.py:2439 apache_beam/io/gcp/bigquery.py:2439 apache_beam/io/gcp/bigquery.py:2439 apache_beam/io/gcp/bigquery.py:2439 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2439: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name apache_beam/io/gcp/bigquery.py:2463 apache_beam/io/gcp/bigquery.py:2463 apache_beam/io/gcp/bigquery.py:2463 apache_beam/io/gcp/bigquery.py:2463 apache_beam/io/gcp/bigquery.py:2463 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2463: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported pipeline_options=pcoll.pipeline.options, apache_beam/io/gcp/tests/utils.py:63 apache_beam/io/gcp/tests/utils.py:63 apache_beam/io/gcp/tests/utils.py:63 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead. dataset_ref = client.dataset(unique_dataset_name, project=project) apache_beam/io/gcp/bigquery.py:2138 apache_beam/io/gcp/bigquery.py:2138 apache_beam/io/gcp/bigquery.py:2138 apache_beam/io/gcp/bigquery.py:2138 apache_beam/io/gcp/bigquery.py:2138 apache_beam/io/gcp/bigquery.py:2138 apache_beam/io/gcp/bigquery.py:2138 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2138: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported is_streaming_pipeline = p.options.view_as(StandardOptions).streaming apache_beam/io/gcp/bigquery.py:2144 apache_beam/io/gcp/bigquery.py:2144 apache_beam/io/gcp/bigquery.py:2144 apache_beam/io/gcp/bigquery.py:2144 apache_beam/io/gcp/bigquery.py:2144 apache_beam/io/gcp/bigquery.py:2144 apache_beam/io/gcp/bigquery.py:2144 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2144: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported experiments = p.options.view_as(DebugOptions).experiments or [] apache_beam/io/gcp/bigquery_file_loads.py:1128 apache_beam/io/gcp/bigquery_file_loads.py:1128 apache_beam/io/gcp/bigquery_file_loads.py:1128 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1128: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported temp_location = p.options.view_as(GoogleCloudOptions).temp_location apache_beam/io/gcp/bigquery_file_loads.py:1130 apache_beam/io/gcp/bigquery_file_loads.py:1130 apache_beam/io/gcp/bigquery_file_loads.py:1130 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1130: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME') apache_beam/io/gcp/bigquery.py:2134 apache_beam/io/gcp/bigquery.py:2134 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2134: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported self.table_reference.projectId = pcoll.pipeline.options.view_as( apache_beam/io/gcp/tests/utils.py:100 apache_beam/io/gcp/tests/utils.py:100 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:100: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead. table_ref = client.dataset(dataset_id).table(table_id) apache_beam/dataframe/io.py:629 apache_beam/dataframe/io.py:629 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/io.py>:629: FutureWarning: WriteToFiles is experimental. return pcoll | fileio.WriteToFiles( apache_beam/io/fileio.py:550 apache_beam/io/fileio.py:550 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/fileio.py>:550: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported p.options.view_as(GoogleCloudOptions).temp_location or apache_beam/examples/dataframe/flight_delays.py:45 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/examples/dataframe/flight_delays.py>:45: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError. Select only valid columns before calling the reduction. return airline_df[at_top_airports].mean() -- Docs: https://docs.pytest.org/en/latest/warnings.html - generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py38.xml> - [31m[1m======== 8 failed, 16 passed, 1 skipped, 59 warnings in 1001.60 seconds ========[0m > Task :sdks:python:test-suites:dataflow:py38:examples FAILED FAILURE: Build failed with an exception. * Where: Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 165 * What went wrong: Execution failed for task ':sdks:python:test-suites:dataflow:py38:examples'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Run with --scan to get full insights. * Get more help at https://help.gradle.org BUILD FAILED in 18m 32s 15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date Publishing build scan... https://gradle.com/s/vwew4shgyf47q Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
