See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/823/display/redirect?page=changes>
Changes: [Brian Hulette] Extract utilities in dataframe.schemas [Brian Hulette] Add pandas_type_compatibility with pandas BatchConverter implementations [Brian Hulette] Use Batched DoFns at DataFrame API boundaries [Brian Hulette] Move dtype conversion to pandas_type_compatibility [Brian Hulette] Always register pandas BatchConverters [Brian Hulette] Fix interactive runner tests [Brian Hulette] Use pandas_type_compatibility BatchConverters for dataframe.schemas [Brian Hulette] Skip test cases broken in pandas 1.1.x [Brian Hulette] Address review comments [Brian Hulette] yapf, typo in test [Brian Hulette] Add test to reproduce https://github.com/apache/beam/issues/22854 [Brian Hulette] Exercise row coder with nested optional struct [Brian Hulette] Make RowTypeConstraint callable [Brian Hulette] Add test to exercise RowTypeConstraint.__call__ [noreply] Run cred rotation every month (#22977) [noreply] [BEAM-12164] Synchronize access queue in ThroughputEstimator and ------------------------------------------ [...truncated 63.42 KB...] [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:28.506Z: JOB_MESSAGE_DETAILED: Fusing consumer CalculateSpammyUsers/CombineGlobally(MeanCombineFn)/CombinePerKey/Combine into CalculateSpammyUsers/CombineGlobally(MeanCombineFn)/CombinePerKey/GroupByKey/ReadStream [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:28.548Z: JOB_MESSAGE_DETAILED: Fusing consumer CalculateSpammyUsers/CombineGlobally(MeanCombineFn)/CombinePerKey/Combine/Extract into CalculateSpammyUsers/CombineGlobally(MeanCombineFn)/CombinePerKey/Combine [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:28.594Z: JOB_MESSAGE_DETAILED: Fusing consumer CalculateSpammyUsers/CombineGlobally(MeanCombineFn)/UnKey into CalculateSpammyUsers/CombineGlobally(MeanCombineFn)/CombinePerKey/Combine/Extract [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:28.626Z: JOB_MESSAGE_DETAILED: Fusing consumer CalculateSpammyUsers/ProcessAndFilter/_DataflowIterableSideInput(UnKey.out.0)/PairWithVoidKey into CalculateSpammyUsers/CombineGlobally(MeanCombineFn)/UnKey [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:28.658Z: JOB_MESSAGE_DETAILED: Fusing consumer CalculateSpammyUsers/ProcessAndFilter/_DataflowIterableSideInput(UnKey.out.0)/GroupByKey/WriteStream into CalculateSpammyUsers/ProcessAndFilter/_DataflowIterableSideInput(UnKey.out.0)/PairWithVoidKey [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:28.694Z: JOB_MESSAGE_DETAILED: Fusing consumer CalculateSpammyUsers/ProcessAndFilter/_DataflowIterableSideInput(UnKey.out.0)/GroupByKey/MergeBuckets into CalculateSpammyUsers/ProcessAndFilter/_DataflowIterableSideInput(UnKey.out.0)/GroupByKey/ReadStream [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:28.737Z: JOB_MESSAGE_DETAILED: Fusing consumer CalculateSpammyUsers/ProcessAndFilter/_DataflowIterableSideInput(UnKey.out.0)/Values into CalculateSpammyUsers/ProcessAndFilter/_DataflowIterableSideInput(UnKey.out.0)/GroupByKey/MergeBuckets [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:28.763Z: JOB_MESSAGE_DETAILED: Fusing consumer CalculateSpammyUsers/ProcessAndFilter/_DataflowIterableSideInput(UnKey.out.0)/StreamingPCollectionViewWriter into CalculateSpammyUsers/ProcessAndFilter/_DataflowIterableSideInput(UnKey.out.0)/Values [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:28.842Z: JOB_MESSAGE_DETAILED: Fusing consumer CreateSpammersView/KeyWithVoid into CalculateSpammyUsers/ProcessAndFilter/ProcessAndFilter [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:28.920Z: JOB_MESSAGE_DETAILED: Fusing consumer CreateSpammersView/CombinePerKey/Combine/ConvertToAccumulators into CreateSpammersView/KeyWithVoid [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:28.956Z: JOB_MESSAGE_DETAILED: Fusing consumer CreateSpammersView/CombinePerKey/GroupByKey/WriteStream into CreateSpammersView/CombinePerKey/Combine/ConvertToAccumulators [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:28.992Z: JOB_MESSAGE_DETAILED: Fusing consumer CreateSpammersView/CombinePerKey/Combine into CreateSpammersView/CombinePerKey/GroupByKey/ReadStream [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:29.025Z: JOB_MESSAGE_DETAILED: Fusing consumer CreateSpammersView/CombinePerKey/Combine/Extract into CreateSpammersView/CombinePerKey/Combine [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:29.059Z: JOB_MESSAGE_DETAILED: Fusing consumer CreateSpammersView/UnKey into CreateSpammersView/CombinePerKey/Combine/Extract [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:29.092Z: JOB_MESSAGE_DETAILED: Fusing consumer FilterOutSpammers/_DataflowIterableSideInput(UnKey.out.0)/PairWithVoidKey into CreateSpammersView/UnKey [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:29.114Z: JOB_MESSAGE_DETAILED: Fusing consumer FilterOutSpammers/_DataflowIterableSideInput(UnKey.out.0)/GroupByKey/WriteStream into FilterOutSpammers/_DataflowIterableSideInput(UnKey.out.0)/PairWithVoidKey [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:29.148Z: JOB_MESSAGE_DETAILED: Fusing consumer FilterOutSpammers/_DataflowIterableSideInput(UnKey.out.0)/GroupByKey/MergeBuckets into FilterOutSpammers/_DataflowIterableSideInput(UnKey.out.0)/GroupByKey/ReadStream [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:29.186Z: JOB_MESSAGE_DETAILED: Fusing consumer FilterOutSpammers/_DataflowIterableSideInput(UnKey.out.0)/Values into FilterOutSpammers/_DataflowIterableSideInput(UnKey.out.0)/GroupByKey/MergeBuckets [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:29.289Z: JOB_MESSAGE_DETAILED: Fusing consumer FilterOutSpammers/_DataflowIterableSideInput(UnKey.out.0)/StreamingPCollectionViewWriter into FilterOutSpammers/_DataflowIterableSideInput(UnKey.out.0)/Values [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:29.324Z: JOB_MESSAGE_DETAILED: Fusing consumer ExtractAndSumScore/Map(<lambda at game_stats.py:142>) into FilterOutSpammers/FilterOutSpammers [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:29.358Z: JOB_MESSAGE_DETAILED: Fusing consumer ExtractAndSumScore/CombinePerKey(sum)/Combine/ConvertToAccumulators into ExtractAndSumScore/Map(<lambda at game_stats.py:142>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:29.394Z: JOB_MESSAGE_DETAILED: Fusing consumer ExtractAndSumScore/CombinePerKey(sum)/GroupByKey/WriteStream into ExtractAndSumScore/CombinePerKey(sum)/Combine/ConvertToAccumulators [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:29.420Z: JOB_MESSAGE_DETAILED: Fusing consumer ExtractAndSumScore/CombinePerKey(sum)/Combine into ExtractAndSumScore/CombinePerKey(sum)/GroupByKey/ReadStream [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:29.451Z: JOB_MESSAGE_DETAILED: Fusing consumer ExtractAndSumScore/CombinePerKey(sum)/Combine/Extract into ExtractAndSumScore/CombinePerKey(sum)/Combine [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:29.489Z: JOB_MESSAGE_DETAILED: Fusing consumer TeamScoresDict into ExtractAndSumScore/CombinePerKey(sum)/Combine/Extract [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:29.525Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteTeamScoreSums/ConvertToRow into TeamScoresDict [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:29.571Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/AppendDestination into WriteTeamScoreSums/ConvertToRow [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:29.600Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/AddInsertIds into WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/AppendDestination [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:29.637Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/ToHashableTableRef into WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/AddInsertIds [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:29.671Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/WithFixedSharding into WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/ToHashableTableRef [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:29.730Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/Map(reify_timestamps) into WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/WithFixedSharding [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:29.769Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/GroupByKey/WriteStream into WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/Map(reify_timestamps) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:29.853Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/GroupByKey/MergeBuckets into WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/GroupByKey/ReadStream [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:29.913Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/FlatMap(restore_timestamps) into WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/GroupByKey/MergeBuckets [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:29.943Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/DropShard into WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/FlatMap(restore_timestamps) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:29.976Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/FromHashableTableRef into WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/DropShard [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:30.008Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/StreamInsertRows/ParDo(BigQueryWriteFn) into WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/FromHashableTableRef [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:30.040Z: JOB_MESSAGE_DETAILED: Fusing consumer CombinePerKey(<lambda at game_stats.py:381>)/Combine/ConvertToAccumulators into WindowIntoSessions [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:30.062Z: JOB_MESSAGE_DETAILED: Fusing consumer CombinePerKey(<lambda at game_stats.py:381>)/GroupByKey/WriteStream into CombinePerKey(<lambda at game_stats.py:381>)/Combine/ConvertToAccumulators [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:30.089Z: JOB_MESSAGE_DETAILED: Fusing consumer CombinePerKey(<lambda at game_stats.py:381>)/Combine into CombinePerKey(<lambda at game_stats.py:381>)/GroupByKey/ReadStream [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:30.127Z: JOB_MESSAGE_DETAILED: Fusing consumer CombinePerKey(<lambda at game_stats.py:381>)/Combine/Extract into CombinePerKey(<lambda at game_stats.py:381>)/Combine [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:30.160Z: JOB_MESSAGE_DETAILED: Fusing consumer UserSessionActivity into CombinePerKey(<lambda at game_stats.py:381>)/Combine/Extract [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:30.198Z: JOB_MESSAGE_DETAILED: Fusing consumer WindowToExtractSessionMean into UserSessionActivity [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:30.290Z: JOB_MESSAGE_DETAILED: Fusing consumer CombineGlobally(MeanCombineFn)/KeyWithVoid into WindowToExtractSessionMean [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:30.325Z: JOB_MESSAGE_DETAILED: Fusing consumer CombineGlobally(MeanCombineFn)/CombinePerKey/Combine/ConvertToAccumulators into CombineGlobally(MeanCombineFn)/KeyWithVoid [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:30.361Z: JOB_MESSAGE_DETAILED: Fusing consumer CombineGlobally(MeanCombineFn)/CombinePerKey/GroupByKey/WriteStream into CombineGlobally(MeanCombineFn)/CombinePerKey/Combine/ConvertToAccumulators [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:30.401Z: JOB_MESSAGE_DETAILED: Fusing consumer CombineGlobally(MeanCombineFn)/CombinePerKey/Combine into CombineGlobally(MeanCombineFn)/CombinePerKey/GroupByKey/ReadStream [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:30.428Z: JOB_MESSAGE_DETAILED: Fusing consumer CombineGlobally(MeanCombineFn)/CombinePerKey/Combine/Extract into CombineGlobally(MeanCombineFn)/CombinePerKey/Combine [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:30.467Z: JOB_MESSAGE_DETAILED: Fusing consumer CombineGlobally(MeanCombineFn)/UnKey into CombineGlobally(MeanCombineFn)/CombinePerKey/Combine/Extract [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:30.493Z: JOB_MESSAGE_DETAILED: Fusing consumer FormatAvgSessionLength into CombineGlobally(MeanCombineFn)/UnKey [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:30.527Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteAvgSessionLength/ConvertToRow into FormatAvgSessionLength [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:30.557Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/AppendDestination into WriteAvgSessionLength/ConvertToRow [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:30.594Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/AddInsertIds into WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/AppendDestination [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:30.620Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/ToHashableTableRef into WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/AddInsertIds [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:30.646Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/WithFixedSharding into WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/ToHashableTableRef [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:30.677Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/Map(reify_timestamps) into WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/WithFixedSharding [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:30.733Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/GroupByKey/WriteStream into WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/Map(reify_timestamps) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:30.771Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/GroupByKey/MergeBuckets into WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/GroupByKey/ReadStream [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:30.853Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/FlatMap(restore_timestamps) into WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/GroupByKey/MergeBuckets [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:30.916Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/DropShard into WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/FlatMap(restore_timestamps) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:30.944Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/FromHashableTableRef into WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/DropShard [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:30.980Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/StreamInsertRows/ParDo(BigQueryWriteFn) into WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/FromHashableTableRef [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:31.149Z: JOB_MESSAGE_BASIC: Running job using Streaming Engine [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:31.177Z: JOB_MESSAGE_BASIC: Using cloud KMLS key to protect persistent state. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:31.294Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:31.327Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:31.375Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:31.412Z: JOB_MESSAGE_DEBUG: Assigning stage ids. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:31.485Z: JOB_MESSAGE_DEBUG: Starting worker pool setup. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:31.517Z: JOB_MESSAGE_DEBUG: Starting worker pool setup. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:31.549Z: JOB_MESSAGE_BASIC: Starting 1 workers in us-central1-a... [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:58:51.331Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:59:08.062Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 1 so that the pipeline can catch up with its backlog and keep up with its input rate. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-09-01T00:59:39.813Z: JOB_MESSAGE_DETAILED: Workers have started successfully. [33mWARNING [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:253 Timing out on waiting for job 2022-08-31_17_58_16-2248431719162601638 after 720 seconds [32mINFO [0m apache_beam.io.gcp.tests.bigquery_matcher:bigquery_matcher.py:121 Attempting to perform query SELECT mean_duration FROM `apache-beam-testing.game_stats_it_dataset16619938147514.game_stats_sessions` WHERE mean_duration=300 LIMIT 1 to BQ [32mINFO [0m apache_beam.io.gcp.tests.bigquery_matcher:bigquery_matcher.py:96 Read from given query (SELECT mean_duration FROM `apache-beam-testing.game_stats_it_dataset16619938147514.game_stats_sessions` WHERE mean_duration=300 LIMIT 1), total rows 1 [32mINFO [0m apache_beam.io.gcp.tests.bigquery_matcher:bigquery_matcher.py:101 Generate checksum: 5288ccaab77d347c8460d77c15a0db234ef5eb4f [31m[1mERROR [0m apache_beam.runners.dataflow.test_dataflow_runner:test_dataflow_runner.py:102 Console URL: https://console.cloud.google.com/dataflow/<regionId>/2022-08-31_17_58_16-2248431719162601638?project=<projectId> [33m=============================== warnings summary ===============================[0m ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses from imp import load_source ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead def call(self, fn, *args, **kwargs): apache_beam/typehints/pandas_type_compatibility_test.py:66 apache_beam/typehints/pandas_type_compatibility_test.py:66 apache_beam/typehints/pandas_type_compatibility_test.py:66 apache_beam/typehints/pandas_type_compatibility_test.py:66 apache_beam/typehints/pandas_type_compatibility_test.py:66 apache_beam/typehints/pandas_type_compatibility_test.py:66 apache_beam/typehints/pandas_type_compatibility_test.py:66 apache_beam/typehints/pandas_type_compatibility_test.py:66 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:66: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead. }).set_index(pd.Int64Index(range(123, 223), name='an_index')), apache_beam/typehints/pandas_type_compatibility_test.py:89 apache_beam/typehints/pandas_type_compatibility_test.py:89 apache_beam/typehints/pandas_type_compatibility_test.py:89 apache_beam/typehints/pandas_type_compatibility_test.py:89 apache_beam/typehints/pandas_type_compatibility_test.py:89 apache_beam/typehints/pandas_type_compatibility_test.py:89 apache_beam/typehints/pandas_type_compatibility_test.py:89 apache_beam/typehints/pandas_type_compatibility_test.py:89 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:89: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead. pd.Int64Index(range(123, 223), name='an_index'), apache_beam/typehints/pandas_type_compatibility_test.py:90 apache_beam/typehints/pandas_type_compatibility_test.py:90 apache_beam/typehints/pandas_type_compatibility_test.py:90 apache_beam/typehints/pandas_type_compatibility_test.py:90 apache_beam/typehints/pandas_type_compatibility_test.py:90 apache_beam/typehints/pandas_type_compatibility_test.py:90 apache_beam/typehints/pandas_type_compatibility_test.py:90 apache_beam/typehints/pandas_type_compatibility_test.py:90 <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead. pd.Int64Index(range(475, 575), name='another_index'), apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/dataframe/io.py>:632: FutureWarning: WriteToFiles is experimental. return pcoll | fileio.WriteToFiles( apache_beam/examples/dataframe/wordcount_test.py::WordCountTest::test_basics apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/fileio.py>:590: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported p.options.view_as(GoogleCloudOptions).temp_location or apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2774: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported temp_location = pcoll.pipeline.options.view_as( apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2776: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported job_name = pcoll.pipeline.options.view_as(GoogleCloudOptions).job_name apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2800: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported pipeline_options=pcoll.pipeline.options, apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2281: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported self.table_reference.projectId = pcoll.pipeline.options.view_as( apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2285: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported is_streaming_pipeline = p.options.view_as(StandardOptions).streaming apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:2291: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported experiments = p.options.view_as(DebugOptions).experiments or [] apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1129: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported temp_location = p.options.view_as(GoogleCloudOptions).temp_location apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:1131: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported p.options.view_as(GoogleCloudOptions).job_name or 'AUTOMATIC_JOB_NAME') apache_beam/examples/cookbook/filters_test.py::FiltersTest::test_filters_output_bigquery_matcher apache_beam/examples/cookbook/bigquery_tornadoes_it_test.py::BigqueryTornadoesIT::test_bigquery_tornadoes_it <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:100: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead. table_ref = client.dataset(dataset_id).table(table_id) apache_beam/examples/complete/game/leader_board_it_test.py::LeaderBoardIT::test_leader_board_it apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/tests/utils.py>:63: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead. dataset_ref = client.dataset(unique_dataset_name, project=project) apache_beam/examples/dataframe/flight_delays_it_test.py::FlightDelaysTest::test_flight_delays <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/apache_beam/examples/dataframe/flight_delays.py>:47: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError. Select only valid columns before calling the reduction. return airline_df[at_top_airports].mean() -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html - generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/pytest_postCommitIT-df-py39-xdist.xml> - =========================== short test summary info ============================ FAILED apache_beam/examples/complete/game/game_stats_it_test.py::GameStatsIT::test_game_stats_it [31m====== [31m[1m1 failed[0m, [32m16 passed[0m, [33m3 skipped[0m, [33m77 warnings[0m[31m in 1529.49s (0:25:29)[0m[31m =======[0m > Task :sdks:python:test-suites:dataflow:py39:examples FAILED FAILURE: Build failed with an exception. * Where: Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 187 * What went wrong: Execution failed for task ':sdks:python:test-suites:dataflow:py39:examples'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0. You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins. See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 26m 46s 15 actionable tasks: 9 executed, 4 from cache, 2 up-to-date Publishing build scan... https://gradle.com/s/gy3bkffhfl2ii Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
