See <https://builds.apache.org/job/beam_PostCommit_Python37/1251/display/redirect?page=changes>
Changes: [udim] [BEAM-9012] Change __init__ hints so they work with pytype (#10466) ------------------------------------------ [...truncated 3.05 MB...] apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.085Z: JOB_MESSAGE_DETAILED: Fusing consumer ParseGameEventFn into DecodeString apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.087Z: JOB_MESSAGE_DETAILED: Fusing consumer AddEventTimestamps into ParseGameEventFn apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.090Z: JOB_MESSAGE_DETAILED: Fusing consumer CalculateSpammyUsers/SumUsersScores/GroupByKey/WriteStream into UserFixedWindows apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.092Z: JOB_MESSAGE_DETAILED: Fusing consumer CalculateSpammyUsers/SumUsersScores/GroupByKey/MergeBuckets into CalculateSpammyUsers/SumUsersScores/GroupByKey/ReadStream apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.095Z: JOB_MESSAGE_DETAILED: Fusing consumer CalculateSpammyUsers/SumUsersScores/Combine into CalculateSpammyUsers/SumUsersScores/GroupByKey/MergeBuckets apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.097Z: JOB_MESSAGE_DETAILED: Fusing consumer CalculateSpammyUsers/CombineGlobally(MeanCombineFn)/KeyWithVoid into CalculateSpammyUsers/Values apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.099Z: JOB_MESSAGE_DETAILED: Fusing consumer CalculateSpammyUsers/CombineGlobally(MeanCombineFn)/CombinePerKey/GroupByKey/WriteStream into CalculateSpammyUsers/CombineGlobally(MeanCombineFn)/KeyWithVoid apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.101Z: JOB_MESSAGE_DETAILED: Fusing consumer CalculateSpammyUsers/CombineGlobally(MeanCombineFn)/CombinePerKey/GroupByKey/MergeBuckets into CalculateSpammyUsers/CombineGlobally(MeanCombineFn)/CombinePerKey/GroupByKey/ReadStream apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.103Z: JOB_MESSAGE_DETAILED: Fusing consumer CalculateSpammyUsers/CombineGlobally(MeanCombineFn)/CombinePerKey/Combine into CalculateSpammyUsers/CombineGlobally(MeanCombineFn)/CombinePerKey/GroupByKey/MergeBuckets apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.106Z: JOB_MESSAGE_DETAILED: Fusing consumer CalculateSpammyUsers/CombineGlobally(MeanCombineFn)/UnKey into CalculateSpammyUsers/CombineGlobally(MeanCombineFn)/CombinePerKey/Combine apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.108Z: JOB_MESSAGE_DETAILED: Fusing consumer CalculateSpammyUsers/ProcessAndFilter/_UnpickledSideInput(MapToVoidKey0.out.0)/PairWithVoidKey into CalculateSpammyUsers/ProcessAndFilter/MapToVoidKey0 apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.110Z: JOB_MESSAGE_DETAILED: Fusing consumer CalculateSpammyUsers/ProcessAndFilter/_UnpickledSideInput(MapToVoidKey0.out.0)/GroupByKey/WriteStream into CalculateSpammyUsers/ProcessAndFilter/_UnpickledSideInput(MapToVoidKey0.out.0)/PairWithVoidKey apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.113Z: JOB_MESSAGE_DETAILED: Fusing consumer CalculateSpammyUsers/ProcessAndFilter/_UnpickledSideInput(MapToVoidKey0.out.0)/GroupByKey/MergeBuckets into CalculateSpammyUsers/ProcessAndFilter/_UnpickledSideInput(MapToVoidKey0.out.0)/GroupByKey/ReadStream apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.115Z: JOB_MESSAGE_DETAILED: Fusing consumer CalculateSpammyUsers/ProcessAndFilter/_UnpickledSideInput(MapToVoidKey0.out.0)/Values into CalculateSpammyUsers/ProcessAndFilter/_UnpickledSideInput(MapToVoidKey0.out.0)/GroupByKey/MergeBuckets apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.118Z: JOB_MESSAGE_DETAILED: Fusing consumer CalculateSpammyUsers/ProcessAndFilter/_UnpickledSideInput(MapToVoidKey0.out.0)/StreamingPCollectionViewWriter into CalculateSpammyUsers/ProcessAndFilter/_UnpickledSideInput(MapToVoidKey0.out.0)/Values apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.120Z: JOB_MESSAGE_DETAILED: Fusing consumer CreateSpammersView/KeyWithVoid into CalculateSpammyUsers/ProcessAndFilter/ProcessAndFilter apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.122Z: JOB_MESSAGE_DETAILED: Fusing consumer CreateSpammersView/CombinePerKey/GroupByKey/WriteStream into CreateSpammersView/KeyWithVoid apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.124Z: JOB_MESSAGE_DETAILED: Fusing consumer CreateSpammersView/CombinePerKey/GroupByKey/MergeBuckets into CreateSpammersView/CombinePerKey/GroupByKey/ReadStream apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.127Z: JOB_MESSAGE_DETAILED: Fusing consumer CreateSpammersView/CombinePerKey/Combine into CreateSpammersView/CombinePerKey/GroupByKey/MergeBuckets apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.129Z: JOB_MESSAGE_DETAILED: Fusing consumer CreateSpammersView/UnKey into CreateSpammersView/CombinePerKey/Combine apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.132Z: JOB_MESSAGE_DETAILED: Fusing consumer FilterOutSpammers/_UnpickledSideInput(MapToVoidKey0.out.0)/PairWithVoidKey into FilterOutSpammers/MapToVoidKey0 apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.134Z: JOB_MESSAGE_DETAILED: Fusing consumer FilterOutSpammers/_UnpickledSideInput(MapToVoidKey0.out.0)/GroupByKey/WriteStream into FilterOutSpammers/_UnpickledSideInput(MapToVoidKey0.out.0)/PairWithVoidKey apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.136Z: JOB_MESSAGE_DETAILED: Fusing consumer FilterOutSpammers/_UnpickledSideInput(MapToVoidKey0.out.0)/GroupByKey/MergeBuckets into FilterOutSpammers/_UnpickledSideInput(MapToVoidKey0.out.0)/GroupByKey/ReadStream apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.138Z: JOB_MESSAGE_DETAILED: Fusing consumer FilterOutSpammers/_UnpickledSideInput(MapToVoidKey0.out.0)/Values into FilterOutSpammers/_UnpickledSideInput(MapToVoidKey0.out.0)/GroupByKey/MergeBuckets apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.140Z: JOB_MESSAGE_DETAILED: Fusing consumer FilterOutSpammers/_UnpickledSideInput(MapToVoidKey0.out.0)/StreamingPCollectionViewWriter into FilterOutSpammers/_UnpickledSideInput(MapToVoidKey0.out.0)/Values apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.142Z: JOB_MESSAGE_DETAILED: Fusing consumer ExtractAndSumScore/Map(<lambda at game_stats.py:142>) into FilterOutSpammers/FilterOutSpammers apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.144Z: JOB_MESSAGE_DETAILED: Fusing consumer ExtractAndSumScore/CombinePerKey(sum)/GroupByKey/WriteStream into ExtractAndSumScore/Map(<lambda at game_stats.py:142>) apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.147Z: JOB_MESSAGE_DETAILED: Fusing consumer ExtractAndSumScore/CombinePerKey(sum)/GroupByKey/MergeBuckets into ExtractAndSumScore/CombinePerKey(sum)/GroupByKey/ReadStream apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.149Z: JOB_MESSAGE_DETAILED: Fusing consumer ExtractAndSumScore/CombinePerKey(sum)/Combine into ExtractAndSumScore/CombinePerKey(sum)/GroupByKey/MergeBuckets apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.151Z: JOB_MESSAGE_DETAILED: Fusing consumer TeamScoresDict into ExtractAndSumScore/CombinePerKey(sum)/Combine apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.153Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteTeamScoreSums/ConvertToRow into TeamScoresDict apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.155Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/AppendDestination into WriteTeamScoreSums/ConvertToRow apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.158Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/AddInsertIds into WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/AppendDestination apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.160Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/AddRandomKeys into WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/AddInsertIds apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.163Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/Map(reify_timestamps) into WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/AddRandomKeys apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.165Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/GroupByKey/WriteStream into WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/Map(reify_timestamps) apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.168Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/GroupByKey/MergeBuckets into WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/GroupByKey/ReadStream apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.170Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/FlatMap(restore_timestamps) into WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/GroupByKey/MergeBuckets apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.173Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/RemoveRandomKeys into WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/FlatMap(restore_timestamps) apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.175Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/StreamInsertRows/ParDo(BigQueryWriteFn) into WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/RemoveRandomKeys apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.177Z: JOB_MESSAGE_DETAILED: Fusing consumer CombinePerKey(<lambda at game_stats.py:380>)/GroupByKey/WriteStream into WindowIntoSessions apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.179Z: JOB_MESSAGE_DETAILED: Fusing consumer CombinePerKey(<lambda at game_stats.py:380>)/GroupByKey/MergeBuckets into CombinePerKey(<lambda at game_stats.py:380>)/GroupByKey/ReadStream apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.181Z: JOB_MESSAGE_DETAILED: Fusing consumer CombinePerKey(<lambda at game_stats.py:380>)/Combine into CombinePerKey(<lambda at game_stats.py:380>)/GroupByKey/MergeBuckets apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.183Z: JOB_MESSAGE_DETAILED: Fusing consumer UserSessionActivity into CombinePerKey(<lambda at game_stats.py:380>)/Combine apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.186Z: JOB_MESSAGE_DETAILED: Fusing consumer WindowToExtractSessionMean into UserSessionActivity apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.188Z: JOB_MESSAGE_DETAILED: Fusing consumer CombineGlobally(MeanCombineFn)/KeyWithVoid into WindowToExtractSessionMean apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.191Z: JOB_MESSAGE_DETAILED: Fusing consumer CombineGlobally(MeanCombineFn)/CombinePerKey/GroupByKey/WriteStream into CombineGlobally(MeanCombineFn)/KeyWithVoid apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.193Z: JOB_MESSAGE_DETAILED: Fusing consumer CombineGlobally(MeanCombineFn)/CombinePerKey/GroupByKey/MergeBuckets into CombineGlobally(MeanCombineFn)/CombinePerKey/GroupByKey/ReadStream apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.196Z: JOB_MESSAGE_DETAILED: Fusing consumer CombineGlobally(MeanCombineFn)/CombinePerKey/Combine into CombineGlobally(MeanCombineFn)/CombinePerKey/GroupByKey/MergeBuckets apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.198Z: JOB_MESSAGE_DETAILED: Fusing consumer CombineGlobally(MeanCombineFn)/UnKey into CombineGlobally(MeanCombineFn)/CombinePerKey/Combine apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.200Z: JOB_MESSAGE_DETAILED: Fusing consumer FormatAvgSessionLength into CombineGlobally(MeanCombineFn)/UnKey apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.203Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteAvgSessionLength/ConvertToRow into FormatAvgSessionLength apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.206Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/AppendDestination into WriteAvgSessionLength/ConvertToRow apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.208Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/AddInsertIds into WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/AppendDestination apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.210Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/AddRandomKeys into WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/AddInsertIds apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.212Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/Map(reify_timestamps) into WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/AddRandomKeys apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.215Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/GroupByKey/WriteStream into WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/Map(reify_timestamps) apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.217Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/GroupByKey/MergeBuckets into WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/GroupByKey/ReadStream apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.219Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/FlatMap(restore_timestamps) into WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/GroupByKey/MergeBuckets apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.221Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/RemoveRandomKeys into WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/FlatMap(restore_timestamps) apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.223Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/StreamInsertRows/ParDo(BigQueryWriteFn) into WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/RemoveRandomKeys apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.257Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph. apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.312Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps. apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.363Z: JOB_MESSAGE_DEBUG: Assigning stage ids. apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.520Z: JOB_MESSAGE_DEBUG: Executing wait step start2 apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.533Z: JOB_MESSAGE_DEBUG: Starting worker pool setup. apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:41.538Z: JOB_MESSAGE_BASIC: Starting 1 workers... apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:44.652Z: JOB_MESSAGE_BASIC: Executing operation ReadPubSub/Read+DecodeString+ParseGameEventFn+AddEventTimestamps+WindowIntoFixedWindows+ExtractUserScores+WindowIntoSessions+UserFixedWindows+CalculateSpammyUsers/SumUsersScores/GroupByKey/WriteStream+CombinePerKey(<lambda at game_stats.py:380>)/GroupByKey/WriteStream+WindowIntoFixedWindows.out/FromValue/WriteStream apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:44.657Z: JOB_MESSAGE_BASIC: Executing operation WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/GroupByKey/ReadStream+WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/GroupByKey/MergeBuckets+WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/FlatMap(restore_timestamps)+WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/RemoveRandomKeys+WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/StreamInsertRows/ParDo(BigQueryWriteFn) apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:44.658Z: JOB_MESSAGE_BASIC: Executing operation CalculateSpammyUsers/SumUsersScores/Combine.out/FromValue/ReadStream+CalculateSpammyUsers/ProcessAndFilter/ProcessAndFilter+CreateSpammersView/KeyWithVoid+CreateSpammersView/CombinePerKey/GroupByKey/WriteStream apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:44.658Z: JOB_MESSAGE_BASIC: Executing operation CalculateSpammyUsers/ProcessAndFilter/_UnpickledSideInput(MapToVoidKey0.out.0)/GroupByKey/ReadStream+CalculateSpammyUsers/ProcessAndFilter/_UnpickledSideInput(MapToVoidKey0.out.0)/GroupByKey/MergeBuckets+CalculateSpammyUsers/ProcessAndFilter/_UnpickledSideInput(MapToVoidKey0.out.0)/Values+CalculateSpammyUsers/ProcessAndFilter/_UnpickledSideInput(MapToVoidKey0.out.0)/StreamingPCollectionViewWriter apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:44.658Z: JOB_MESSAGE_BASIC: Executing operation ExtractAndSumScore/CombinePerKey(sum)/GroupByKey/ReadStream+ExtractAndSumScore/CombinePerKey(sum)/GroupByKey/MergeBuckets+ExtractAndSumScore/CombinePerKey(sum)/Combine+TeamScoresDict+WriteTeamScoreSums/ConvertToRow+WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/AppendDestination+WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/AddInsertIds+WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/AddRandomKeys+WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/Map(reify_timestamps)+WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/GroupByKey/WriteStream apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:44.658Z: JOB_MESSAGE_BASIC: Executing operation CalculateSpammyUsers/SumUsersScores/GroupByKey/ReadStream+CalculateSpammyUsers/SumUsersScores/GroupByKey/MergeBuckets+CalculateSpammyUsers/SumUsersScores/Combine+CalculateSpammyUsers/Values+CalculateSpammyUsers/CombineGlobally(MeanCombineFn)/KeyWithVoid+CalculateSpammyUsers/CombineGlobally(MeanCombineFn)/CombinePerKey/GroupByKey/WriteStream+CalculateSpammyUsers/SumUsersScores/Combine.out/FromValue/WriteStream apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:44.658Z: JOB_MESSAGE_BASIC: Executing operation CombineGlobally(MeanCombineFn)/CombinePerKey/GroupByKey/ReadStream+CombineGlobally(MeanCombineFn)/CombinePerKey/GroupByKey/MergeBuckets+CombineGlobally(MeanCombineFn)/CombinePerKey/Combine+CombineGlobally(MeanCombineFn)/UnKey+FormatAvgSessionLength+WriteAvgSessionLength/ConvertToRow+WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/AppendDestination+WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/AddInsertIds+WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/AddRandomKeys+WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/Map(reify_timestamps)+WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/GroupByKey/WriteStream apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:44.658Z: JOB_MESSAGE_BASIC: Executing operation CombinePerKey(<lambda at game_stats.py:380>)/GroupByKey/ReadStream+CombinePerKey(<lambda at game_stats.py:380>)/GroupByKey/MergeBuckets+CombinePerKey(<lambda at game_stats.py:380>)/Combine+UserSessionActivity+WindowToExtractSessionMean+CombineGlobally(MeanCombineFn)/KeyWithVoid+CombineGlobally(MeanCombineFn)/CombinePerKey/GroupByKey/WriteStream apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:44.658Z: JOB_MESSAGE_BASIC: Executing operation WindowIntoFixedWindows.out/FromValue/ReadStream+FilterOutSpammers/FilterOutSpammers+ExtractAndSumScore/Map(<lambda at game_stats.py:142>)+ExtractAndSumScore/CombinePerKey(sum)/GroupByKey/WriteStream apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:44.658Z: JOB_MESSAGE_BASIC: Executing operation CalculateSpammyUsers/CombineGlobally(MeanCombineFn)/CombinePerKey/GroupByKey/ReadStream+CalculateSpammyUsers/CombineGlobally(MeanCombineFn)/CombinePerKey/GroupByKey/MergeBuckets+CalculateSpammyUsers/CombineGlobally(MeanCombineFn)/CombinePerKey/Combine+CalculateSpammyUsers/CombineGlobally(MeanCombineFn)/UnKey+CalculateSpammyUsers/ProcessAndFilter/MapToVoidKey0+CalculateSpammyUsers/ProcessAndFilter/MapToVoidKey0+CalculateSpammyUsers/ProcessAndFilter/_UnpickledSideInput(MapToVoidKey0.out.0)/PairWithVoidKey+CalculateSpammyUsers/ProcessAndFilter/_UnpickledSideInput(MapToVoidKey0.out.0)/GroupByKey/WriteStream apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:44.658Z: JOB_MESSAGE_BASIC: Executing operation CreateSpammersView/CombinePerKey/GroupByKey/ReadStream+CreateSpammersView/CombinePerKey/GroupByKey/MergeBuckets+CreateSpammersView/CombinePerKey/Combine+CreateSpammersView/UnKey+FilterOutSpammers/MapToVoidKey0+FilterOutSpammers/MapToVoidKey0+FilterOutSpammers/_UnpickledSideInput(MapToVoidKey0.out.0)/PairWithVoidKey+FilterOutSpammers/_UnpickledSideInput(MapToVoidKey0.out.0)/GroupByKey/WriteStream apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:44.660Z: JOB_MESSAGE_BASIC: Executing operation FilterOutSpammers/_UnpickledSideInput(MapToVoidKey0.out.0)/GroupByKey/ReadStream+FilterOutSpammers/_UnpickledSideInput(MapToVoidKey0.out.0)/GroupByKey/MergeBuckets+FilterOutSpammers/_UnpickledSideInput(MapToVoidKey0.out.0)/Values+FilterOutSpammers/_UnpickledSideInput(MapToVoidKey0.out.0)/StreamingPCollectionViewWriter apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:44.664Z: JOB_MESSAGE_BASIC: Executing operation WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/GroupByKey/ReadStream+WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/GroupByKey/MergeBuckets+WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/FlatMap(restore_timestamps)+WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/RemoveRandomKeys+WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/StreamInsertRows/ParDo(BigQueryWriteFn) apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:53:51.089Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:54:14.131Z: JOB_MESSAGE_DETAILED: Checking permissions granted to controller Service Account. apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:54:14.170Z: JOB_MESSAGE_DEBUG: Executing input step topology_init_attach_disk_input_step apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:54:14.724Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-4 in us-central1-f. apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T19:54:40.837Z: JOB_MESSAGE_DETAILED: Workers have started successfully. apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T20:00:14.126Z: JOB_MESSAGE_DETAILED: Checking permissions granted to controller Service Account. apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T20:00:15.492Z: JOB_MESSAGE_ERROR: Workflow failed. apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T20:00:15.545Z: JOB_MESSAGE_BASIC: Finished operation CreateSpammersView/CombinePerKey/GroupByKey/ReadStream+CreateSpammersView/CombinePerKey/GroupByKey/MergeBuckets+CreateSpammersView/CombinePerKey/Combine+CreateSpammersView/UnKey+FilterOutSpammers/MapToVoidKey0+FilterOutSpammers/MapToVoidKey0+FilterOutSpammers/_UnpickledSideInput(MapToVoidKey0.out.0)/PairWithVoidKey+FilterOutSpammers/_UnpickledSideInput(MapToVoidKey0.out.0)/GroupByKey/WriteStream apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T20:00:15.545Z: JOB_MESSAGE_BASIC: Finished operation CalculateSpammyUsers/CombineGlobally(MeanCombineFn)/CombinePerKey/GroupByKey/ReadStream+CalculateSpammyUsers/CombineGlobally(MeanCombineFn)/CombinePerKey/GroupByKey/MergeBuckets+CalculateSpammyUsers/CombineGlobally(MeanCombineFn)/CombinePerKey/Combine+CalculateSpammyUsers/CombineGlobally(MeanCombineFn)/UnKey+CalculateSpammyUsers/ProcessAndFilter/MapToVoidKey0+CalculateSpammyUsers/ProcessAndFilter/MapToVoidKey0+CalculateSpammyUsers/ProcessAndFilter/_UnpickledSideInput(MapToVoidKey0.out.0)/PairWithVoidKey+CalculateSpammyUsers/ProcessAndFilter/_UnpickledSideInput(MapToVoidKey0.out.0)/GroupByKey/WriteStream apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T20:00:15.546Z: JOB_MESSAGE_BASIC: Finished operation FilterOutSpammers/_UnpickledSideInput(MapToVoidKey0.out.0)/GroupByKey/ReadStream+FilterOutSpammers/_UnpickledSideInput(MapToVoidKey0.out.0)/GroupByKey/MergeBuckets+FilterOutSpammers/_UnpickledSideInput(MapToVoidKey0.out.0)/Values+FilterOutSpammers/_UnpickledSideInput(MapToVoidKey0.out.0)/StreamingPCollectionViewWriter apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T20:00:15.546Z: JOB_MESSAGE_BASIC: Finished operation CombineGlobally(MeanCombineFn)/CombinePerKey/GroupByKey/ReadStream+CombineGlobally(MeanCombineFn)/CombinePerKey/GroupByKey/MergeBuckets+CombineGlobally(MeanCombineFn)/CombinePerKey/Combine+CombineGlobally(MeanCombineFn)/UnKey+FormatAvgSessionLength+WriteAvgSessionLength/ConvertToRow+WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/AppendDestination+WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/AddInsertIds+WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/AddRandomKeys+WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/Map(reify_timestamps)+WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/GroupByKey/WriteStream apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T20:00:15.546Z: JOB_MESSAGE_BASIC: Finished operation WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/GroupByKey/ReadStream+WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/GroupByKey/MergeBuckets+WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/FlatMap(restore_timestamps)+WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/RemoveRandomKeys+WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/StreamInsertRows/ParDo(BigQueryWriteFn) apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T20:00:15.546Z: JOB_MESSAGE_BASIC: Finished operation CalculateSpammyUsers/SumUsersScores/GroupByKey/ReadStream+CalculateSpammyUsers/SumUsersScores/GroupByKey/MergeBuckets+CalculateSpammyUsers/SumUsersScores/Combine+CalculateSpammyUsers/Values+CalculateSpammyUsers/CombineGlobally(MeanCombineFn)/KeyWithVoid+CalculateSpammyUsers/CombineGlobally(MeanCombineFn)/CombinePerKey/GroupByKey/WriteStream+CalculateSpammyUsers/SumUsersScores/Combine.out/FromValue/WriteStream apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T20:00:15.546Z: JOB_MESSAGE_BASIC: Finished operation CombinePerKey(<lambda at game_stats.py:380>)/GroupByKey/ReadStream+CombinePerKey(<lambda at game_stats.py:380>)/GroupByKey/MergeBuckets+CombinePerKey(<lambda at game_stats.py:380>)/Combine+UserSessionActivity+WindowToExtractSessionMean+CombineGlobally(MeanCombineFn)/KeyWithVoid+CombineGlobally(MeanCombineFn)/CombinePerKey/GroupByKey/WriteStream apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T20:00:15.546Z: JOB_MESSAGE_BASIC: Finished operation CalculateSpammyUsers/ProcessAndFilter/_UnpickledSideInput(MapToVoidKey0.out.0)/GroupByKey/ReadStream+CalculateSpammyUsers/ProcessAndFilter/_UnpickledSideInput(MapToVoidKey0.out.0)/GroupByKey/MergeBuckets+CalculateSpammyUsers/ProcessAndFilter/_UnpickledSideInput(MapToVoidKey0.out.0)/Values+CalculateSpammyUsers/ProcessAndFilter/_UnpickledSideInput(MapToVoidKey0.out.0)/StreamingPCollectionViewWriter apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T20:00:15.546Z: JOB_MESSAGE_BASIC: Finished operation ReadPubSub/Read+DecodeString+ParseGameEventFn+AddEventTimestamps+WindowIntoFixedWindows+ExtractUserScores+WindowIntoSessions+UserFixedWindows+CalculateSpammyUsers/SumUsersScores/GroupByKey/WriteStream+CombinePerKey(<lambda at game_stats.py:380>)/GroupByKey/WriteStream+WindowIntoFixedWindows.out/FromValue/WriteStream apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T20:00:15.546Z: JOB_MESSAGE_BASIC: Finished operation ExtractAndSumScore/CombinePerKey(sum)/GroupByKey/ReadStream+ExtractAndSumScore/CombinePerKey(sum)/GroupByKey/MergeBuckets+ExtractAndSumScore/CombinePerKey(sum)/Combine+TeamScoresDict+WriteTeamScoreSums/ConvertToRow+WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/AppendDestination+WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/AddInsertIds+WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/AddRandomKeys+WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/Map(reify_timestamps)+WriteTeamScoreSums/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/GroupByKey/WriteStream apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T20:00:15.546Z: JOB_MESSAGE_BASIC: Finished operation CalculateSpammyUsers/SumUsersScores/Combine.out/FromValue/ReadStream+CalculateSpammyUsers/ProcessAndFilter/ProcessAndFilter+CreateSpammersView/KeyWithVoid+CreateSpammersView/CombinePerKey/GroupByKey/WriteStream apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T20:00:15.546Z: JOB_MESSAGE_BASIC: Finished operation WindowIntoFixedWindows.out/FromValue/ReadStream+FilterOutSpammers/FilterOutSpammers+ExtractAndSumScore/Map(<lambda at game_stats.py:142>)+ExtractAndSumScore/CombinePerKey(sum)/GroupByKey/WriteStream apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T20:00:15.546Z: JOB_MESSAGE_BASIC: Finished operation WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/GroupByKey/ReadStream+WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/GroupByKey/MergeBuckets+WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/ReshufflePerKey/FlatMap(restore_timestamps)+WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/CommitInsertIds/RemoveRandomKeys+WriteAvgSessionLength/WriteToBigQuery/_StreamToBigQuery/StreamInsertRows/ParDo(BigQueryWriteFn) apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T20:00:15.710Z: JOB_MESSAGE_DETAILED: Cleaning up. apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T20:00:15.747Z: JOB_MESSAGE_BASIC: Worker pool stopped. apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T20:00:15.752Z: JOB_MESSAGE_DEBUG: Tearing down pending resources... apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T20:00:15.755Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown. apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T20:00:15.761Z: JOB_MESSAGE_BASIC: Stopping worker pool... apache_beam.runners.dataflow.dataflow_runner: INFO: 2019-12-30T20:01:38.218Z: JOB_MESSAGE_BASIC: Worker pool stopped. apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2019-12-30_11_53_36-14348226119756296982 is in state JOB_STATE_FAILED google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254 google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None) google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144 google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/[email protected]/token urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/[email protected]/token HTTP/1.1" 200 181 urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): www.googleapis.com:443 urllib3.connectionpool: DEBUG: https://www.googleapis.com:443 "DELETE /bigquery/v2/projects/apache-beam-testing/datasets/game_stats_it_dataset15777355992078?deleteContents=true HTTP/1.1" 204 0 --------------------- >> end captured logging << --------------------- <https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1419: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_11_53_36-14348226119756296982?project=apache-beam-testing experiments = p.options.view_as(DebugOptions).experiments or [] <https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:738: BeamDeprecationWarning: BigQuerySink is deprecated since 2.11.0. Use WriteToBigQuery instead. kms_key=transform.kms_key)) Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_01_58-13252039874734005529?project=apache-beam-testing <https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:738: BeamDeprecationWarning: BigQuerySink is deprecated since 2.11.0. Use WriteToBigQuery instead. Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_09_36-12979806866778676576?project=apache-beam-testing Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_16_34-4619127899463122575?project=apache-beam-testing Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_23_07-13421850025067188086?project=apache-beam-testing kms_key=transform.kms_key)) <https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:738: BeamDeprecationWarning: BigQuerySink is deprecated since 2.11.0. Use WriteToBigQuery instead. kms_key=transform.kms_key)) Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_30_16-821612746332450051?project=apache-beam-testing <https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_pipeline.py>:73: BeamDeprecationWarning: BigQuerySink is deprecated since 2.11.0. Use WriteToBigQuery instead. kms_key=kms_key)) Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_11_53_32-17626760183557177692?project=apache-beam-testing Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_13_42-9618025078830332585?project=apache-beam-testing <https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:738: BeamDeprecationWarning: BigQuerySink is deprecated since 2.11.0. Use WriteToBigQuery instead. kms_key=transform.kms_key)) <https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1416: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported self.table_reference.projectId = pcoll.pipeline.options.view_as( Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_11_53_43-17634546776590373825?project=apache-beam-testing <https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1419: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported experiments = p.options.view_as(DebugOptions).experiments or [] Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_05_15-246967168736526216?project=apache-beam-testing Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_13_02-14111849260756409399?project=apache-beam-testing <https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py>:257: FutureWarning: _ReadFromBigQuery is experimental. Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_20_14-4115068758920310357?project=apache-beam-testing Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_28_03-12627326566448895748?project=apache-beam-testing query=self.query, use_standard_sql=True, project=self.project)) <https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1603: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported temp_location = pcoll.pipeline.options.view_as( <https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/fileio_test.py>:296: FutureWarning: MatchAll is experimental. | 'GetPath' >> beam.Map(lambda metadata: metadata.path)) <https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/fileio_test.py>:307: FutureWarning: MatchAll is experimental. | 'Checksums' >> beam.Map(compute_hash)) <https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/fileio_test.py>:307: FutureWarning: ReadMatches is experimental. | 'Checksums' >> beam.Map(compute_hash)) Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_11_53_33-11834164941394018606?project=apache-beam-testing Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_13_28-4665684761932790492?project=apache-beam-testing <https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1419: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported experiments = p.options.view_as(DebugOptions).experiments or [] Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_21_59-15693394061658925804?project=apache-beam-testing <https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:769: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported temp_location = p.options.view_as(GoogleCloudOptions).temp_location Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_11_53_33-17170209896003486147?project=apache-beam-testing <https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/gcp/bigquery_test.py>:755: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported streaming = self.test_pipeline.options.view_as(StandardOptions).streaming <https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1419: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported experiments = p.options.view_as(DebugOptions).experiments or [] <https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:738: BeamDeprecationWarning: BigQuerySink is deprecated since 2.11.0. Use WriteToBigQuery instead. kms_key=transform.kms_key)) Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_02_12-6021964349359335249?project=apache-beam-testing Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_10_25-6037096648637209957?project=apache-beam-testing Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_18_40-2294080045690828858?project=apache-beam-testing <https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:738: BeamDeprecationWarning: BigQuerySink is deprecated since 2.11.0. Use WriteToBigQuery instead. kms_key=transform.kms_key)) <https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:738: BeamDeprecationWarning: BigQuerySink is deprecated since 2.11.0. Use WriteToBigQuery instead. kms_key=transform.kms_key)) Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_26_24-17341544584248682533?project=apache-beam-testing Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_34_13-4899540391510303767?project=apache-beam-testing Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_11_53_34-945117617953345915?project=apache-beam-testing <https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py>:738: BeamDeprecationWarning: BigQuerySink is deprecated since 2.11.0. Use WriteToBigQuery instead. kms_key=transform.kms_key)) Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_01_28-5081143457058694566?project=apache-beam-testing <https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py>:153: FutureWarning: _ReadFromBigQuery is experimental. query=self.query, use_standard_sql=True, project=self.project)) Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_09_42-17466297344613354667?project=apache-beam-testing Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_17_28-17639710491240318297?project=apache-beam-testing Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_24_39-2148658246662963851?project=apache-beam-testing Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_32_32-10814144863972347642?project=apache-beam-testing <https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1603: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported temp_location = pcoll.pipeline.options.view_as( Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_11_53_36-11033240698793927201?project=apache-beam-testing <https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1419: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_02_13-8571912086936072151?project=apache-beam-testing Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_10_55-10940162090418432843?project=apache-beam-testing experiments = p.options.view_as(DebugOptions).experiments or [] Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_19_48-17843392176946716707?project=apache-beam-testing <https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:769: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported temp_location = p.options.view_as(GoogleCloudOptions).temp_location Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_26_45-11795513877592026122?project=apache-beam-testing <https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/gcp/bigquery.py>:1419: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported experiments = p.options.view_as(DebugOptions).experiments or [] <https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py>:769: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported temp_location = p.options.view_as(GoogleCloudOptions).temp_location Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_11_53_35-454656096880260115?project=apache-beam-testing Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_01_42-4428276239640077477?project=apache-beam-testing Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_09_03-40683692824687434?project=apache-beam-testing Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_17_20-4990297014580277887?project=apache-beam-testing Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_25_52-14861515761820564999?project=apache-beam-testing Worker logs: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-12-30_12_33_13-11686507863410369686?project=apache-beam-testing ---------------------------------------------------------------------- XML: nosetests-postCommitIT-df-py37.xml ---------------------------------------------------------------------- XML: <https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/nosetests.xml> ---------------------------------------------------------------------- Ran 47 tests in 2903.462s FAILED (SKIP=6, errors=1) > Task :sdks:python:test-suites:dataflow:py37:postCommitIT FAILED FAILURE: Build failed with an exception. * Where: Build file '<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/test-suites/dataflow/py37/build.gradle'> line: 89 * What went wrong: Execution failed for task ':sdks:python:test-suites:dataflow:py37:postCommitIT'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 49m 48s 85 actionable tasks: 64 executed, 21 from cache Publishing build scan... https://gradle.com/s/a7uav7hkhjpbu Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
