See <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/119/display/redirect?page=changes>
Changes: [noreply] Bump google.golang.org/grpc from 1.54.0 to 1.55.0 in /sdks (#26568) ------------------------------------------ [...truncated 1.83 MB...] [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:00.803Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_20#vocabulary]/WriteToText/Write/WriteImpl/PreFinalize [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:00.832Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_3#vocabulary]/CreatePath/Impulse+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_3#vocabulary]/CreatePath/FlatMap(<lambda at core.py:3638>)+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_3#vocabulary]/CreatePath/Map(decode)+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_3#vocabulary]/WaitForVocabularyFile+Analyze/CreateTensorBinding[compute_and_apply_vocabulary_3#vocabulary#temporary_analyzer_output_2#Const]/ToTensorBinding [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:00.886Z: JOB_MESSAGE_DEBUG: Value "Analyze/CreateTensorBinding[compute_and_apply_vocabulary_3#vocabulary#temporary_analyzer_output_2#Const]/ToTensorBinding.None" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:00.939Z: JOB_MESSAGE_BASIC: Executing operation Analyze/CreateSavedModel[tf_v2_only]/ReplaceWithConstants/View-python_side_input12-Analyze/CreateSavedModel[tf_v2_only]/ReplaceWithConstants [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:00.990Z: JOB_MESSAGE_BASIC: Finished operation Analyze/CreateSavedModel[tf_v2_only]/ReplaceWithConstants/View-python_side_input12-Analyze/CreateSavedModel[tf_v2_only]/ReplaceWithConstants [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:01.027Z: JOB_MESSAGE_DEBUG: Value "Analyze/CreateSavedModel[tf_v2_only]/ReplaceWithConstants/View-python_side_input12-Analyze/CreateSavedModel[tf_v2_only]/ReplaceWithConstants.out" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:05.998Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_20#vocabulary]/WriteToText/Write/WriteImpl/PreFinalize [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:06.059Z: JOB_MESSAGE_DEBUG: Value "Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_20#vocabulary]/WriteToText/Write/WriteImpl/PreFinalize.None" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:06.125Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_20#vocabulary]/WriteToText/Write/WriteImpl/FinalizeWrite/View-python_side_input2-Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_20#vocabulary]/WriteToText/Write/WriteImpl/FinalizeWrite [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:06.165Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_20#vocabulary]/WriteToText/Write/WriteImpl/FinalizeWrite/View-python_side_input2-Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_20#vocabulary]/WriteToText/Write/WriteImpl/FinalizeWrite [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:06.206Z: JOB_MESSAGE_DEBUG: Value "Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_20#vocabulary]/WriteToText/Write/WriteImpl/FinalizeWrite/View-python_side_input2-Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_20#vocabulary]/WriteToText/Write/WriteImpl/FinalizeWrite.out" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:06.260Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_20#vocabulary]/WriteToText/Write/WriteImpl/FinalizeWrite [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:16.438Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_20#vocabulary]/WriteToText/Write/WriteImpl/FinalizeWrite [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:16.493Z: JOB_MESSAGE_DEBUG: Value "Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_20#vocabulary]/WriteToText/Write/WriteImpl/FinalizeWrite.None" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:16.550Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_20#vocabulary]/WaitForVocabularyFile/View-python_side_input0-Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_20#vocabulary]/WaitForVocabularyFile [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:16.585Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_20#vocabulary]/WaitForVocabularyFile/View-python_side_input0-Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_20#vocabulary]/WaitForVocabularyFile [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:16.639Z: JOB_MESSAGE_DEBUG: Value "Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_20#vocabulary]/WaitForVocabularyFile/View-python_side_input0-Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_20#vocabulary]/WaitForVocabularyFile.out" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:16.694Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_20#vocabulary]/CreatePath/Impulse+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_20#vocabulary]/CreatePath/FlatMap(<lambda at core.py:3638>)+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_20#vocabulary]/CreatePath/Map(decode)+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_20#vocabulary]/WaitForVocabularyFile+Analyze/CreateTensorBinding[compute_and_apply_vocabulary_20#vocabulary#temporary_analyzer_output_2#Const]/ToTensorBinding [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:18.493Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_20#vocabulary]/CreatePath/Impulse+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_20#vocabulary]/CreatePath/FlatMap(<lambda at core.py:3638>)+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_20#vocabulary]/CreatePath/Map(decode)+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_20#vocabulary]/WaitForVocabularyFile+Analyze/CreateTensorBinding[compute_and_apply_vocabulary_20#vocabulary#temporary_analyzer_output_2#Const]/ToTensorBinding [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:18.542Z: JOB_MESSAGE_DEBUG: Value "Analyze/CreateTensorBinding[compute_and_apply_vocabulary_20#vocabulary#temporary_analyzer_output_2#Const]/ToTensorBinding.None" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:18.583Z: JOB_MESSAGE_BASIC: Executing operation Analyze/CreateSavedModel[tf_v2_only]/ReplaceWithConstants/View-python_side_input63-Analyze/CreateSavedModel[tf_v2_only]/ReplaceWithConstants [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:18.632Z: JOB_MESSAGE_BASIC: Finished operation Analyze/CreateSavedModel[tf_v2_only]/ReplaceWithConstants/View-python_side_input63-Analyze/CreateSavedModel[tf_v2_only]/ReplaceWithConstants [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:18.673Z: JOB_MESSAGE_DEBUG: Value "Analyze/CreateSavedModel[tf_v2_only]/ReplaceWithConstants/View-python_side_input63-Analyze/CreateSavedModel[tf_v2_only]/ReplaceWithConstants.out" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:20.269Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyAccumulate[compute_and_apply_vocabulary_11#vocabulary]/CountPerToken/CombinePerKey(CountCombineFn)/GroupByKey/Read+Analyze/VocabularyAccumulate[compute_and_apply_vocabulary_11#vocabulary]/CountPerToken/CombinePerKey(CountCombineFn)/Combine+Analyze/VocabularyAccumulate[compute_and_apply_vocabulary_11#vocabulary]/CountPerToken/CombinePerKey(CountCombineFn)/Combine/Extract+Analyze/VocabularyMerge[compute_and_apply_vocabulary_11#vocabulary]/MergeCountPerToken/GroupByKey+Analyze/VocabularyMerge[compute_and_apply_vocabulary_11#vocabulary]/MergeCountPerToken/Combine/Partial+Analyze/VocabularyMerge[compute_and_apply_vocabulary_11#vocabulary]/MergeCountPerToken/GroupByKey/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:20.325Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyMerge[compute_and_apply_vocabulary_11#vocabulary]/MergeCountPerToken/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:20.599Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyMerge[compute_and_apply_vocabulary_11#vocabulary]/MergeCountPerToken/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:20.657Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:20.700Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:20.781Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:20.822Z: JOB_MESSAGE_DEBUG: Value "Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Session" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:20.824Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:20.875Z: JOB_MESSAGE_DEBUG: Value "Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Session" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:08:20.930Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyMerge[compute_and_apply_vocabulary_11#vocabulary]/MergeCountPerToken/GroupByKey/Read+Analyze/VocabularyMerge[compute_and_apply_vocabulary_11#vocabulary]/MergeCountPerToken/Combine+Analyze/VocabularyMerge[compute_and_apply_vocabulary_11#vocabulary]/MergeCountPerToken/Combine/Extract+Analyze/VocabularyMerge[compute_and_apply_vocabulary_11#vocabulary]/SwapTokensAndCounts/KvSwap+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/KeyWithVoid+Analyze/VocabularyPrune[compute_and_apply_vocabulary_11#vocabulary]/ApplyThresholdsAndTopK/FilterByThresholds(5.0)+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write+Analyze/VocabularyPrune[compute_and_apply_vocabulary_11#vocabulary]/ApplyThresholdsAndTopK/FlattenToSingleMetric+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/KeyWithVoid+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/BatchAndPreSort/BatchVocabulary/ParDo(_GlobalWindowsBatchingDoFn)+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/BatchAndPreSort/SortBatches [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:09:56.039Z: JOB_MESSAGE_DETAILED: Autoscaling: Reduced the number of workers to 1 based on the rate of progress in the currently running stage(s). [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:10:01.065Z: JOB_MESSAGE_DETAILED: Autoscaling: Resizing worker pool from 2 to 1. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:15:10.582Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running stage(s). [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:23:22.672Z: JOB_MESSAGE_DETAILED: Autoscaling: Resizing worker pool from 1 to 2. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:23:27.963Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 2 based on the rate of progress in the currently running stage(s). [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:30:08.410Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 2 based on the rate of progress in the currently running stage(s). [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:31:34.143Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/MergeCountPerToken/GroupByKey/Read+Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/MergeCountPerToken/Combine+Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/MergeCountPerToken/Combine/Extract+Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/SwapTokensAndCounts/KvSwap+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/KeyWithVoid+Analyze/VocabularyPrune[compute_and_apply_vocabulary_2#vocabulary]/ApplyThresholdsAndTopK/FilterByThresholds(5.0)+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write+Analyze/VocabularyPrune[compute_and_apply_vocabulary_2#vocabulary]/ApplyThresholdsAndTopK/FlattenToSingleMetric+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/KeyWithVoid+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_2#vocabulary]/BatchAndPreSort/BatchVocabulary/ParDo(_GlobalWindowsBatchingDoFn)+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_2#vocabulary]/BatchAndPreSort/SortBatches [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:31:34.200Z: JOB_MESSAGE_DEBUG: Executing failure step failure2106 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:31:34.222Z: JOB_MESSAGE_ERROR: Workflow failed. Causes: S751:Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/MergeCountPerToken/GroupByKey/Read+Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/MergeCountPerToken/Combine+Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/MergeCountPerToken/Combine/Extract+Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/SwapTokensAndCounts/KvSwap+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/KeyWithVoid+Analyze/VocabularyPrune[compute_and_apply_vocabulary_2#vocabulary]/ApplyThresholdsAndTopK/FilterByThresholds(5.0)+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write+Analyze/VocabularyPrune[compute_and_apply_vocabulary_2#vocabulary]/ApplyThresholdsAndTopK/FlattenToSingleMetric+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/KeyWithVoid+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_2#vocabulary]/BatchAndPreSort/BatchVocabulary/ParDo(_GlobalWindowsBatchingDoFn)+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_2#vocabulary]/BatchAndPreSort/SortBatches failed., The job failed because a work item has failed 4 times. Look in previous log entries for the cause of each one of the 4 failures. If the logs only contain generic timeout errors related to accessing external resources, such as MongoDB, verify that the worker service account has permission to access the resource's subnetwork. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors. The work item was attempted on these workers: Root cause: Timed out waiting for an update from the worker. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors#worker-lost-contact. Worker ID: beamapp-jenkins-050601364-05051838-od3u-harness-z55l, Root cause: Timed out waiting for an update from the worker. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors#worker-lost-contact. Worker ID: beamapp-jenkins-050601364-05051838-od3u-harness-92bf, Root cause: Timed out waiting for an update from the worker. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors#worker-lost-contact. Worker ID: beamapp-jenkins-050601364-05051838-od3u-harness-9r6n, Root cause: Timed out waiting for an update from the worker. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors#worker-lost-contact. Worker ID: beamapp-jenkins-050601364-05051838-od3u-harness-gtww [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:31:34.320Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyAccumulate[compute_and_apply_vocabulary_15#vocabulary]/CountPerToken/CombinePerKey(CountCombineFn)/GroupByKey/Read+Analyze/VocabularyAccumulate[compute_and_apply_vocabulary_15#vocabulary]/CountPerToken/CombinePerKey(CountCombineFn)/Combine+Analyze/VocabularyAccumulate[compute_and_apply_vocabulary_15#vocabulary]/CountPerToken/CombinePerKey(CountCombineFn)/Combine/Extract+Analyze/VocabularyMerge[compute_and_apply_vocabulary_15#vocabulary]/MergeCountPerToken/GroupByKey+Analyze/VocabularyMerge[compute_and_apply_vocabulary_15#vocabulary]/MergeCountPerToken/Combine/Partial+Analyze/VocabularyMerge[compute_and_apply_vocabulary_15#vocabulary]/MergeCountPerToken/GroupByKey/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:31:34.388Z: JOB_MESSAGE_WARNING: Unable to delete temp files: "gs://temp-storage-for-perf-tests/loadtests/beamapp-jenkins-0506013646-040211-gscolod6.1683337006.040432/dax-tmp-2023-05-05_18_38_11-18265271942747178913-S322-0-eab9f876a4e0fdf4/[email protected]." [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:31:34.412Z: JOB_MESSAGE_WARNING: S322:Analyze/VocabularyMerge[compute_and_apply_vocabulary_11#vocabulary]/MergeCountPerToken/GroupByKey/Read+Analyze/VocabularyMerge[compute_and_apply_vocabulary_11#vocabulary]/MergeCountPerToken/Combine+Analyze/VocabularyMerge[compute_and_apply_vocabulary_11#vocabulary]/MergeCountPerToken/Combine/Extract+Analyze/VocabularyMerge[compute_and_apply_vocabulary_11#vocabulary]/SwapTokensAndCounts/KvSwap+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/KeyWithVoid+Analyze/VocabularyPrune[compute_and_apply_vocabulary_11#vocabulary]/ApplyThresholdsAndTopK/FilterByThresholds(5.0)+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write+Analyze/VocabularyPrune[compute_and_apply_vocabulary_11#vocabulary]/ApplyThresholdsAndTopK/FlattenToSingleMetric+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/KeyWithVoid+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/BatchAndPreSort/BatchVocabulary/ParDo(_GlobalWindowsBatchingDoFn)+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/BatchAndPreSort/SortBatches failed. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:31:34.439Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyMerge[compute_and_apply_vocabulary_11#vocabulary]/MergeCountPerToken/GroupByKey/Read+Analyze/VocabularyMerge[compute_and_apply_vocabulary_11#vocabulary]/MergeCountPerToken/Combine+Analyze/VocabularyMerge[compute_and_apply_vocabulary_11#vocabulary]/MergeCountPerToken/Combine/Extract+Analyze/VocabularyMerge[compute_and_apply_vocabulary_11#vocabulary]/SwapTokensAndCounts/KvSwap+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/KeyWithVoid+Analyze/VocabularyPrune[compute_and_apply_vocabulary_11#vocabulary]/ApplyThresholdsAndTopK/FilterByThresholds(5.0)+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write+Analyze/VocabularyPrune[compute_and_apply_vocabulary_11#vocabulary]/ApplyThresholdsAndTopK/FlattenToSingleMetric+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/KeyWithVoid+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/BatchAndPreSort/BatchVocabulary/ParDo(_GlobalWindowsBatchingDoFn)+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/BatchAndPreSort/SortBatches [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:31:34.658Z: JOB_MESSAGE_DETAILED: Cleaning up. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:31:34.981Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:31:35.004Z: JOB_MESSAGE_BASIC: Stopping worker pool... [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:34:23.336Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized worker pool from 2 to 0. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:34:23.375Z: JOB_MESSAGE_BASIC: Worker pool stopped. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:238 2023-05-06T02:34:23.402Z: JOB_MESSAGE_DEBUG: Tearing down pending resources... [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:193 Job 2023-05-05_18_38_11-18265271942747178913 is in state JOB_STATE_FAILED [1m[31mERROR [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:1564 Console URL: https://console.cloud.google.com/dataflow/jobs/<RegionId>/2023-05-05_18_38_11-18265271942747178913?project=<ProjectId> [33m=============================== warnings summary ===============================[0m ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15 <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py>:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses from imp import load_source ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:121 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:121 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:121 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:121 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:121 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:121 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:121 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:121 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:121 <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py>:121: DeprecationWarning: pkg_resources is deprecated as an API warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning) ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870: 162 warnings <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py>:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870: 117 warnings <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py>:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.cloud')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2349: 36 warnings <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py>:2349: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(parent) ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py>:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.logging')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2870 <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py>:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.iam')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:20 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:20 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:20 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:20 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:20 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:20 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:20 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:20 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:20 <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py>:20: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.rpc')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages pkg_resources.declare_namespace(__name__) apache_beam/typehints/pandas_type_compatibility_test.py:67 apache_beam/typehints/pandas_type_compatibility_test.py:67 apache_beam/typehints/pandas_type_compatibility_test.py:67 apache_beam/typehints/pandas_type_compatibility_test.py:67 apache_beam/typehints/pandas_type_compatibility_test.py:67 apache_beam/typehints/pandas_type_compatibility_test.py:67 apache_beam/typehints/pandas_type_compatibility_test.py:67 apache_beam/typehints/pandas_type_compatibility_test.py:67 <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:67: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead. }).set_index(pd.Int64Index(range(123, 223), name='an_index')), apache_beam/typehints/pandas_type_compatibility_test.py:90 apache_beam/typehints/pandas_type_compatibility_test.py:90 apache_beam/typehints/pandas_type_compatibility_test.py:90 apache_beam/typehints/pandas_type_compatibility_test.py:90 apache_beam/typehints/pandas_type_compatibility_test.py:90 apache_beam/typehints/pandas_type_compatibility_test.py:90 apache_beam/typehints/pandas_type_compatibility_test.py:90 apache_beam/typehints/pandas_type_compatibility_test.py:90 <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:90: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead. pd.Int64Index(range(123, 223), name='an_index'), apache_beam/typehints/pandas_type_compatibility_test.py:91 apache_beam/typehints/pandas_type_compatibility_test.py:91 apache_beam/typehints/pandas_type_compatibility_test.py:91 apache_beam/typehints/pandas_type_compatibility_test.py:91 apache_beam/typehints/pandas_type_compatibility_test.py:91 apache_beam/typehints/pandas_type_compatibility_test.py:91 apache_beam/typehints/pandas_type_compatibility_test.py:91 apache_beam/typehints/pandas_type_compatibility_test.py:91 <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/sdks/python/apache_beam/typehints/pandas_type_compatibility_test.py>:91: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead. pd.Int64Index(range(475, 575), name='another_index'), apache_beam/testing/benchmarks/cloudml/cloudml_benchmark_test.py::CloudMLTFTBenchmarkTest::test_cloudml_benchmark_cirteo_no_shuffle_10GB apache_beam/testing/benchmarks/cloudml/cloudml_benchmark_test.py::CloudMLTFTBenchmarkTest::test_cloudml_benchmark_criteo_small <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py>:484: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead. bq_dataset_ref = self._client.dataset(dataset_name) -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html - generated xml file: <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/sdks/python/pytest_TFTransformTests-df-py39.xml> - [36m[1m=========================== short test summary info ============================[0m [31mFAILED[0m apache_beam/testing/benchmarks/cloudml/cloudml_benchmark_test.py::[1mCloudMLTFTBenchmarkTest::test_cloudml_benchmark_criteo_10GB[0m - apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error: Workflow failed. Causes: S751:Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/MergeCountPerToken/GroupByKey/Read+Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/MergeCountPerToken/Combine+Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/MergeCountPerToken/Combine/Extract+Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/SwapTokensAndCounts/KvSwap+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/KeyWithVoid+Analyze/VocabularyPrune[compute_and_apply_vocabulary_2#vocabulary]/ApplyThresholdsAndTopK/FilterByThresholds(5.0)+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write+Analyze/VocabularyPrune[compute_and_apply_vocabulary_2#vocabulary]/ApplyThresholdsAndTopK/FlattenToSingleMetric+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/KeyWithVoid+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_2#vocabulary]/BatchAndPreSort/BatchVocabulary/ParDo(_GlobalWindowsBatchingDoFn)+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_2#vocabulary]/BatchAndPreSort/SortBatches failed., The job failed because a work item has failed 4 times. Look in previous log entries for the cause of each one of the 4 failures. If the logs only contain generic timeout errors related to accessing external resources, such as MongoDB, verify that the worker service account has permission to access the resource's subnetwork. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors. The work item was attempted on these workers: Root cause: Timed out waiting for an update from the worker. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors#worker-lost-contact. Worker ID: beamapp-jenkins-050601364-05051838-od3u-harness-z55l, Root cause: Timed out waiting for an update from the worker. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors#worker-lost-contact. Worker ID: beamapp-jenkins-050601364-05051838-od3u-harness-92bf, Root cause: Timed out waiting for an update from the worker. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors#worker-lost-contact. Worker ID: beamapp-jenkins-050601364-05051838-od3u-harness-9r6n, Root cause: Timed out waiting for an update from the worker. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors#worker-lost-contact. Worker ID: beamapp-jenkins-050601364-05051838-od3u-harness-gtww [31m====== [31m[1m1 failed[0m, [32m2 passed[0m, [33m8 skipped[0m, [33m386 warnings[0m[31m in 7750.22s (2:09:10)[0m[31m =======[0m > Task :sdks:python:test-suites:dataflow:py39:tftTests FAILED FAILURE: Build failed with an exception. * Where: Script '<https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 428 * What went wrong: Execution failed for task ':sdks:python:test-suites:dataflow:py39:tftTests'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0. You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins. See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 2h 11m 51s 16 actionable tasks: 10 executed, 4 from cache, 2 up-to-date Build scan background action failed. java.lang.IllegalArgumentException: com.gradle.enterprise.gradleplugin.internal.extension.a is not an interface at java.lang.reflect.Proxy$ProxyClassFactory.apply(Proxy.java:590) at java.lang.reflect.Proxy$ProxyClassFactory.apply(Proxy.java:557) at java.lang.reflect.WeakCache$Factory.get(WeakCache.java:230) at java.lang.reflect.WeakCache.get(WeakCache.java:127) at java.lang.reflect.Proxy.getProxyClass0(Proxy.java:419) at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:719) at com.gradle.ProxyFactory$ProxyingInvocationHandler.createLocalProxy(ProxyFactory.java:66) at com.gradle.ProxyFactory$ProxyingInvocationHandler.lambda$adaptActionArg$1(ProxyFactory.java:61) at com.gradle.enterprise.gradleplugin.internal.extension.b$3.run(SourceFile:100) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Publishing build scan... https://ge.apache.org/s/y6dcv6rs3tuvu Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
