See <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/253/display/redirect>
Changes: ------------------------------------------ [...truncated 986.50 KB...] [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:08.677Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:08.699Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:08.732Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/OrderElements/View-python_side_input0 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:08.737Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:08.775Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:08.794Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/OrderElements/View-python_side_input0 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:08.804Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Read+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/Combine+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Extract+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/UnKey [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:08.836Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Read+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/Combine+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Extract+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/UnKey [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:08.904Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/WriteToText/Write/WriteImpl/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:09.039Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/WriteToText/Write/WriteImpl/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:09.129Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/Prepare/Impulse+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/Prepare/FlatMap(<lambda at core.py:3736>)+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/Prepare/Map(decode)+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/OrderElements+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/WriteToText/Write/WriteImpl/Map(<lambda at iobase.py:1144>)+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/WriteToText/Write/WriteImpl/WindowInto(WindowIntoFn)+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/WriteToText/Write/WriteImpl/GroupByKey/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:10.054Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/Prepare/Impulse+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/Prepare/FlatMap(<lambda at core.py:3736>)+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/Prepare/Map(decode)+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/OrderElements+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/WriteToText/Write/WriteImpl/Map(<lambda at iobase.py:1144>)+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/WriteToText/Write/WriteImpl/WindowInto(WindowIntoFn)+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/WriteToText/Write/WriteImpl/GroupByKey/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:10.114Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/WriteToText/Write/WriteImpl/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:10.214Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Read+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/Combine+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Extract+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/UnKey [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:10.354Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/InjectDefault/View-python_side_input0 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:10.400Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/InjectDefault/View-python_side_input0 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:10.508Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/DoOnce/Impulse+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/DoOnce/FlatMap(<lambda at core.py:3736>)+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/DoOnce/Map(decode)+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/InjectDefault+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/FormatCount+Analyze/CreateTensorBinding[compute_and_apply_vocabulary_11#vocabulary#temporary_analyzer_output_1#vocab_compute_and_apply_vocabulary_11_vocabulary_pruned_vocab_size]/ToTensorBinding [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:10.534Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Read+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/Combine+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Extract+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/UnKey [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:10.642Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/InjectDefault/View-python_side_input0 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:10.704Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/InjectDefault/View-python_side_input0 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:10.813Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/DoOnce/Impulse+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/DoOnce/FlatMap(<lambda at core.py:3736>)+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/DoOnce/Map(decode)+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/InjectDefault+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/FormatCount+Analyze/CreateTensorBinding[compute_and_apply_vocabulary_11#vocabulary#temporary_analyzer_output#vocab_compute_and_apply_vocabulary_11_vocabulary_unpruned_vocab_size]/ToTensorBinding [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:11.266Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/WriteToText/Write/WriteImpl/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:11.329Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/WriteToText/Write/WriteImpl/GroupByKey/Read+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/WriteToText/Write/WriteImpl/WriteBundles [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:11.913Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/DoOnce/Impulse+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/DoOnce/FlatMap(<lambda at core.py:3736>)+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/DoOnce/Map(decode)+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/InjectDefault+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_11#vocabulary]/FormatCount+Analyze/CreateTensorBinding[compute_and_apply_vocabulary_11#vocabulary#temporary_analyzer_output_1#vocab_compute_and_apply_vocabulary_11_vocabulary_pruned_vocab_size]/ToTensorBinding [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:12.044Z: JOB_MESSAGE_BASIC: Executing operation Analyze/CreateSavedModel[tf_v2_only]/ReplaceWithConstants/View-python_side_input35 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:12.089Z: JOB_MESSAGE_BASIC: Finished operation Analyze/CreateSavedModel[tf_v2_only]/ReplaceWithConstants/View-python_side_input35 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:12.231Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/DoOnce/Impulse+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/DoOnce/FlatMap(<lambda at core.py:3736>)+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/DoOnce/Map(decode)+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/InjectDefault+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_11#vocabulary]/FormatCount+Analyze/CreateTensorBinding[compute_and_apply_vocabulary_11#vocabulary#temporary_analyzer_output#vocab_compute_and_apply_vocabulary_11_vocabulary_unpruned_vocab_size]/ToTensorBinding [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:12.351Z: JOB_MESSAGE_BASIC: Executing operation Analyze/CreateSavedModel[tf_v2_only]/ReplaceWithConstants/View-python_side_input34 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:12.407Z: JOB_MESSAGE_BASIC: Finished operation Analyze/CreateSavedModel[tf_v2_only]/ReplaceWithConstants/View-python_side_input34 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:20.320Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/WriteToText/Write/WriteImpl/GroupByKey/Read+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/WriteToText/Write/WriteImpl/WriteBundles [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:20.463Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/WriteToText/Write/WriteImpl/FinalizeWrite/View-python_side_input1 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:20.490Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/WriteToText/Write/WriteImpl/PreFinalize/View-python_side_input1 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:20.527Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/WriteToText/Write/WriteImpl/FinalizeWrite/View-python_side_input1 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:20.548Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/WriteToText/Write/WriteImpl/PreFinalize/View-python_side_input1 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:20.660Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/WriteToText/Write/WriteImpl/PreFinalize [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:22.522Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/WriteToText/Write/WriteImpl/PreFinalize [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:22.655Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/WriteToText/Write/WriteImpl/FinalizeWrite/View-python_side_input2 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:22.673Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/Prepare/Impulse+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/Prepare/FlatMap(<lambda at core.py:3736>)+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/Prepare/Map(decode)+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/OrderElements+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/WriteToText/Write/WriteImpl/Map(<lambda at iobase.py:1144>)+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/WriteToText/Write/WriteImpl/WindowInto(WindowIntoFn)+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/WriteToText/Write/WriteImpl/GroupByKey/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:22.719Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/WriteToText/Write/WriteImpl/FinalizeWrite/View-python_side_input2 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:22.741Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/WriteToText/Write/WriteImpl/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:22.824Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/WriteToText/Write/WriteImpl/FinalizeWrite [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:23.531Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/WriteToText/Write/WriteImpl/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:23.617Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/WriteToText/Write/WriteImpl/GroupByKey/Read+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/WriteToText/Write/WriteImpl/WriteBundles [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:25.342Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/WriteToText/Write/WriteImpl/FinalizeWrite [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:25.482Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/WaitForVocabularyFile/View-python_side_input0 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:25.530Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/WaitForVocabularyFile/View-python_side_input0 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:25.651Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/CreatePath/Impulse+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/CreatePath/FlatMap(<lambda at core.py:3736>)+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/CreatePath/Map(decode)+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/WaitForVocabularyFile+Analyze/CreateTensorBinding[compute_and_apply_vocabulary_15#vocabulary#temporary_analyzer_output_2#Const]/ToTensorBinding [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:26.614Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/WriteToText/Write/WriteImpl/GroupByKey/Read+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/WriteToText/Write/WriteImpl/WriteBundles [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:26.720Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/WriteToText/Write/WriteImpl/FinalizeWrite/View-python_side_input1 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:26.743Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/WriteToText/Write/WriteImpl/PreFinalize/View-python_side_input1 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:26.771Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/WriteToText/Write/WriteImpl/FinalizeWrite/View-python_side_input1 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:26.795Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/WriteToText/Write/WriteImpl/PreFinalize/View-python_side_input1 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:26.905Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/CreatePath/Impulse+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/CreatePath/FlatMap(<lambda at core.py:3736>)+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/CreatePath/Map(decode)+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_15#vocabulary]/WaitForVocabularyFile+Analyze/CreateTensorBinding[compute_and_apply_vocabulary_15#vocabulary#temporary_analyzer_output_2#Const]/ToTensorBinding [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:26.921Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/WriteToText/Write/WriteImpl/PreFinalize [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:27.029Z: JOB_MESSAGE_BASIC: Executing operation Analyze/CreateSavedModel[tf_v2_only]/ReplaceWithConstants/View-python_side_input48 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:27.077Z: JOB_MESSAGE_BASIC: Finished operation Analyze/CreateSavedModel[tf_v2_only]/ReplaceWithConstants/View-python_side_input48 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:28.623Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/WriteToText/Write/WriteImpl/PreFinalize [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:28.786Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/WriteToText/Write/WriteImpl/FinalizeWrite/View-python_side_input2 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:28.838Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/WriteToText/Write/WriteImpl/FinalizeWrite/View-python_side_input2 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:28.935Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/WriteToText/Write/WriteImpl/FinalizeWrite [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:31.177Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/WriteToText/Write/WriteImpl/FinalizeWrite [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:31.292Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/WaitForVocabularyFile/View-python_side_input0 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:31.349Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/WaitForVocabularyFile/View-python_side_input0 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:31.476Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/CreatePath/Impulse+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/CreatePath/FlatMap(<lambda at core.py:3736>)+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/CreatePath/Map(decode)+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/WaitForVocabularyFile+Analyze/CreateTensorBinding[compute_and_apply_vocabulary_11#vocabulary#temporary_analyzer_output_2#Const]/ToTensorBinding [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:32.788Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/CreatePath/Impulse+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/CreatePath/FlatMap(<lambda at core.py:3736>)+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/CreatePath/Map(decode)+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_11#vocabulary]/WaitForVocabularyFile+Analyze/CreateTensorBinding[compute_and_apply_vocabulary_11#vocabulary#temporary_analyzer_output_2#Const]/ToTensorBinding [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:32.976Z: JOB_MESSAGE_BASIC: Executing operation Analyze/CreateSavedModel[tf_v2_only]/ReplaceWithConstants/View-python_side_input36 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:10:33.030Z: JOB_MESSAGE_BASIC: Finished operation Analyze/CreateSavedModel[tf_v2_only]/ReplaceWithConstants/View-python_side_input36 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:15:55.559Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyAccumulate[compute_and_apply_vocabulary_2#vocabulary]/CountPerToken/CombinePerKey(CountCombineFn)/GroupByKey/Read+Analyze/VocabularyAccumulate[compute_and_apply_vocabulary_2#vocabulary]/CountPerToken/CombinePerKey(CountCombineFn)/Combine+Analyze/VocabularyAccumulate[compute_and_apply_vocabulary_2#vocabulary]/CountPerToken/CombinePerKey(CountCombineFn)/Combine/Extract+Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/MergeCountPerToken/GroupByKey+Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/MergeCountPerToken/Combine/Partial+Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/MergeCountPerToken/GroupByKey/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:15:55.613Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/MergeCountPerToken/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:15:55.665Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/MergeCountPerToken/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:15:55.708Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:15:55.725Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:15:55.868Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:15:55.878Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:15:55.999Z: JOB_MESSAGE_BASIC: Executing operation Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/MergeCountPerToken/GroupByKey/Read+Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/MergeCountPerToken/Combine+Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/MergeCountPerToken/Combine/Extract+Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/SwapTokensAndCounts/KvSwap+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/KeyWithVoid+Analyze/VocabularyPrune[compute_and_apply_vocabulary_2#vocabulary]/ApplyThresholdsAndTopK/FilterByThresholds(5.0)+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write+Analyze/VocabularyPrune[compute_and_apply_vocabulary_2#vocabulary]/ApplyThresholdsAndTopK/FlattenToSingleMetric+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/KeyWithVoid+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_2#vocabulary]/BatchAndPreSort/BatchVocabulary/ParDo(_GlobalWindowsBatchingDoFn)+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_2#vocabulary]/BatchAndPreSort/SortBatches [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:45:36.052Z: JOB_MESSAGE_BASIC: Finished operation Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/MergeCountPerToken/GroupByKey/Read+Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/MergeCountPerToken/Combine+Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/MergeCountPerToken/Combine/Extract+Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/SwapTokensAndCounts/KvSwap+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/KeyWithVoid+Analyze/VocabularyPrune[compute_and_apply_vocabulary_2#vocabulary]/ApplyThresholdsAndTopK/FilterByThresholds(5.0)+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write+Analyze/VocabularyPrune[compute_and_apply_vocabulary_2#vocabulary]/ApplyThresholdsAndTopK/FlattenToSingleMetric+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/KeyWithVoid+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_2#vocabulary]/BatchAndPreSort/BatchVocabulary/ParDo(_GlobalWindowsBatchingDoFn)+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_2#vocabulary]/BatchAndPreSort/SortBatches [1m[31mERROR [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:208 2023-09-17T01:45:36.143Z: JOB_MESSAGE_ERROR: Workflow failed. Causes: S751:Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/MergeCountPerToken/GroupByKey/Read+Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/MergeCountPerToken/Combine+Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/MergeCountPerToken/Combine/Extract+Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/SwapTokensAndCounts/KvSwap+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/KeyWithVoid+Analyze/VocabularyPrune[compute_and_apply_vocabulary_2#vocabulary]/ApplyThresholdsAndTopK/FilterByThresholds(5.0)+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write+Analyze/VocabularyPrune[compute_and_apply_vocabulary_2#vocabulary]/ApplyThresholdsAndTopK/FlattenToSingleMetric+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/KeyWithVoid+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_2#vocabulary]/BatchAndPreSort/BatchVocabulary/ParDo(_GlobalWindowsBatchingDoFn)+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_2#vocabulary]/BatchAndPreSort/SortBatches failed., The job failed because a work item has failed 4 times. Look in previous log entries for the cause of each one of the 4 failures. If the logs only contain generic timeout errors related to accessing external resources, such as MongoDB, verify that the worker service account has permission to access the resource's subnetwork. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors. The work item was attempted on these workers: Root cause: Timed out waiting for an update from the worker. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors#worker-lost-contact. Worker ID: beamapp-jenkins-091700492-09161749-leii-harness-1hqz, Root cause: Timed out waiting for an update from the worker. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors#worker-lost-contact. Worker ID: beamapp-jenkins-091700492-09161749-leii-harness-pw5f, Root cause: Timed out waiting for an update from the worker. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors#worker-lost-contact. Worker ID: beamapp-jenkins-091700492-09161749-leii-harness-3rcc, Root cause: Timed out waiting for an update from the worker. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors#worker-lost-contact. Worker ID: beamapp-jenkins-091700492-09161749-leii-harness-q6k7 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:45:36.934Z: JOB_MESSAGE_BASIC: Stopping worker pool... [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:204 2023-09-17T01:48:19.848Z: JOB_MESSAGE_BASIC: Worker pool stopped. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:154 Job 2023-09-16_17_49_36-11999137878165459442 is in state JOB_STATE_FAILED [1m[31mERROR [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:766 Console URL: https://console.cloud.google.com/dataflow/jobs/<RegionId>/2023-09-16_17_49_36-11999137878165459442?project=<ProjectId> [33m=============================== warnings summary ===============================[0m ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:28 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:28 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:28 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:28 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:28 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:28 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:28 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:28 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:28 <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py>:28: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses from imp import load_source ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:18 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:18 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:18 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:18 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:18 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:18 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:18 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:18 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:18 <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py>:18: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html import pkg_resources ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2871: 180 warnings <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py>:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2871: 144 warnings <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py>:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.cloud')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2350: 36 warnings <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py>:2350: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(parent) ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2871 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2871 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2871 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2871 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2871 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2871 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2871 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2871 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2871 <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py>:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.logging')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2871 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2871 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2871 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2871 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2871 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2871 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2871 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2871 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py:2871 <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/pkg_resources/__init__.py>:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.iam')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:20 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:20 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:20 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:20 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:20 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:20 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:20 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:20 ../../build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py:20 <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/google/rpc/__init__.py>:20: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.rpc')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages pkg_resources.declare_namespace(__name__) apache_beam/examples/ml_transform/ml_transform_it_test.py::LargeMovieReviewDatasetProcessTest::test_process_large_movie_review_dataset apache_beam/testing/benchmarks/cloudml/cloudml_benchmark_test.py::CloudMLTFTBenchmarkTest::test_cloudml_benchmark_criteo_10GB apache_beam/testing/benchmarks/cloudml/cloudml_benchmark_test.py::CloudMLTFTBenchmarkTest::test_cloudml_benchmark_criteo_small <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py>:484: PendingDeprecationWarning: Client.dataset is deprecated and will be removed in a future version. Use a string like 'my_project.my_dataset' or a cloud.google.bigquery.DatasetReference object, instead. bq_dataset_ref = self._client.dataset(dataset_name) -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html - generated xml file: <https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/sdks/python/pytest_TFTransformTests-df-py39.xml> - [36m[1m=========================== short test summary info ============================[0m [31mFAILED[0m apache_beam/testing/benchmarks/cloudml/cloudml_benchmark_test.py::[1mCloudMLTFTBenchmarkTest::test_cloudml_benchmark_cirteo_no_shuffle_10GB[0m - apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error: Workflow failed. Causes: S751:Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/MergeCountPerToken/GroupByKey/Read+Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/MergeCountPerToken/Combine+Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/MergeCountPerToken/Combine/Extract+Analyze/VocabularyMerge[compute_and_apply_vocabulary_2#vocabulary]/SwapTokensAndCounts/KvSwap+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/KeyWithVoid+Analyze/VocabularyPrune[compute_and_apply_vocabulary_2#vocabulary]/ApplyThresholdsAndTopK/FilterByThresholds(5.0)+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+Analyze/VocabularyCountUnfiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write+Analyze/VocabularyPrune[compute_and_apply_vocabulary_2#vocabulary]/ApplyThresholdsAndTopK/FlattenToSingleMetric+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/KeyWithVoid+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_2#vocabulary]/BatchAndPreSort/BatchVocabulary/ParDo(_GlobalWindowsBatchingDoFn)+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+Analyze/VocabularyCountFiltered[compute_and_apply_vocabulary_2#vocabulary]/TotalVocabSize/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write+Analyze/VocabularyOrderAndWrite[compute_and_apply_vocabulary_2#vocabulary]/BatchAndPreSort/SortBatches failed., The job failed because a work item has failed 4 times. Look in previous log entries for the cause of each one of the 4 failures. If the logs only contain generic timeout errors related to accessing external resources, such as MongoDB, verify that the worker service account has permission to access the resource's subnetwork. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors. The work item was attempted on these workers: Root cause: Timed out waiting for an update from the worker. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors#worker-lost-contact. Worker ID: beamapp-jenkins-091700492-09161749-leii-harness-1hqz, Root cause: Timed out waiting for an update from the worker. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors#worker-lost-contact. Worker ID: beamapp-jenkins-091700492-09161749-leii-harness-pw5f, Root cause: Timed out waiting for an update from the worker. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors#worker-lost-contact. Worker ID: beamapp-jenkins-091700492-09161749-leii-harness-3rcc, Root cause: Timed out waiting for an update from the worker. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors#worker-lost-contact. Worker ID: beamapp-jenkins-091700492-09161749-leii-harness-q6k7 [31m====== [31m[1m1 failed[0m, [32m3 passed[0m, [33m10 skipped[0m, [33m408 warnings[0m[31m in 6763.38s (1:52:43)[0m[31m ======[0m > Task :sdks:python:test-suites:dataflow:py39:tftTests FAILED FAILURE: Build failed with an exception. * Where: Script '<https://ci-beam.apache.org/job/beam_CloudML_Benchmarks_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 459 * What went wrong: Execution failed for task ':sdks:python:test-suites:dataflow:py39:tftTests'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0. You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins. See https://docs.gradle.org/7.6.2/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 1h 55m 37s 16 actionable tasks: 10 executed, 4 from cache, 2 up-to-date Publishing build scan... Publishing build scan failed due to network error 'java.net.SocketTimeoutException: Read timed out' (2 retries remaining)... Publishing build scan failed due to 'An unexpected error occurred creating your build scan. Please report this problem at https://ge.apache.org/help, quoting the following reference code: 67k456eguv4fi' (2 retries remaining)... Publishing build scan failed due to network error 'java.net.SocketTimeoutException: Read timed out' (2 retries remaining)... Publishing build scan failed due to network error 'java.net.SocketTimeoutException: Read timed out' (1 retry remaining)... Publishing failed. The response from https://ge.apache.org/scans/publish/gradle/3.13.2/upload was not from Gradle Enterprise. The specified server address may be incorrect, or your network environment may be interfering. Please report this problem to your Gradle Enterprise administrator via https://ge.apache.org/help and include the following via copy/paste: ---------- Gradle version: 7.6.2 Plugin version: 3.13.2 Request URL: https://ge.apache.org/scans/publish/gradle/3.13.2/upload Request ID: 7da9f628-0e5a-43e2-a867-dcf26b8ee724 Response status code: 502 Response content type: text/html ---------- Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
