See <https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/814/display/redirect?page=changes>
Changes: [Robert Bradshaw] [BEAM-22923] Allow sharding specification for dataframe writes. [noreply] Add set state in Go (#22919) [noreply] Go Map State integration test (#22898) [noreply] Add clear function for bag state types (#22917) [noreply] [Playground] Update build_playground_backend.yml - add "Index creation" [noreply] [Playground] [Backend] added SDK validation to save a code snippet [noreply] Fix linting violations (#22934) [noreply] [akvelon][tour-of-beam] backend bootstraps (#22556) ------------------------------------------ [...truncated 48.77 KB...] >>> collect markers: -m=examples_postcommit and no_xdist and not >>> sickbay_dataflow [1m============================= test session starts ==============================[0m platform linux -- Python 3.9.10, pytest-7.1.2, pluggy-1.0.0 rootdir: /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Examples_Dataflow/src/sdks/python, configfile: pytest.ini plugins: xdist-2.5.0, timeout-2.1.0, forked-1.4.0, requests-mock-1.9.3 timeout: 4500.0s timeout method: signal timeout func_only: False [1m----------------------------- live log collection ------------------------------[0m [33mWARNING [0m root:avroio_test.py:54 python-snappy is not installed; some tests will be skipped. [33mWARNING [0m root:tfrecordio_test.py:55 Tensorflow is not installed, so skipping some tests. [33mWARNING [0m apache_beam.runners.interactive.interactive_environment:interactive_environment.py:190 Dependencies required for Interactive Beam PCollection visualization are not available, please use: `pip install apache-beam[interactive]` to install necessary dependencies to enable all data visualization features. [33mWARNING [0m apache_beam.runners.interactive.interactive_environment:interactive_environment.py:199 You cannot use Interactive Beam features when you are not in an interactive environment such as a Jupyter notebook or ipython terminal. [32mINFO [0m root:environments.py:376 Default Python SDK image for environment is apache/beam_python3.9_sdk:2.42.0.dev collected 6553 items / 6547 deselected / 2 skipped / 6 selected apache_beam/examples/complete/autocomplete_it_test.py::AutocompleteIT::test_autocomplete_output_files_on_small_input [1m-------------------------------- live log call ---------------------------------[0m [32mINFO [0m root:autocomplete_it_test.py:48 Creating file: gs://temp-storage-for-end-to-end-tests/py-it-cloud/input/564b5661-16a0-4307-adbf-db5160e81f69/input.txt [32mINFO [0m apache_beam.internal.gcp.auth:auth.py:136 Setting socket default timeout to 60 seconds. [32mINFO [0m apache_beam.internal.gcp.auth:auth.py:138 socket default timeout is 60.0 seconds. [32mINFO [0m apache_beam.runners.portability.stager:stager.py:754 Executing command: ['/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Examples_Dataflow/src/build/gradleenv/-1734967050/bin/python3.9', '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/tmp/tmpekasrtz_/tmp_requirements.txt', '--exists-action', 'i', '--no-deps', '--implementation', 'cp', '--abi', 'cp39', '--platform', 'manylinux2014_x86_64'] [32mINFO [0m apache_beam.runners.portability.stager:stager.py:325 Copying Beam SDK "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Examples_Dataflow/src/sdks/python/build/apache-beam.tar.gz" to staging location. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:477 Pipeline has additional dependencies to be installed in SDK worker container, consider using the SDK container image pre-building workflow to avoid repetitive installations. Learn more on https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild [32mINFO [0m root:environments.py:376 Default Python SDK image for environment is apache/beam_python3.9_sdk:2.42.0.dev [32mINFO [0m root:environments.py:295 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python39-fnapi:beam-master-20220811 [32mINFO [0m root:environments.py:302 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python39-fnapi:beam-master-20220811" for Docker environment [32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function pack_combiners at 0x7f5f0b71e1f0> ==================== [32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function sort_stages at 0x7f5f0b71e9d0> ==================== [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/requirements.txt... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/requirements.txt in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/pickled_main_session... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/pickled_main_session in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/mock-2.0.0-py2.py3-none-any.whl... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/mock-2.0.0-py2.py3-none-any.whl in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/seaborn-0.11.2-py3-none-any.whl... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/seaborn-0.11.2-py3-none-any.whl in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/PyHamcrest-1.10.1-py3-none-any.whl... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/PyHamcrest-1.10.1-py3-none-any.whl in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/transformers-4.21.1-py3-none-any.whl... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/transformers-4.21.1-py3-none-any.whl in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/beautifulsoup4-4.11.1-py3-none-any.whl... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/beautifulsoup4-4.11.1-py3-none-any.whl in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/parameterized-0.7.5-py2.py3-none-any.whl... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/parameterized-0.7.5-py2.py3-none-any.whl in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/torch-1.12.1-cp37-cp37m-manylinux1_x86_64.whl... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/torch-1.12.1-cp37-cp37m-manylinux1_x86_64.whl in 32 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/torchvision-0.13.1-cp37-cp37m-manylinux1_x86_64.whl... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/torchvision-0.13.1-cp37-cp37m-manylinux1_x86_64.whl in 1 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/matplotlib-3.5.2-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/matplotlib-3.5.2-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl in 2 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/Pillow-9.2.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/Pillow-9.2.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/matplotlib-3.5.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/matplotlib-3.5.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/matplotlib-3.5.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/matplotlib-3.5.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/dataflow_python_sdk.tar... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/dataflow_python_sdk.tar in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/pipeline.pb... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829192256-534019-zxz6xnth.1661800976.534329/pipeline.pb in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:911 Create job: <Job clientRequestId: '20220829192256536003-5768' createTime: '2022-08-29T19:23:38.621289Z' currentStateTime: '1970-01-01T00:00:00Z' id: '2022-08-29_12_23_38-16433102854740846745' location: 'us-central1' name: 'beamapp-jenkins-0829192256-534019-zxz6xnth' projectId: 'apache-beam-testing' stageStates: [] startTime: '2022-08-29T19:23:38.621289Z' steps: [] tempFiles: [] type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)> [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:913 Created job with id: [2022-08-29_12_23_38-16433102854740846745] [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:914 Submitted job: 2022-08-29_12_23_38-16433102854740846745 [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:915 To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-08-29_12_23_38-16433102854740846745?project=apache-beam-testing Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2022-08-29_12_23_38-16433102854740846745?project=apache-beam-testing [32mINFO [0m apache_beam.runners.dataflow.test_dataflow_runner:test_dataflow_runner.py:58 Console log: [32mINFO [0m apache_beam.runners.dataflow.test_dataflow_runner:test_dataflow_runner.py:59 https://console.cloud.google.com/dataflow/jobs/us-central1/2022-08-29_12_23_38-16433102854740846745?project=apache-beam-testing [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:197 Job 2022-08-29_12_23_38-16433102854740846745 is in state JOB_STATE_RUNNING [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:39.308Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2022-08-29_12_23_38-16433102854740846745. The number of workers will be between 1 and 1000. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:39.712Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2022-08-29_12_23_38-16433102854740846745. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:42.426Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:44.241Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:44.276Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:44.333Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:44.373Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:44.495Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:44.531Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:44.607Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:44.706Z: JOB_MESSAGE_DETAILED: Fusing consumer write/Write/WriteImpl/InitializeWrite into write/Write/WriteImpl/DoOnce/Map(decode) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:44.739Z: JOB_MESSAGE_DETAILED: Fusing consumer read/Read/Map(<lambda at iobase.py:908>) into read/Read/Impulse [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:44.779Z: JOB_MESSAGE_DETAILED: Fusing consumer ref_AppliedPTransform_read-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/PairWithRestriction into read/Read/Map(<lambda at iobase.py:908>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:44.805Z: JOB_MESSAGE_DETAILED: Fusing consumer ref_AppliedPTransform_read-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/SplitWithSizing into ref_AppliedPTransform_read-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/PairWithRestriction [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:44.834Z: JOB_MESSAGE_DETAILED: Fusing consumer split into ref_AppliedPTransform_read-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/ProcessElementAndRestrictionWithSizing [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:44.870Z: JOB_MESSAGE_DETAILED: Fusing consumer TopPerPrefix/PerElement/PerElement:PairWithVoid into split [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:44.907Z: JOB_MESSAGE_DETAILED: Fusing consumer TopPerPrefix/PerElement/CombinePerKey(CountCombineFn)/GroupByKey+TopPerPrefix/PerElement/CombinePerKey(CountCombineFn)/Combine/Partial into TopPerPrefix/PerElement/PerElement:PairWithVoid [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:44.949Z: JOB_MESSAGE_DETAILED: Fusing consumer TopPerPrefix/PerElement/CombinePerKey(CountCombineFn)/GroupByKey/Write into TopPerPrefix/PerElement/CombinePerKey(CountCombineFn)/GroupByKey+TopPerPrefix/PerElement/CombinePerKey(CountCombineFn)/Combine/Partial [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:44.989Z: JOB_MESSAGE_DETAILED: Fusing consumer TopPerPrefix/PerElement/CombinePerKey(CountCombineFn)/Combine into TopPerPrefix/PerElement/CombinePerKey(CountCombineFn)/GroupByKey/Read [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:45.034Z: JOB_MESSAGE_DETAILED: Fusing consumer TopPerPrefix/PerElement/CombinePerKey(CountCombineFn)/Combine/Extract into TopPerPrefix/PerElement/CombinePerKey(CountCombineFn)/Combine [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:45.069Z: JOB_MESSAGE_DETAILED: Fusing consumer TopPerPrefix/FlatMap(extract_prefixes) into TopPerPrefix/PerElement/CombinePerKey(CountCombineFn)/Combine/Extract [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:45.106Z: JOB_MESSAGE_DETAILED: Fusing consumer TopPerPrefix/LargestPerKey(5)/TopPerKey(5)/CombinePerKey(TopCombineFn)/GroupByKey+TopPerPrefix/LargestPerKey(5)/TopPerKey(5)/CombinePerKey(TopCombineFn)/Combine/Partial into TopPerPrefix/FlatMap(extract_prefixes) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:45.144Z: JOB_MESSAGE_DETAILED: Fusing consumer TopPerPrefix/LargestPerKey(5)/TopPerKey(5)/CombinePerKey(TopCombineFn)/GroupByKey/Write into TopPerPrefix/LargestPerKey(5)/TopPerKey(5)/CombinePerKey(TopCombineFn)/GroupByKey+TopPerPrefix/LargestPerKey(5)/TopPerKey(5)/CombinePerKey(TopCombineFn)/Combine/Partial [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:45.186Z: JOB_MESSAGE_DETAILED: Fusing consumer TopPerPrefix/LargestPerKey(5)/TopPerKey(5)/CombinePerKey(TopCombineFn)/Combine into TopPerPrefix/LargestPerKey(5)/TopPerKey(5)/CombinePerKey(TopCombineFn)/GroupByKey/Read [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:45.219Z: JOB_MESSAGE_DETAILED: Fusing consumer TopPerPrefix/LargestPerKey(5)/TopPerKey(5)/CombinePerKey(TopCombineFn)/Combine/Extract into TopPerPrefix/LargestPerKey(5)/TopPerKey(5)/CombinePerKey(TopCombineFn)/Combine [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:45.260Z: JOB_MESSAGE_DETAILED: Fusing consumer format into TopPerPrefix/LargestPerKey(5)/TopPerKey(5)/CombinePerKey(TopCombineFn)/Combine/Extract [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:45.288Z: JOB_MESSAGE_DETAILED: Fusing consumer write/Write/WriteImpl/WindowInto(WindowIntoFn) into format [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:45.316Z: JOB_MESSAGE_DETAILED: Fusing consumer write/Write/WriteImpl/DoOnce/FlatMap(<lambda at core.py:3481>) into write/Write/WriteImpl/DoOnce/Impulse [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:45.342Z: JOB_MESSAGE_DETAILED: Fusing consumer write/Write/WriteImpl/DoOnce/Map(decode) into write/Write/WriteImpl/DoOnce/FlatMap(<lambda at core.py:3481>) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:45.469Z: JOB_MESSAGE_DETAILED: Fusing consumer write/Write/WriteImpl/WriteBundles into write/Write/WriteImpl/WindowInto(WindowIntoFn) [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:45.535Z: JOB_MESSAGE_DETAILED: Fusing consumer write/Write/WriteImpl/Pair into write/Write/WriteImpl/WriteBundles [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:45.579Z: JOB_MESSAGE_DETAILED: Fusing consumer write/Write/WriteImpl/GroupByKey/Write into write/Write/WriteImpl/Pair [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:45.617Z: JOB_MESSAGE_DETAILED: Fusing consumer write/Write/WriteImpl/Extract into write/Write/WriteImpl/GroupByKey/Read [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:45.734Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:45.771Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:45.809Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:45.842Z: JOB_MESSAGE_DEBUG: Assigning stage ids. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:46.060Z: JOB_MESSAGE_DEBUG: Executing wait step start49 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:46.132Z: JOB_MESSAGE_BASIC: Executing operation write/Write/WriteImpl/DoOnce/Impulse+write/Write/WriteImpl/DoOnce/FlatMap(<lambda at core.py:3481>)+write/Write/WriteImpl/DoOnce/Map(decode)+write/Write/WriteImpl/InitializeWrite [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:46.171Z: JOB_MESSAGE_BASIC: Executing operation read/Read/Impulse+read/Read/Map(<lambda at iobase.py:908>)+ref_AppliedPTransform_read-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/PairWithRestriction+ref_AppliedPTransform_read-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/SplitWithSizing [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:46.185Z: JOB_MESSAGE_DEBUG: Starting worker pool setup. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:46.205Z: JOB_MESSAGE_BASIC: Executing operation TopPerPrefix/LargestPerKey(5)/TopPerKey(5)/CombinePerKey(TopCombineFn)/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:46.227Z: JOB_MESSAGE_BASIC: Starting 1 workers in us-central1-a... [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:46.242Z: JOB_MESSAGE_BASIC: Executing operation TopPerPrefix/PerElement/CombinePerKey(CountCombineFn)/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:46.270Z: JOB_MESSAGE_BASIC: Executing operation write/Write/WriteImpl/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:46.303Z: JOB_MESSAGE_BASIC: Finished operation TopPerPrefix/LargestPerKey(5)/TopPerKey(5)/CombinePerKey(TopCombineFn)/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:46.303Z: JOB_MESSAGE_BASIC: Finished operation TopPerPrefix/PerElement/CombinePerKey(CountCombineFn)/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:46.334Z: JOB_MESSAGE_BASIC: Finished operation write/Write/WriteImpl/GroupByKey/Create [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:46.385Z: JOB_MESSAGE_DEBUG: Value "TopPerPrefix/PerElement/CombinePerKey(CountCombineFn)/GroupByKey/Session" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:46.472Z: JOB_MESSAGE_DEBUG: Value "TopPerPrefix/LargestPerKey(5)/TopPerKey(5)/CombinePerKey(TopCombineFn)/GroupByKey/Session" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:23:46.507Z: JOB_MESSAGE_DEBUG: Value "write/Write/WriteImpl/GroupByKey/Session" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:24:13.181Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:24:23.513Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running stage(s). [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:24:43.609Z: JOB_MESSAGE_DETAILED: Workers have started successfully. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:10.048Z: JOB_MESSAGE_BASIC: Finished operation write/Write/WriteImpl/DoOnce/Impulse+write/Write/WriteImpl/DoOnce/FlatMap(<lambda at core.py:3481>)+write/Write/WriteImpl/DoOnce/Map(decode)+write/Write/WriteImpl/InitializeWrite [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:10.134Z: JOB_MESSAGE_DEBUG: Value "write/Write/WriteImpl/DoOnce/Map(decode).None" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:10.167Z: JOB_MESSAGE_DEBUG: Value "write/Write/WriteImpl/InitializeWrite.None" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:10.245Z: JOB_MESSAGE_BASIC: Executing operation write/Write/WriteImpl/FinalizeWrite/View-python_side_input0-write/Write/WriteImpl/FinalizeWrite [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:10.285Z: JOB_MESSAGE_BASIC: Executing operation write/Write/WriteImpl/WriteBundles/View-python_side_input0-write/Write/WriteImpl/WriteBundles [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:10.308Z: JOB_MESSAGE_BASIC: Finished operation write/Write/WriteImpl/FinalizeWrite/View-python_side_input0-write/Write/WriteImpl/FinalizeWrite [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:10.319Z: JOB_MESSAGE_BASIC: Executing operation write/Write/WriteImpl/PreFinalize/View-python_side_input0-write/Write/WriteImpl/PreFinalize [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:10.345Z: JOB_MESSAGE_BASIC: Finished operation write/Write/WriteImpl/WriteBundles/View-python_side_input0-write/Write/WriteImpl/WriteBundles [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:10.376Z: JOB_MESSAGE_DEBUG: Value "write/Write/WriteImpl/FinalizeWrite/View-python_side_input0-write/Write/WriteImpl/FinalizeWrite.out" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:10.376Z: JOB_MESSAGE_BASIC: Finished operation read/Read/Impulse+read/Read/Map(<lambda at iobase.py:908>)+ref_AppliedPTransform_read-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/PairWithRestriction+ref_AppliedPTransform_read-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/SplitWithSizing [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:10.380Z: JOB_MESSAGE_BASIC: Finished operation write/Write/WriteImpl/PreFinalize/View-python_side_input0-write/Write/WriteImpl/PreFinalize [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:10.422Z: JOB_MESSAGE_DEBUG: Value "write/Write/WriteImpl/WriteBundles/View-python_side_input0-write/Write/WriteImpl/WriteBundles.out" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:10.458Z: JOB_MESSAGE_DEBUG: Value "ref_AppliedPTransform_read-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7-split-with-sizing-out3" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:10.526Z: JOB_MESSAGE_DEBUG: Value "write/Write/WriteImpl/PreFinalize/View-python_side_input0-write/Write/WriteImpl/PreFinalize.out" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:10.561Z: JOB_MESSAGE_BASIC: Executing operation ref_AppliedPTransform_read-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/ProcessElementAndRestrictionWithSizing+split+TopPerPrefix/PerElement/PerElement:PairWithVoid+TopPerPrefix/PerElement/CombinePerKey(CountCombineFn)/GroupByKey+TopPerPrefix/PerElement/CombinePerKey(CountCombineFn)/Combine/Partial+TopPerPrefix/PerElement/CombinePerKey(CountCombineFn)/GroupByKey/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:17.233Z: JOB_MESSAGE_BASIC: Finished operation ref_AppliedPTransform_read-Read-SDFBoundedSourceReader-ParDo-SDFBoundedSourceDoFn-_7/ProcessElementAndRestrictionWithSizing+split+TopPerPrefix/PerElement/PerElement:PairWithVoid+TopPerPrefix/PerElement/CombinePerKey(CountCombineFn)/GroupByKey+TopPerPrefix/PerElement/CombinePerKey(CountCombineFn)/Combine/Partial+TopPerPrefix/PerElement/CombinePerKey(CountCombineFn)/GroupByKey/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:17.310Z: JOB_MESSAGE_BASIC: Executing operation TopPerPrefix/PerElement/CombinePerKey(CountCombineFn)/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:17.382Z: JOB_MESSAGE_BASIC: Finished operation TopPerPrefix/PerElement/CombinePerKey(CountCombineFn)/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:17.517Z: JOB_MESSAGE_BASIC: Executing operation TopPerPrefix/PerElement/CombinePerKey(CountCombineFn)/GroupByKey/Read+TopPerPrefix/PerElement/CombinePerKey(CountCombineFn)/Combine+TopPerPrefix/PerElement/CombinePerKey(CountCombineFn)/Combine/Extract+TopPerPrefix/FlatMap(extract_prefixes)+TopPerPrefix/LargestPerKey(5)/TopPerKey(5)/CombinePerKey(TopCombineFn)/GroupByKey+TopPerPrefix/LargestPerKey(5)/TopPerKey(5)/CombinePerKey(TopCombineFn)/Combine/Partial+TopPerPrefix/LargestPerKey(5)/TopPerKey(5)/CombinePerKey(TopCombineFn)/GroupByKey/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:18.978Z: JOB_MESSAGE_BASIC: Finished operation TopPerPrefix/PerElement/CombinePerKey(CountCombineFn)/GroupByKey/Read+TopPerPrefix/PerElement/CombinePerKey(CountCombineFn)/Combine+TopPerPrefix/PerElement/CombinePerKey(CountCombineFn)/Combine/Extract+TopPerPrefix/FlatMap(extract_prefixes)+TopPerPrefix/LargestPerKey(5)/TopPerKey(5)/CombinePerKey(TopCombineFn)/GroupByKey+TopPerPrefix/LargestPerKey(5)/TopPerKey(5)/CombinePerKey(TopCombineFn)/Combine/Partial+TopPerPrefix/LargestPerKey(5)/TopPerKey(5)/CombinePerKey(TopCombineFn)/GroupByKey/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:19.084Z: JOB_MESSAGE_BASIC: Executing operation TopPerPrefix/LargestPerKey(5)/TopPerKey(5)/CombinePerKey(TopCombineFn)/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:19.151Z: JOB_MESSAGE_BASIC: Finished operation TopPerPrefix/LargestPerKey(5)/TopPerKey(5)/CombinePerKey(TopCombineFn)/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:19.229Z: JOB_MESSAGE_BASIC: Executing operation TopPerPrefix/LargestPerKey(5)/TopPerKey(5)/CombinePerKey(TopCombineFn)/GroupByKey/Read+TopPerPrefix/LargestPerKey(5)/TopPerKey(5)/CombinePerKey(TopCombineFn)/Combine+TopPerPrefix/LargestPerKey(5)/TopPerKey(5)/CombinePerKey(TopCombineFn)/Combine/Extract+format+write/Write/WriteImpl/WindowInto(WindowIntoFn)+write/Write/WriteImpl/WriteBundles+write/Write/WriteImpl/Pair+write/Write/WriteImpl/GroupByKey/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:21.089Z: JOB_MESSAGE_BASIC: Finished operation TopPerPrefix/LargestPerKey(5)/TopPerKey(5)/CombinePerKey(TopCombineFn)/GroupByKey/Read+TopPerPrefix/LargestPerKey(5)/TopPerKey(5)/CombinePerKey(TopCombineFn)/Combine+TopPerPrefix/LargestPerKey(5)/TopPerKey(5)/CombinePerKey(TopCombineFn)/Combine/Extract+format+write/Write/WriteImpl/WindowInto(WindowIntoFn)+write/Write/WriteImpl/WriteBundles+write/Write/WriteImpl/Pair+write/Write/WriteImpl/GroupByKey/Write [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:21.277Z: JOB_MESSAGE_BASIC: Executing operation write/Write/WriteImpl/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:21.385Z: JOB_MESSAGE_BASIC: Finished operation write/Write/WriteImpl/GroupByKey/Close [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:21.559Z: JOB_MESSAGE_BASIC: Executing operation write/Write/WriteImpl/GroupByKey/Read+write/Write/WriteImpl/Extract [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:24.050Z: JOB_MESSAGE_BASIC: Finished operation write/Write/WriteImpl/GroupByKey/Read+write/Write/WriteImpl/Extract [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:24.129Z: JOB_MESSAGE_DEBUG: Value "write/Write/WriteImpl/Extract.None" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:24.216Z: JOB_MESSAGE_BASIC: Executing operation write/Write/WriteImpl/FinalizeWrite/View-python_side_input1-write/Write/WriteImpl/FinalizeWrite [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:24.248Z: JOB_MESSAGE_BASIC: Executing operation write/Write/WriteImpl/PreFinalize/View-python_side_input1-write/Write/WriteImpl/PreFinalize [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:24.282Z: JOB_MESSAGE_BASIC: Finished operation write/Write/WriteImpl/FinalizeWrite/View-python_side_input1-write/Write/WriteImpl/FinalizeWrite [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:24.309Z: JOB_MESSAGE_BASIC: Finished operation write/Write/WriteImpl/PreFinalize/View-python_side_input1-write/Write/WriteImpl/PreFinalize [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:24.376Z: JOB_MESSAGE_DEBUG: Value "write/Write/WriteImpl/FinalizeWrite/View-python_side_input1-write/Write/WriteImpl/FinalizeWrite.out" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:24.410Z: JOB_MESSAGE_DEBUG: Value "write/Write/WriteImpl/PreFinalize/View-python_side_input1-write/Write/WriteImpl/PreFinalize.out" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:24.524Z: JOB_MESSAGE_BASIC: Executing operation write/Write/WriteImpl/PreFinalize [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:27.561Z: JOB_MESSAGE_BASIC: Finished operation write/Write/WriteImpl/PreFinalize [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:27.687Z: JOB_MESSAGE_DEBUG: Value "write/Write/WriteImpl/PreFinalize.None" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:27.819Z: JOB_MESSAGE_BASIC: Executing operation write/Write/WriteImpl/FinalizeWrite/View-python_side_input2-write/Write/WriteImpl/FinalizeWrite [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:27.890Z: JOB_MESSAGE_BASIC: Finished operation write/Write/WriteImpl/FinalizeWrite/View-python_side_input2-write/Write/WriteImpl/FinalizeWrite [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:27.974Z: JOB_MESSAGE_DEBUG: Value "write/Write/WriteImpl/FinalizeWrite/View-python_side_input2-write/Write/WriteImpl/FinalizeWrite.out" materialized. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:28.088Z: JOB_MESSAGE_BASIC: Executing operation write/Write/WriteImpl/FinalizeWrite [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:30.582Z: JOB_MESSAGE_BASIC: Finished operation write/Write/WriteImpl/FinalizeWrite [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:30.769Z: JOB_MESSAGE_DEBUG: Executing success step success47 [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:30.877Z: JOB_MESSAGE_DETAILED: Cleaning up. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:30.946Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:30:31.031Z: JOB_MESSAGE_BASIC: Stopping worker pool... [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:32:44.690Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized worker pool from 1 to 0. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:32:44.825Z: JOB_MESSAGE_BASIC: Worker pool stopped. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242 2022-08-29T19:32:44.883Z: JOB_MESSAGE_DEBUG: Tearing down pending resources... [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:197 Job 2022-08-29_12_23_38-16433102854740846745 is in state JOB_STATE_DONE [32mINFO [0m apache_beam.io.gcp.gcsio:gcsio.py:607 Finished listing 1 files in 0.10911321640014648 seconds. [32mPASSED[0m apache_beam/examples/complete/estimate_pi_it_test.py::EstimatePiIT::test_estimate_pi_output_file [1m-------------------------------- live log call ---------------------------------[0m [32mINFO [0m apache_beam.runners.portability.stager:stager.py:754 Executing command: ['/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Examples_Dataflow/src/build/gradleenv/-1734967050/bin/python3.9', '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/tmp/tmplhd8_ftj/tmp_requirements.txt', '--exists-action', 'i', '--no-deps', '--implementation', 'cp', '--abi', 'cp39', '--platform', 'manylinux2014_x86_64'] [32mINFO [0m apache_beam.runners.portability.stager:stager.py:325 Copying Beam SDK "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Examples_Dataflow/src/sdks/python/build/apache-beam.tar.gz" to staging location. [32mINFO [0m apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:477 Pipeline has additional dependencies to be installed in SDK worker container, consider using the SDK container image pre-building workflow to avoid repetitive installations. Learn more on https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild [32mINFO [0m root:environments.py:376 Default Python SDK image for environment is apache/beam_python3.9_sdk:2.42.0.dev [32mINFO [0m root:environments.py:295 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python39-fnapi:beam-master-20220811 [32mINFO [0m root:environments.py:302 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python39-fnapi:beam-master-20220811" for Docker environment [32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function pack_combiners at 0x7f5f0b71e1f0> ==================== [32mINFO [0m apache_beam.runners.portability.fn_api_runner.translations:translations.py:714 ==================== <function sort_stages at 0x7f5f0b71e9d0> ==================== [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829193259-013303-otkribgs.1661801579.013630/requirements.txt... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829193259-013303-otkribgs.1661801579.013630/requirements.txt in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829193259-013303-otkribgs.1661801579.013630/pickled_main_session... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829193259-013303-otkribgs.1661801579.013630/pickled_main_session in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829193259-013303-otkribgs.1661801579.013630/mock-2.0.0-py2.py3-none-any.whl... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829193259-013303-otkribgs.1661801579.013630/mock-2.0.0-py2.py3-none-any.whl in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829193259-013303-otkribgs.1661801579.013630/seaborn-0.11.2-py3-none-any.whl... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829193259-013303-otkribgs.1661801579.013630/seaborn-0.11.2-py3-none-any.whl in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829193259-013303-otkribgs.1661801579.013630/PyHamcrest-1.10.1-py3-none-any.whl... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829193259-013303-otkribgs.1661801579.013630/PyHamcrest-1.10.1-py3-none-any.whl in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829193259-013303-otkribgs.1661801579.013630/transformers-4.21.1-py3-none-any.whl... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829193259-013303-otkribgs.1661801579.013630/transformers-4.21.1-py3-none-any.whl in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829193259-013303-otkribgs.1661801579.013630/beautifulsoup4-4.11.1-py3-none-any.whl... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829193259-013303-otkribgs.1661801579.013630/beautifulsoup4-4.11.1-py3-none-any.whl in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829193259-013303-otkribgs.1661801579.013630/parameterized-0.7.5-py2.py3-none-any.whl... [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829193259-013303-otkribgs.1661801579.013630/parameterized-0.7.5-py2.py3-none-any.whl in 0 seconds. [32mINFO [0m apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0829193259-013303-otkribgs.1661801579.013630/torch-1.12.1-cp37-cp37m-manylinux1_x86_64.whl... FATAL: command execution failed hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@5c56cd0e:apache-beam-jenkins-10": Remote call on apache-beam-jenkins-10 failed. The channel is closing down or has closed down at hudson.remoting.Channel.call(Channel.java:993) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:285) at com.sun.proxy.$Proxy138.isAlive(Unknown Source) at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1215) at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1207) at hudson.Launcher$ProcStarter.join(Launcher.java:524) at hudson.plugins.gradle.Gradle.perform(Gradle.java:317) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:814) at hudson.model.Build$BuildExecution.build(Build.java:199) at hudson.model.Build$BuildExecution.doRun(Build.java:164) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:522) at hudson.model.Run.execute(Run.java:1896) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44) at hudson.model.ResourceController.execute(ResourceController.java:101) at hudson.model.Executor.run(Executor.java:442) Caused by: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:75) Caused by: java.io.EOFException at java.base/java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2905) at java.base/java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3400) at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:936) at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.Command.readFrom(Command.java:128) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61) Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure ERROR: apache-beam-jenkins-10 is offline; cannot locate jdk_1.8_latest --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
