See
<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/3862/display/redirect?page=changes>
Changes:
[noreply] Fix yaml duplicated mapping key (#22952)
[noreply] [Playground] [Infrastructure] Adding the Cloud Datastore client to
save
[noreply] Fix jdbc date conversion offset 1 day (#22738)
[noreply] Set state integration test (#22935)
------------------------------------------
[...truncated 73.40 KB...]
[1m[gw0] Python 3.9.10 (main, Jan 15 2022, 18:17:56) -- [GCC 9.3.0]
[0m[1m[gw1] Python 3.9.10 (main, Jan 15 2022, 18:17:56) -- [GCC 9.3.0]
[0m[1m[gw3] Python 3.9.10 (main, Jan 15 2022, 18:17:56) -- [GCC 9.3.0]
[0m[1m[gw2] Python 3.9.10 (main, Jan 15 2022, 18:17:56) -- [GCC 9.3.0]
[0m[1m[gw7] Python 3.9.10 (main, Jan 15 2022, 18:17:56) -- [GCC 9.3.0]
[0m[1m[gw4] Python 3.9.10 (main, Jan 15 2022, 18:17:56) -- [GCC 9.3.0]
[0m[1m[gw5] Python 3.9.10 (main, Jan 15 2022, 18:17:56) -- [GCC 9.3.0]
[0m[1m[gw6] Python 3.9.10 (main, Jan 15 2022, 18:17:56) -- [GCC 9.3.0]
[0mgw0 [33] / gw1 [33] / gw2 [33] / gw3 [33] / gw4 [33] / gw5 [33] / gw6 [33]
/ gw7 [33]
scheduling tests via LoadScheduling
apache_beam/pipeline_test.py::DoFnTest::test_element_param
apache_beam/pipeline_test.py::DoFnTest::test_key_param
apache_beam/transforms/combinefn_lifecycle_test.py::CombineFnLifecycleTest::test_combine
apache_beam/runners/portability/fn_api_runner/translations_test.py::TranslationsTest::test_run_packable_combine_globally
apache_beam/runners/portability/fn_api_runner/translations_test.py::TranslationsTest::test_run_packable_combine_per_key
apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiBasedStateBackedCoderTest::test_gbk_many_values
apache_beam/runners/portability/fn_api_runner/translations_test.py::TranslationsTest::test_run_packable_combine_limit
apache_beam/metrics/metric_test.py::MetricsTest::test_user_counter_using_pardo
[gw2] [32mPASSED[0m
apache_beam/runners/portability/fn_api_runner/fn_runner_test.py::FnApiBasedStateBackedCoderTest::test_gbk_many_values
apache_beam/transforms/combiners_test.py::TimestampCombinerTest::test_combiner_latest
[gw2] [32mPASSED[0m
apache_beam/transforms/combiners_test.py::TimestampCombinerTest::test_combiner_latest
apache_beam/transforms/ptransform_test.py::PTransformTest::test_flatten_pcollections
[gw0] [32mPASSED[0m
apache_beam/pipeline_test.py::DoFnTest::test_element_param
apache_beam/transforms/combinefn_lifecycle_test.py::CombineFnLifecycleTest::test_combining_value_state
[gw0] [33mSKIPPED[0m
apache_beam/transforms/combinefn_lifecycle_test.py::CombineFnLifecycleTest::test_combining_value_state
apache_beam/transforms/ptransform_test.py::PTransformTest::test_multiple_empty_outputs
[gw3] [32mPASSED[0m
apache_beam/metrics/metric_test.py::MetricsTest::test_user_counter_using_pardo
apache_beam/transforms/combiners_test.py::TimestampCombinerTest::test_combiner_earliest
[gw3] [32mPASSED[0m
apache_beam/transforms/combiners_test.py::TimestampCombinerTest::test_combiner_earliest
apache_beam/transforms/ptransform_test.py::PTransformTest::test_par_do_with_multiple_outputs_and_using_yield
[gw1] [32mPASSED[0m apache_beam/pipeline_test.py::DoFnTest::test_key_param
apache_beam/transforms/combinefn_lifecycle_test.py::CombineFnLifecycleTest::test_non_liftable_combine
[gw6] [32mPASSED[0m
apache_beam/transforms/combinefn_lifecycle_test.py::CombineFnLifecycleTest::test_combine
apache_beam/transforms/ptransform_test.py::PTransformTest::test_flatten_one_single_pcollection
[gw5] [32mPASSED[0m
apache_beam/runners/portability/fn_api_runner/translations_test.py::TranslationsTest::test_run_packable_combine_per_key
apache_beam/transforms/ptransform_test.py::PTransformTest::test_flatten_multiple_pcollections_having_multiple_consumers
[gw7] [32mPASSED[0m
apache_beam/runners/portability/fn_api_runner/translations_test.py::TranslationsTest::test_run_packable_combine_globally
apache_beam/transforms/dofn_lifecycle_test.py::DoFnLifecycleTest::test_dofn_lifecycle
[gw2] [32mPASSED[0m
apache_beam/transforms/ptransform_test.py::PTransformTest::test_flatten_pcollections
apache_beam/transforms/ptransform_test.py::PTransformTest::test_impulse
[gw4] [32mPASSED[0m
apache_beam/runners/portability/fn_api_runner/translations_test.py::TranslationsTest::test_run_packable_combine_limit
apache_beam/transforms/ptransform_test.py::PTransformTest::test_flatten_a_flattened_pcollection
[gw7] [32mPASSED[0m
apache_beam/transforms/dofn_lifecycle_test.py::DoFnLifecycleTest::test_dofn_lifecycle
apache_beam/transforms/sideinputs_test.py::SideInputsTest::test_as_singleton_with_different_defaults
[gw3] [32mPASSED[0m
apache_beam/transforms/ptransform_test.py::PTransformTest::test_par_do_with_multiple_outputs_and_using_yield
apache_beam/transforms/ptransform_test.py::PTransformTest::test_undeclared_outputs
[gw6] [32mPASSED[0m
apache_beam/transforms/ptransform_test.py::PTransformTest::test_flatten_one_single_pcollection
apache_beam/transforms/sideinputs_test.py::SideInputsTest::test_as_list_and_as_dict_side_inputs
[gw0] [32mPASSED[0m
apache_beam/transforms/ptransform_test.py::PTransformTest::test_multiple_empty_outputs
apache_beam/transforms/ptransform_test.py::PTransformTest::test_par_do_with_multiple_outputs_and_using_return
[gw5] [32mPASSED[0m
apache_beam/transforms/ptransform_test.py::PTransformTest::test_flatten_multiple_pcollections_having_multiple_consumers
apache_beam/transforms/sideinputs_test.py::SideInputsTest::test_as_list_twice
[gw4] [32mPASSED[0m
apache_beam/transforms/ptransform_test.py::PTransformTest::test_flatten_a_flattened_pcollection
apache_beam/transforms/sideinputs_test.py::SideInputsTest::test_default_value_singleton_side_input
[gw2] [32mPASSED[0m
apache_beam/transforms/ptransform_test.py::PTransformTest::test_impulse
apache_beam/transforms/sideinputs_test.py::SideInputsTest::test_as_singleton_without_unique_labels
[gw1] [32mPASSED[0m
apache_beam/transforms/combinefn_lifecycle_test.py::CombineFnLifecycleTest::test_non_liftable_combine
apache_beam/transforms/sideinputs_test.py::SideInputsTest::test_as_dict_twice
[gw6] [32mPASSED[0m
apache_beam/transforms/sideinputs_test.py::SideInputsTest::test_as_list_and_as_dict_side_inputs
apache_beam/transforms/sideinputs_test.py::SideInputsTest::test_iterable_side_input
[gw3] [32mPASSED[0m
apache_beam/transforms/ptransform_test.py::PTransformTest::test_undeclared_outputs
apache_beam/transforms/sideinputs_test.py::SideInputsTest::test_flattened_side_input
[gw7] [32mPASSED[0m
apache_beam/transforms/sideinputs_test.py::SideInputsTest::test_as_singleton_with_different_defaults
apache_beam/transforms/sideinputs_test.py::SideInputsTest::test_empty_singleton_side_input
[gw4] [32mPASSED[0m
apache_beam/transforms/sideinputs_test.py::SideInputsTest::test_default_value_singleton_side_input
[gw2] [32mPASSED[0m
apache_beam/transforms/sideinputs_test.py::SideInputsTest::test_as_singleton_without_unique_labels
[gw0] [32mPASSED[0m
apache_beam/transforms/ptransform_test.py::PTransformTest::test_par_do_with_multiple_outputs_and_using_return
apache_beam/transforms/sideinputs_test.py::SideInputsTest::test_reiterable_side_input
[gw5] [32mPASSED[0m
apache_beam/transforms/sideinputs_test.py::SideInputsTest::test_as_list_twice
apache_beam/transforms/util_test.py::ReshuffleTest::test_reshuffle_preserves_timestamps
[gw1] [32mPASSED[0m
apache_beam/transforms/sideinputs_test.py::SideInputsTest::test_as_dict_twice
[gw6] [32mPASSED[0m
apache_beam/transforms/sideinputs_test.py::SideInputsTest::test_iterable_side_input
[gw3] [32mPASSED[0m
apache_beam/transforms/sideinputs_test.py::SideInputsTest::test_flattened_side_input
[gw7] [32mPASSED[0m
apache_beam/transforms/sideinputs_test.py::SideInputsTest::test_empty_singleton_side_input
[gw0] [32mPASSED[0m
apache_beam/transforms/sideinputs_test.py::SideInputsTest::test_reiterable_side_input
[gw5] [32mPASSED[0m
apache_beam/transforms/util_test.py::ReshuffleTest::test_reshuffle_preserves_timestamps
[33m=============================== warnings summary
===============================[0m
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow_V2/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/hdfs/config.py:15:
DeprecationWarning: the imp module is deprecated in favour of importlib; see
the module's documentation for alternative uses
from imp import load_source
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42
../../build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow_V2/src/build/gradleenv/-1734967050/lib/python3.9/site-packages/tenacity/_asyncio.py:42:
DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use
"async def" instead
def call(self, fn, *args, **kwargs):
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file:
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow_V2/src/sdks/python/pytest_validatesRunnerStreamingTests-df-py39-xdist.xml
-
[33m=========== [32m32 passed[0m, [33m[1m3 skipped[0m, [33m[1m17
warnings[0m[33m in 3126.80s (0:52:06)[0m[33m ============[0m
>>> RUNNING integration tests with pipeline options:
>>> --runner=TestDataflowRunner --project=apache-beam-testing
>>> --region=us-central1
>>> --staging_location=gs://temp-storage-for-end-to-end-tests/staging-it
>>> --temp_location=gs://temp-storage-for-end-to-end-tests/temp-it
>>> --output=gs://temp-storage-for-end-to-end-tests/py-it-cloud/output
>>> --sdk_location=/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow_V2/src/sdks/python/build/apache-beam.tar.gz
>>> --requirements_file=postcommit_requirements.txt --num_workers=1
>>> --sleep_secs=20 --streaming
>>> --dataflow_worker_jar=/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow_V2/src/runners/google-cloud-dataflow-java/worker/build/libs/beam-runners-google-cloud-dataflow-java-fn-api-worker-2.42.0-SNAPSHOT.jar
>>> --experiments=use_runner_v2 --experiments=shuffle_mode=appliance
>>> --enable_streaming_engine
>>> pytest options: --capture=no --timeout=4500 --color=yes
>>> --log-cli-level=INFO
>>> collect markers: -m=it_validatesrunner and not no_sickbay_streaming and
>>> no_xdist
[1m============================= test session starts
==============================[0m
platform linux -- Python 3.9.10, pytest-7.1.2, pluggy-1.0.0
rootdir:
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow_V2/src/sdks/python,
configfile: pytest.ini
plugins: xdist-2.5.0, timeout-2.1.0, forked-1.4.0, requests-mock-1.10.0
timeout: 4500.0s
timeout method: signal
timeout func_only: False
[1m----------------------------- live log collection
------------------------------[0m
[33mWARNING [0m root:avroio_test.py:54 python-snappy is not installed; some
tests will be skipped.
[33mWARNING [0m root:tfrecordio_test.py:55 Tensorflow is not installed, so
skipping some tests.
[33mWARNING [0m
apache_beam.runners.interactive.interactive_environment:interactive_environment.py:190
Dependencies required for Interactive Beam PCollection visualization are not
available, please use: `pip install apache-beam[interactive]` to install
necessary dependencies to enable all data visualization features.
[33mWARNING [0m
apache_beam.runners.interactive.interactive_environment:interactive_environment.py:199
You cannot use Interactive Beam features when you are not in an interactive
environment such as a Jupyter notebook or ipython terminal.
[32mINFO [0m root:environments.py:376 Default Python SDK image for
environment is apache/beam_python3.9_sdk:2.42.0.dev
collected 6553 items / 6552 deselected / 2 skipped / 1 selected
apache_beam/runners/dataflow/dataflow_exercise_streaming_metrics_pipeline_test.py::ExerciseStreamingMetricsPipelineTest::test_streaming_pipeline_returns_expected_user_metrics_fnapi_it
[1m-------------------------------- live log call
---------------------------------[0m
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:754
Executing command:
['/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow_V2/src/build/gradleenv/-1734967050/bin/python3.9',
'-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r',
'/tmp/tmp2utwupw3/tmp_requirements.txt', '--exists-action', 'i', '--no-deps',
'--implementation', 'cp', '--abi', 'cp39', '--platform', 'manylinux2014_x86_64']
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:325 Copying
Beam SDK
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow_V2/src/sdks/python/build/apache-beam.tar.gz"
to staging location.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:477 Pipeline
has additional dependencies to be installed in SDK worker container, consider
using the SDK container image pre-building workflow to avoid repetitive
installations. Learn more on
https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild
[32mINFO [0m root:environments.py:376 Default Python SDK image for
environment is apache/beam_python3.9_sdk:2.42.0.dev
[32mINFO [0m root:environments.py:295 Using provided Python SDK container
image: gcr.io/cloud-dataflow/v1beta3/python39-fnapi:beam-master-20220811
[32mINFO [0m root:environments.py:302 Python SDK container image set to
"gcr.io/cloud-dataflow/v1beta3/python39-fnapi:beam-master-20220811" for Docker
environment
[32mINFO [0m apache_beam.internal.gcp.auth:auth.py:136 Setting socket
default timeout to 60 seconds.
[32mINFO [0m apache_beam.internal.gcp.auth:auth.py:138 socket default
timeout is 60.0 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/requirements.txt...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/requirements.txt
in 0 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/pickled_main_session...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/pickled_main_session
in 0 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/matplotlib-3.5.3.tar.gz...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/matplotlib-3.5.3.tar.gz
in 1 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/mock-2.0.0-py2.py3-none-any.whl...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/mock-2.0.0-py2.py3-none-any.whl
in 0 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/seaborn-0.11.2-py3-none-any.whl...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/seaborn-0.11.2-py3-none-any.whl
in 0 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/PyHamcrest-1.10.1-py3-none-any.whl...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/PyHamcrest-1.10.1-py3-none-any.whl
in 0 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/beautifulsoup4-4.11.1-py3-none-any.whl...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/beautifulsoup4-4.11.1-py3-none-any.whl
in 0 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/parameterized-0.7.5-py2.py3-none-any.whl...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/parameterized-0.7.5-py2.py3-none-any.whl
in 0 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/matplotlib-3.5.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/matplotlib-3.5.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl
in 0 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/matplotlib-3.5.2-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/matplotlib-3.5.2-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl
in 0 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/matplotlib-3.5.3-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/matplotlib-3.5.3-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl
in 0 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/matplotlib-3.5.3-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/matplotlib-3.5.3-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl
in 0 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/matplotlib-3.5.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/matplotlib-3.5.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl
in 0 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/matplotlib-3.5.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/matplotlib-3.5.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl
in 0 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/dataflow_python_sdk.tar...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/dataflow_python_sdk.tar
in 0 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/dataflow-worker.jar...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/dataflow-worker.jar
in 6 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/pipeline.pb...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:748 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0830211207-313202-nthvafm2.1661893927.313417/pipeline.pb
in 0 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:911 Create job:
<Job
clientRequestId: '20220830211207315022-2535'
createTime: '2022-08-30T21:12:21.207689Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2022-08-30_14_12_20-7718550243467521514'
location: 'us-central1'
name: 'beamapp-jenkins-0830211207-313202-nthvafm2'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2022-08-30T21:12:21.207689Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:913 Created job
with id: [2022-08-30_14_12_20-7718550243467521514]
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:914 Submitted job:
2022-08-30_14_12_20-7718550243467521514
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:915 To access the
Dataflow monitoring console, please navigate to
https://console.cloud.google.com/dataflow/jobs/us-central1/2022-08-30_14_12_20-7718550243467521514?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2022-08-30_14_12_20-7718550243467521514?project=apache-beam-testing
[32mINFO [0m
apache_beam.runners.dataflow.test_dataflow_runner:test_dataflow_runner.py:58
Console log:
[32mINFO [0m
apache_beam.runners.dataflow.test_dataflow_runner:test_dataflow_runner.py:59
https://console.cloud.google.com/dataflow/jobs/us-central1/2022-08-30_14_12_20-7718550243467521514?project=apache-beam-testing
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:197 Job
2022-08-30_14_12_20-7718550243467521514 is in state JOB_STATE_RUNNING
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242
2022-08-30T21:12:21.952Z: JOB_MESSAGE_WARNING: Autoscaling is enabled for
Dataflow Streaming Engine. Workers will scale between 1 and 100 unless
maxNumWorkers is specified.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242
2022-08-30T21:12:22.116Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job
2022-08-30_14_12_20-7718550243467521514. The number of workers will be between
1 and 100.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242
2022-08-30T21:12:22.151Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically
enabled for job 2022-08-30_14_12_20-7718550243467521514.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242
2022-08-30T21:12:26.822Z: JOB_MESSAGE_BASIC: Worker configuration:
e2-standard-2 in us-central1-a.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242
2022-08-30T21:12:27.651Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo
operations into optimizable parts.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242
2022-08-30T21:12:27.687Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton
operations into optimizable parts.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242
2022-08-30T21:12:27.758Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey
operations into optimizable parts.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242
2022-08-30T21:12:27.801Z: JOB_MESSAGE_DETAILED: Expanding
SplittableProcessKeyed operations into optimizable parts.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242
2022-08-30T21:12:27.831Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations
into streaming Read/Write steps
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242
2022-08-30T21:12:27.861Z: JOB_MESSAGE_DETAILED: Lifting
ValueCombiningMappingFns into MergeBucketsMappingFns
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242
2022-08-30T21:12:27.890Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner
information.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242
2022-08-30T21:12:27.959Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read,
Write, and Flatten operations
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242
2022-08-30T21:12:27.991Z: JOB_MESSAGE_DETAILED: Fusing consumer
generate_metrics into ReadFromPubSub/Read
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242
2022-08-30T21:12:28.028Z: JOB_MESSAGE_DETAILED: Fusing consumer
dump_to_pub/ToProtobuf into generate_metrics
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242
2022-08-30T21:12:28.071Z: JOB_MESSAGE_DETAILED: Fusing consumer
dump_to_pub/Write into dump_to_pub/ToProtobuf
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242
2022-08-30T21:12:28.112Z: JOB_MESSAGE_BASIC: Running job using Streaming Engine
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242
2022-08-30T21:12:28.145Z: JOB_MESSAGE_DEBUG: Workflow config is missing a
default resource spec.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242
2022-08-30T21:12:28.177Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and
teardown to workflow graph.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242
2022-08-30T21:12:28.214Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop
steps.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242
2022-08-30T21:12:28.271Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242
2022-08-30T21:12:28.319Z: JOB_MESSAGE_DEBUG: Starting worker pool setup.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242
2022-08-30T21:12:28.345Z: JOB_MESSAGE_BASIC: Starting 1 workers in
us-central1-a...
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242
2022-08-30T21:12:28.391Z: JOB_MESSAGE_DEBUG: Starting worker pool setup.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242
2022-08-30T21:12:59.956Z: JOB_MESSAGE_BASIC: Your project already contains 100
Dataflow-created metric descriptors, so new user metrics of the form
custom.googleapis.com/* will not be created. However, all user metrics are also
available in the metric dataflow.googleapis.com/job/user_counter. If you rely
on the custom metrics, you can delete old / unused metric descriptors. See
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list
and
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242
2022-08-30T21:13:02.027Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number
of workers to 1 so that the pipeline can catch up with its backlog and keep up
with its input rate.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:242
2022-08-30T21:13:32.017Z: JOB_MESSAGE_DETAILED: Workers have started
successfully.
[33mWARNING [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:253 Timing out
on waiting for job 2022-08-30_14_12_20-7718550243467521514 after 61 seconds
FATAL: command execution failed
java.io.IOException: Backing channel 'apache-beam-jenkins-13' is disconnected.
at
hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:215)
at
hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:285)
at com.sun.proxy.$Proxy138.isAlive(Unknown Source)
at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1215)
at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1207)
at hudson.Launcher$ProcStarter.join(Launcher.java:524)
at hudson.plugins.gradle.Gradle.perform(Gradle.java:317)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at
hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:814)
at hudson.model.Build$BuildExecution.build(Build.java:199)
at hudson.model.Build$BuildExecution.doRun(Build.java:164)
at
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:522)
at hudson.model.Run.execute(Run.java:1896)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:101)
at hudson.model.Executor.run(Executor.java:442)
Caused by: java.io.IOException: Unexpected termination of the channel
at
hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:75)
Caused by: java.io.EOFException
at
java.base/java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2905)
at
java.base/java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3400)
at
java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:936)
at
java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379)
at
hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
at hudson.remoting.Command.readFrom(Command.java:142)
at hudson.remoting.Command.readFrom(Command.java:128)
at
hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
at
hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
ERROR: apache-beam-jenkins-13 is offline; cannot locate jdk_1.8_latest
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
