See
<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/4201/display/redirect?page=changes>
Changes:
[Robert Bradshaw] Add a portable runner that renders pipelines as a dot graph.
[Robert Bradshaw] Add basic tests for render runner.
[Robert Bradshaw] Add the ability to pass a pipeline proto directly.
[Robert Bradshaw] lint
[Valentyn Tymofieiev] Fix typo.
[Valentyn Tymofieiev] Serve the graph when output file is not specified.
[Valentyn Tymofieiev] Serve the graph when output file is not specified.
[Valentyn Tymofieiev] Fix parsing of standalone protos.
[Valentyn Tymofieiev] Support reading from GCS.
[Valentyn Tymofieiev] Add text logging.
[Valentyn Tymofieiev] fix typo.
[Valentyn Tymofieiev] Some lint and yapf.
[Robert Bradshaw] Fix dot detection logic.
[Robert Bradshaw] fix error detected by lint
[Robert Bradshaw] Make gcs an optional dependency.
[Robert Bradshaw] return rather than sys.exit
[Robert Bradshaw] lint
[noreply] Apply suggestions from code review
[Robert Bradshaw] More cleanup, mypy.
[noreply] Dask runner tests action (#24324)
[Robert Bradshaw] lint
------------------------------------------
[...truncated 193.05 KB...]
[33m==== [32m1 passed[0m, [33m[1m5 skipped[0m, [33m[1m6744
deselected[0m, [33m[1m8 warnings[0m[33m in 737.25s (0:12:17)[0m[33m
=====[0m
> Task :sdks:python:test-suites:dataflow:py37:validatesRunnerStreamingTests
[31mFAILED[0m
=================================== FAILURES ===================================
[31m[1m_
ExerciseStreamingMetricsPipelineTest.test_streaming_pipeline_returns_expected_user_metrics_fnapi_it
_[0m
args = (subscription:
"projects/apache-beam-testing/subscriptions/exercise_streaming_metrics_subscription_output5b8c165a-ad46-4f26-b11d-b8ab5c25e90c"
max_messages: 50
,)
kwargs = {'metadata': [('x-goog-request-params',
'subscription=projects/apache-beam-testing/subscriptions/exercise_streaming_me...6-b11d-b8ab5c25e90c'),
('x-goog-api-client', 'gl-python/3.7.12 grpc/1.50.0 gax/2.10.2
gccl/2.13.11')], 'timeout': 30.0}
@functools.wraps(callable_)
def error_remapped_callable(*args, **kwargs):
try:
> return callable_(*args, **kwargs)
[1m[31m../../build/gradleenv/-1734967052/lib/python3.7/site-packages/google/api_core/grpc_helpers.py[0m:72:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <grpc._channel._UnaryUnaryMultiCallable object at 0x7fd0a0462b50>
request = subscription:
"projects/apache-beam-testing/subscriptions/exercise_streaming_metrics_subscription_output5b8c165a-ad46-4f26-b11d-b8ab5c25e90c"
max_messages: 50
timeout = 30.0
metadata = [('x-goog-request-params',
'subscription=projects/apache-beam-testing/subscriptions/exercise_streaming_metrics_subscri...t5b8c165a-ad46-4f26-b11d-b8ab5c25e90c'),
('x-goog-api-client', 'gl-python/3.7.12 grpc/1.50.0 gax/2.10.2 gccl/2.13.11')]
credentials = None, wait_for_ready = None, compression = None
def __call__(self,
request,
timeout=None,
metadata=None,
credentials=None,
wait_for_ready=None,
compression=None):
state, call, = self._blocking(request, timeout, metadata, credentials,
wait_for_ready, compression)
> return _end_unary_response_blocking(state, call, False, None)
[1m[31m../../build/gradleenv/-1734967052/lib/python3.7/site-packages/grpc/_channel.py[0m:946:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
state = <grpc._channel._RPCState object at 0x7fd0a0436f50>
call = <grpc._cython.cygrpc.SegregatedCall object at 0x7fd0a04795a0>
with_call = False, deadline = None
def _end_unary_response_blocking(state, call, with_call, deadline):
if state.code is grpc.StatusCode.OK:
if with_call:
rendezvous = _MultiThreadedRendezvous(state, call, None,
deadline)
return state.response, rendezvous
else:
return state.response
else:
> raise _InactiveRpcError(state)
[1m[31mE grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC
that terminated with:[0m
[1m[31mE status = StatusCode.DEADLINE_EXCEEDED[0m
[1m[31mE details = "Deadline Exceeded"[0m
[1m[31mE debug_error_string = "UNKNOWN:Deadline Exceeded
{created_time:"2022-11-23T10:58:42.967373148+00:00", grpc_status:4}"[0m
[1m[31mE >[0m
[1m[31m../../build/gradleenv/-1734967052/lib/python3.7/site-packages/grpc/_channel.py[0m:849:
_InactiveRpcError
[33mThe above exception was the direct cause of the following exception:[0m
self =
<apache_beam.runners.dataflow.dataflow_exercise_streaming_metrics_pipeline_test.ExerciseStreamingMetricsPipelineTest
testMethod=test_streaming_pipeline_returns_expected_user_metrics_fnapi_it>
@pytest.mark.it_validatesrunner
@pytest.mark.no_sickbay_batch
@pytest.mark.no_xdist
def test_streaming_pipeline_returns_expected_user_metrics_fnapi_it(self):
"""
Runs streaming Dataflow job and verifies that user metrics are reported
correctly.
"""
self._inject_words(self.input_topic, MESSAGES_TO_PUBLISH)
> result = self.run_pipeline()
[1m[31mapache_beam/runners/dataflow/dataflow_exercise_streaming_metrics_pipeline_test.py[0m:140:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[1m[31mapache_beam/runners/dataflow/dataflow_exercise_streaming_metrics_pipeline_test.py[0m:128:
in run_pipeline
return dataflow_exercise_streaming_metrics_pipeline.run(argv)
[1m[31mapache_beam/runners/dataflow/dataflow_exercise_streaming_metrics_pipeline.py[0m:99:
in run
result = pipeline.run()
[1m[31mapache_beam/pipeline.py[0m:577: in run
return self.runner.run_pipeline(self, self._options)
[1m[31mapache_beam/runners/dataflow/test_dataflow_runner.py[0m:70: in
run_pipeline
hc_assert_that(self.result, pickler.loads(on_success_matcher))
[1m[31m../../build/gradleenv/-1734967052/lib/python3.7/site-packages/hamcrest/core/core/allof.py[0m:19:
in matches
if not matcher.matches(item):
[1m[31m../../build/gradleenv/-1734967052/lib/python3.7/site-packages/hamcrest/core/base_matcher.py[0m:28:
in matches
match_result = self._matches(item)
[1m[31mapache_beam/io/gcp/tests/pubsub_matcher.py[0m:117: in _matches
self.expected_msg_len, self.timeout)
[1m[31mapache_beam/io/gcp/tests/pubsub_matcher.py[0m:134: in
_wait_for_messages
timeout=self.pull_timeout)
[1m[31m../../build/gradleenv/-1734967052/lib/python3.7/site-packages/google/pubsub_v1/services/subscriber/client.py[0m:1449:
in pull
metadata=metadata,
[1m[31m../../build/gradleenv/-1734967052/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py[0m:154:
in __call__
return wrapped_func(*args, **kwargs)
[1m[31m../../build/gradleenv/-1734967052/lib/python3.7/site-packages/google/api_core/retry.py[0m:288:
in retry_wrapped_func
on_error=on_error,
[1m[31m../../build/gradleenv/-1734967052/lib/python3.7/site-packages/google/api_core/retry.py[0m:190:
in retry_target
return target()
[1m[31m../../build/gradleenv/-1734967052/lib/python3.7/site-packages/google/api_core/timeout.py[0m:99:
in func_with_timeout
return func(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
args = (subscription:
"projects/apache-beam-testing/subscriptions/exercise_streaming_metrics_subscription_output5b8c165a-ad46-4f26-b11d-b8ab5c25e90c"
max_messages: 50
,)
kwargs = {'metadata': [('x-goog-request-params',
'subscription=projects/apache-beam-testing/subscriptions/exercise_streaming_me...6-b11d-b8ab5c25e90c'),
('x-goog-api-client', 'gl-python/3.7.12 grpc/1.50.0 gax/2.10.2
gccl/2.13.11')], 'timeout': 30.0}
@functools.wraps(callable_)
def error_remapped_callable(*args, **kwargs):
try:
return callable_(*args, **kwargs)
except grpc.RpcError as exc:
> raise exceptions.from_grpc_error(exc) from exc
[1m[31mE google.api_core.exceptions.DeadlineExceeded: 504 Deadline
Exceeded[0m
[1m[31m../../build/gradleenv/-1734967052/lib/python3.7/site-packages/google/api_core/grpc_helpers.py[0m:74:
DeadlineExceeded
------------------------------ Captured log call -------------------------------
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:780
Executing command:
['<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/ws/src/build/gradleenv/-1734967052/bin/python3.7',>
'-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r',
'/tmp/tmpjiy1pqm_/tmp_requirements.txt', '--exists-action', 'i', '--no-deps',
'--implementation', 'cp', '--abi', 'cp37m', '--platform',
'manylinux2014_x86_64']
[32mINFO [0m apache_beam.runners.portability.stager:stager.py:330 Copying
Beam SDK
"<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/ws/src/sdks/python/build/apache-beam.tar.gz";>
to staging location.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:485 Pipeline
has additional dependencies to be installed in SDK worker container, consider
using the SDK container image pre-building workflow to avoid repetitive
installations. Learn more on
https://cloud.google.com/dataflow/docs/guides/using-custom-containers#prebuild
[32mINFO [0m root:environments.py:376 Default Python SDK image for
environment is apache/beam_python3.7_sdk:2.44.0.dev
[32mINFO [0m root:environments.py:296 Using provided Python SDK container
image: gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20221021
[32mINFO [0m root:environments.py:304 Python SDK container image set to
"gcr.io/cloud-dataflow/v1beta3/python37-fnapi:beam-master-20221021" for Docker
environment
[32mINFO [0m apache_beam.internal.gcp.auth:auth.py:130 Setting socket
default timeout to 60 seconds.
[32mINFO [0m apache_beam.internal.gcp.auth:auth.py:133 socket default
timeout is 60.0 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1123104912-507650-nthvafm2.1669200552.507836/requirements.txt...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:751 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1123104912-507650-nthvafm2.1669200552.507836/requirements.txt
in 0 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1123104912-507650-nthvafm2.1669200552.507836/pickled_main_session...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:751 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1123104912-507650-nthvafm2.1669200552.507836/pickled_main_session
in 0 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1123104912-507650-nthvafm2.1669200552.507836/mock-2.0.0-py2.py3-none-any.whl...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:751 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1123104912-507650-nthvafm2.1669200552.507836/mock-2.0.0-py2.py3-none-any.whl
in 0 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1123104912-507650-nthvafm2.1669200552.507836/seaborn-0.12.1-py3-none-any.whl...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:751 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1123104912-507650-nthvafm2.1669200552.507836/seaborn-0.12.1-py3-none-any.whl
in 0 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1123104912-507650-nthvafm2.1669200552.507836/PyHamcrest-1.10.1-py3-none-any.whl...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:751 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1123104912-507650-nthvafm2.1669200552.507836/PyHamcrest-1.10.1-py3-none-any.whl
in 0 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1123104912-507650-nthvafm2.1669200552.507836/beautifulsoup4-4.11.1-py3-none-any.whl...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:751 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1123104912-507650-nthvafm2.1669200552.507836/beautifulsoup4-4.11.1-py3-none-any.whl
in 0 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1123104912-507650-nthvafm2.1669200552.507836/parameterized-0.7.5-py2.py3-none-any.whl...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:751 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1123104912-507650-nthvafm2.1669200552.507836/parameterized-0.7.5-py2.py3-none-any.whl
in 0 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1123104912-507650-nthvafm2.1669200552.507836/matplotlib-3.5.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:751 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1123104912-507650-nthvafm2.1669200552.507836/matplotlib-3.5.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl
in 1 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1123104912-507650-nthvafm2.1669200552.507836/matplotlib-3.6.1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:751 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1123104912-507650-nthvafm2.1669200552.507836/matplotlib-3.6.1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
in 0 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1123104912-507650-nthvafm2.1669200552.507836/matplotlib-3.6.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:751 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1123104912-507650-nthvafm2.1669200552.507836/matplotlib-3.6.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
in 0 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1123104912-507650-nthvafm2.1669200552.507836/matplotlib-3.6.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:751 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1123104912-507650-nthvafm2.1669200552.507836/matplotlib-3.6.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
in 0 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1123104912-507650-nthvafm2.1669200552.507836/matplotlib-3.6.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:751 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1123104912-507650-nthvafm2.1669200552.507836/matplotlib-3.6.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
in 0 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1123104912-507650-nthvafm2.1669200552.507836/dataflow_python_sdk.tar...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:751 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1123104912-507650-nthvafm2.1669200552.507836/dataflow_python_sdk.tar
in 0 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1123104912-507650-nthvafm2.1669200552.507836/dataflow-worker.jar...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:751 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1123104912-507650-nthvafm2.1669200552.507836/dataflow-worker.jar
in 5 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:732 Starting GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1123104912-507650-nthvafm2.1669200552.507836/pipeline.pb...
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:751 Completed GCS
upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-1123104912-507650-nthvafm2.1669200552.507836/pipeline.pb
in 0 seconds.
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:911 Create job:
<Job
clientRequestId: '20221123104912509006-2535'
createTime: '2022-11-23T10:49:24.074924Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2022-11-23_02_49_23-3000346731198265577'
location: 'us-central1'
name: 'beamapp-jenkins-1123104912-507650-nthvafm2'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2022-11-23T10:49:24.074924Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:913 Created job
with id: [2022-11-23_02_49_23-3000346731198265577]
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:914 Submitted job:
2022-11-23_02_49_23-3000346731198265577
[32mINFO [0m
apache_beam.runners.dataflow.internal.apiclient:apiclient.py:920 To access the
Dataflow monitoring console, please navigate to
https://console.cloud.google.com/dataflow/jobs/us-central1/2022-11-23_02_49_23-3000346731198265577?project=apache-beam-testing
[32mINFO [0m
apache_beam.runners.dataflow.test_dataflow_runner:test_dataflow_runner.py:58
Console log:
[32mINFO [0m
apache_beam.runners.dataflow.test_dataflow_runner:test_dataflow_runner.py:59
https://console.cloud.google.com/dataflow/jobs/us-central1/2022-11-23_02_49_23-3000346731198265577?project=apache-beam-testing
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:198 Job
2022-11-23_02_49_23-3000346731198265577 is in state JOB_STATE_RUNNING
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243
2022-11-23T10:49:41.951Z: JOB_MESSAGE_WARNING: Autoscaling is enabled for
Dataflow Streaming Engine. Workers will scale between 1 and 100 unless
maxNumWorkers is specified.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243
2022-11-23T10:49:42.216Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job
2022-11-23_02_49_23-3000346731198265577. The number of workers will be between
1 and 100.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243
2022-11-23T10:49:42.241Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically
enabled for job 2022-11-23_02_49_23-3000346731198265577.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243
2022-11-23T10:49:47.450Z: JOB_MESSAGE_BASIC: Worker configuration:
e2-standard-2 in us-central1-b.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243
2022-11-23T10:49:49.040Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo
operations into optimizable parts.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243
2022-11-23T10:49:49.070Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton
operations into optimizable parts.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243
2022-11-23T10:49:49.126Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey
operations into optimizable parts.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243
2022-11-23T10:49:49.156Z: JOB_MESSAGE_DETAILED: Expanding
SplittableProcessKeyed operations into optimizable parts.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243
2022-11-23T10:49:49.190Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations
into streaming Read/Write steps
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243
2022-11-23T10:49:49.217Z: JOB_MESSAGE_DETAILED: Lifting
ValueCombiningMappingFns into MergeBucketsMappingFns
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243
2022-11-23T10:49:49.251Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner
information.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243
2022-11-23T10:49:49.293Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read,
Write, and Flatten operations
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243
2022-11-23T10:49:49.322Z: JOB_MESSAGE_DETAILED: Fusing consumer
generate_metrics into ReadFromPubSub/Read
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243
2022-11-23T10:49:49.357Z: JOB_MESSAGE_DETAILED: Fusing consumer
dump_to_pub/ToProtobuf into generate_metrics
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243
2022-11-23T10:49:49.390Z: JOB_MESSAGE_DETAILED: Fusing consumer
dump_to_pub/Write into dump_to_pub/ToProtobuf
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243
2022-11-23T10:49:49.429Z: JOB_MESSAGE_BASIC: Running job using Streaming Engine
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243
2022-11-23T10:49:49.459Z: JOB_MESSAGE_DEBUG: Workflow config is missing a
default resource spec.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243
2022-11-23T10:49:49.491Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and
teardown to workflow graph.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243
2022-11-23T10:49:49.523Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop
steps.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243
2022-11-23T10:49:49.558Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243
2022-11-23T10:49:50.629Z: JOB_MESSAGE_DEBUG: Starting worker pool setup.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243
2022-11-23T10:49:50.656Z: JOB_MESSAGE_BASIC: Starting 1 workers in
us-central1-b...
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243
2022-11-23T10:49:50.700Z: JOB_MESSAGE_DEBUG: Starting worker pool setup.
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243
2022-11-23T10:50:17.372Z: JOB_MESSAGE_BASIC: Your project already contains 100
Dataflow-created metric descriptors, so new user metrics of the form
custom.googleapis.com/* will not be created. However, all user metrics are also
available in the metric dataflow.googleapis.com/job/user_counter. If you rely
on the custom metrics, you can delete old / unused metric descriptors. See
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list
and
https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
[32mINFO [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:243
2022-11-23T10:50:34.308Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number
of workers to 1 so that the pipeline can catch up with its backlog and keep up
with its input rate.
[33mWARNING [0m
apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:257 Timing out
on waiting for job 2022-11-23_02_49_23-3000346731198265577 after 61 seconds
[33m=============================== warnings summary
===============================[0m
../../build/gradleenv/-1734967052/lib/python3.7/site-packages/hdfs/config.py:15
<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/ws/src/build/gradleenv/-1734967052/lib/python3.7/site-packages/hdfs/config.py>:15:
DeprecationWarning: the imp module is deprecated in favour of importlib; see
the module's documentation for alternative uses
from imp import load_source
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated xml file:
<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/ws/src/sdks/python/pytest_validatesRunnerStreamingTests-df-py37-noxdist.xml>
-
[36m[1m=========================== short test summary info
============================[0m
[31mFAILED[0m
apache_beam/runners/dataflow/dataflow_exercise_streaming_metrics_pipeline_test.py::[1mExerciseStreamingMetricsPipelineTest::test_streaming_pipeline_returns_expected_user_metrics_fnapi_it[0m
- google.api_core.exceptions.DeadlineExceeded: 504 Deadline Exceeded
[31m===== [31m[1m1 failed[0m, [33m5 skipped[0m, [33m6744 deselected[0m,
[33m1 warning[0m[31m in 753.01s (0:12:33)[0m[31m =====[0m
> Task :sdks:python:test-suites:dataflow:py37:validatesRunnerStreamingTests
> FAILED
FAILURE: Build failed with an exception.
* Where:
Script
'<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/ws/src/sdks/python/test-suites/dataflow/common.gradle'>
line: 291
* What went wrong:
Execution failed for task
':sdks:python:test-suites:dataflow:py37:validatesRunnerStreamingTests'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings
and determine if they come from your own scripts or plugins.
See
https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 53m 2s
97 actionable tasks: 62 executed, 33 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/idteskzdxtidq
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
