See
<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/4434/display/redirect?page=changes>
Changes:
[kirillkozlov] [BEAM-8275] Beam SQL should support BigQuery in DIRECT_READ mode
[github] Addressed review comments
[github] Added a test for BigQuery SQL read in EXPORT mode
------------------------------------------
[...truncated 419.82 KB...]
at java.lang.Thread.run(Thread.java:745)
-------------------- >> begin captured stdout << ---------------------
Worker logs:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-09-30_10_12_46-17312328951666134401?project=apache-beam-testing
--------------------- >> end captured stdout << ----------------------
======================================================================
ERROR: test_metrics_fnapi_it
(apache_beam.runners.dataflow.dataflow_exercise_metrics_pipeline_test.ExerciseMetricsPipelineTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py";,>
line 60, in test_metrics_fnapi_it
result = self.run_pipeline(experiment='beam_fn_api')
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py";,>
line 48, in run_pipeline
return dataflow_exercise_metrics_pipeline.apply_and_run(p)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline.py";,>
line 180, in apply_and_run
result = pipeline.run()
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/pipeline.py";,>
line 407, in run
self._options).run(False)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/pipeline.py";,>
line 420, in run
return self.runner.run_pipeline(self, self._options)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py";,>
line 64, in run_pipeline
self.result.wait_until_finish(duration=wait_duration)
File
"<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py";,>
line 1365, in wait_until_finish
(self.state, getattr(self._runner, 'last_error_msg', None)), self)
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow
pipeline failed. State: FAILED, Error:
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
received from SDK harness for instruction -128: Traceback (most recent call
last):
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 300, in get
processor = self.cached_bundle_processors[bundle_descriptor_id].pop()
IndexError: pop from empty list
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/usr/local/lib/python3.5/site-packages/apache_beam/internal/pickler.py", line
261, in loads
return dill.loads(s)
File "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 317, in
loads
return load(file, ignore)
File "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 305, in load
obj = pik.load()
TypeError: _create_function() takes from 2 to 6 positional arguments but 7 were
given
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 158, in _execute
response = task()
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 191, in <lambda>
self._execute(lambda: worker.do_instruction(work), work)
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 343, in do_instruction
request.instruction_id)
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 363, in process_bundle
instruction_id, request.process_bundle_descriptor_id)
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 306, in get
self.data_channel_factory)
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 580, in __init__
self.ops = self.create_execution_tree(self.process_bundle_descriptor)
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 624, in create_execution_tree
descriptor.transforms, key=topological_height, reverse=True)])
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 623, in <listcomp>
for transform_id in sorted(
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 548, in wrapper
result = cache[args] = func(*args)
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 607, in get_operation
in descriptor.transforms[transform_id].outputs.items()
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 606, in <dictcomp>
for tag, pcoll_id
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 605, in <listcomp>
tag: [get_operation(op) for op in pcoll_consumers[pcoll_id]]
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 548, in wrapper
result = cache[args] = func(*args)
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 607, in get_operation
in descriptor.transforms[transform_id].outputs.items()
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 606, in <dictcomp>
for tag, pcoll_id
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 605, in <listcomp>
tag: [get_operation(op) for op in pcoll_consumers[pcoll_id]]
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 548, in wrapper
result = cache[args] = func(*args)
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 607, in get_operation
in descriptor.transforms[transform_id].outputs.items()
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 606, in <dictcomp>
for tag, pcoll_id
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 605, in <listcomp>
tag: [get_operation(op) for op in pcoll_consumers[pcoll_id]]
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 548, in wrapper
result = cache[args] = func(*args)
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 610, in get_operation
transform_id, transform_consumers)
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 869, in create_operation
return creator(self, transform_id, transform_proto, payload, consumers)
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 1112, in create
serialized_fn, parameter)
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 1150, in _create_pardo_operation
dofn_data = pickler.loads(serialized_fn)
File
"/usr/local/lib/python3.5/site-packages/apache_beam/internal/pickler.py", line
265, in loads
return dill.loads(s)
File "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 317, in
loads
return load(file, ignore)
File "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 305, in load
obj = pik.load()
TypeError: _create_function() takes from 2 to 6 positional arguments but 7 were
given
at
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at org.apache.beam.sdk.util.MoreFutures.get(MoreFutures.java:57)
at
org.apache.beam.runners.dataflow.worker.fn.control.RegisterAndProcessBundleOperation.finish(RegisterAndProcessBundleOperation.java:285)
at
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:85)
at
org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:411)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:380)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:305)
at
org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.start(DataflowRunnerHarness.java:195)
at
org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.main(DataflowRunnerHarness.java:123)
Suppressed: java.lang.IllegalStateException: Already closed.
at
org.apache.beam.sdk.fn.data.BeamFnDataBufferingOutboundObserver.close(BeamFnDataBufferingOutboundObserver.java:95)
at
org.apache.beam.runners.dataflow.worker.fn.data.RemoteGrpcPortWriteOperation.abort(RemoteGrpcPortWriteOperation.java:220)
at
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:91)
... 6 more
Caused by: java.lang.RuntimeException: Error received from SDK harness for
instruction -128: Traceback (most recent call last):
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 300, in get
processor = self.cached_bundle_processors[bundle_descriptor_id].pop()
IndexError: pop from empty list
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/usr/local/lib/python3.5/site-packages/apache_beam/internal/pickler.py", line
261, in loads
return dill.loads(s)
File "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 317, in
loads
return load(file, ignore)
File "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 305, in load
obj = pik.load()
TypeError: _create_function() takes from 2 to 6 positional arguments but 7 were
given
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 158, in _execute
response = task()
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 191, in <lambda>
self._execute(lambda: worker.do_instruction(work), work)
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 343, in do_instruction
request.instruction_id)
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 363, in process_bundle
instruction_id, request.process_bundle_descriptor_id)
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 306, in get
self.data_channel_factory)
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 580, in __init__
self.ops = self.create_execution_tree(self.process_bundle_descriptor)
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 624, in create_execution_tree
descriptor.transforms, key=topological_height, reverse=True)])
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 623, in <listcomp>
for transform_id in sorted(
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 548, in wrapper
result = cache[args] = func(*args)
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 607, in get_operation
in descriptor.transforms[transform_id].outputs.items()
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 606, in <dictcomp>
for tag, pcoll_id
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 605, in <listcomp>
tag: [get_operation(op) for op in pcoll_consumers[pcoll_id]]
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 548, in wrapper
result = cache[args] = func(*args)
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 607, in get_operation
in descriptor.transforms[transform_id].outputs.items()
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 606, in <dictcomp>
for tag, pcoll_id
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 605, in <listcomp>
tag: [get_operation(op) for op in pcoll_consumers[pcoll_id]]
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 548, in wrapper
result = cache[args] = func(*args)
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 607, in get_operation
in descriptor.transforms[transform_id].outputs.items()
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 606, in <dictcomp>
for tag, pcoll_id
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 605, in <listcomp>
tag: [get_operation(op) for op in pcoll_consumers[pcoll_id]]
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 548, in wrapper
result = cache[args] = func(*args)
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 610, in get_operation
transform_id, transform_consumers)
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 869, in create_operation
return creator(self, transform_id, transform_proto, payload, consumers)
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 1112, in create
serialized_fn, parameter)
File
"/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 1150, in _create_pardo_operation
dofn_data = pickler.loads(serialized_fn)
File
"/usr/local/lib/python3.5/site-packages/apache_beam/internal/pickler.py", line
265, in loads
return dill.loads(s)
File "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 317, in
loads
return load(file, ignore)
File "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 305, in load
obj = pik.load()
TypeError: _create_function() takes from 2 to 6 positional arguments but 7 were
given
at
org.apache.beam.runners.fnexecution.control.FnApiControlClient$ResponseStreamObserver.onNext(FnApiControlClient.java:157)
at
org.apache.beam.runners.fnexecution.control.FnApiControlClient$ResponseStreamObserver.onNext(FnApiControlClient.java:140)
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:249)
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33)
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Contexts$ContextualizedServerCallListener.onMessage(Contexts.java:76)
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:297)
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:738)
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
-------------------- >> begin captured stdout << ---------------------
Worker logs:
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-09-30_10_21_25-9480454263269971434?project=apache-beam-testing
--------------------- >> end captured stdout << ----------------------
----------------------------------------------------------------------
XML: nosetests-python3.5_sdk.xml
----------------------------------------------------------------------
XML:
<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 2 tests in 1087.207s
FAILED (errors=2)
cleanup_container
Untagged: us.gcr.io/apache-beam-testing/jenkins/python3.5_sdk:20190930-170714
Untagged:
us.gcr.io/apache-beam-testing/jenkins/python3.5_sdk@sha256:3061b3f32fc0a7fd091b9629e781440d9f7150b9f372b54c12573e6b2481bf73
Deleted: sha256:69840bf119c24c949c94ea3385ee5c9e29d588bd1d954ca4c5a82776b554d44a
Deleted: sha256:262405a3c1e4b490470be9bc41ebcf331a5dc2c6f0d8a019ec5cae0b80295845
Deleted: sha256:affa671d3394e11129aacf2a84490561556df09c8b2c4c5169663e7411422422
Deleted: sha256:e95b458031a5a53455f620f52965abfc42083fb88996082174319dab4c56e0d9
Deleted: sha256:5dac1a58840fa916cf78e042367d7c63624d720e592db03a189d613023014902
Deleted: sha256:61c6dcd378b85a4b6a6ea6e753f2e59c114fe40aa40a389b4834219ac35e8cd9
Deleted: sha256:06480cd9895d24cb22aa38ce11c5cbfba71987bea71f8acff2aaa65061a0029c
Deleted: sha256:89751160bd086ed275659dc942194bd4f1418b8c2ad4629fff3c632244628f9c
Deleted: sha256:1f7c8d693864151677654b5bb66c58f2f787b9f5a4319ed90a528267ec2bf034
Deleted: sha256:47c87b8d41bd622549ae0a1ed0e6b42d0dbb7f68af31b94b61c00c7eb59247e6
Deleted: sha256:6650b5548c23246f3a3b0ec3b84dd8b8c3a08a79e3f4d38b68c5aa6c507dabb8
Deleted: sha256:d7a1a8c11c34ed8270ec23c59533b295cd65b96edbf1e14128a30c93310c6bf8
Deleted: sha256:15b873e3665c060f7e7ba0d185568bcbc7fa824a3abd82868ccde56bfbe556d8
Deleted: sha256:7047cdbcf07e87c246a927955c7db1a0cc9f45b099542209f8b898708a06d8a3
Digests:
-
us.gcr.io/apache-beam-testing/jenkins/python3.5_sdk@sha256:3061b3f32fc0a7fd091b9629e781440d9f7150b9f372b54c12573e6b2481bf73
Associated tags:
- 20190930-170714
Tags:
- us.gcr.io/apache-beam-testing/jenkins/python3.5_sdk:20190930-170714
Deleted [us.gcr.io/apache-beam-testing/jenkins/python3.5_sdk:20190930-170714].
Deleted
[us.gcr.io/apache-beam-testing/jenkins/python3.5_sdk@sha256:3061b3f32fc0a7fd091b9629e781440d9f7150b9f372b54c12573e6b2481bf73].
Removed the container
Build step 'Execute shell' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
