See
<https://builds.apache.org/job/beam_PostCommit_Python37/1106/display/redirect?page=changes>
Changes:
[kirillkozlov] Updated the cost model to favor IO with push-down
[kirillkozlov] BigQueryFilter numSupported method
[kirillkozlov] Update SQL BigQuery doc
------------------------------------------
[...truncated 622.41 KB...]
return sink.pre_finalize(init_result, write_results)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/filebasedsink.py";,>
line 218, in pre_finalize
FileSystems.delete(dst_glob_files)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/filesystems.py";,>
line 318, in delete
return filesystem.delete(paths)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/localfilesystem.py";,>
line 335, in delete
raise BeamIOError("Delete operation failed", exceptions)
apache_beam.io.filesystem.BeamIOError: Delete operation failed with exceptions
{'/tmp/py-wordcount-direct-00001-of-00004': OSError('No files found to delete
under: /tmp/py-wordcount-direct-00001-of-00004')}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker.py";,>
line 138, in _execute
response = task()
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker.py";,>
line 188, in <lambda>
lambda: self.create_worker().do_instruction(request), request)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker.py";,>
line 286, in do_instruction
request.instruction_id)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker.py";,>
line 312, in process_bundle
bundle_processor.process_bundle(instruction_id))
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/worker/bundle_processor.py";,>
line 675, in process_bundle
data.transform_id].process_encoded(data.data)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/worker/bundle_processor.py";,>
line 146, in process_encoded
self.output(decoded_value)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/worker/operations.py";,>
line 259, in output
cython.cast(Receiver, self.receivers[output_index]).receive(windowed_value)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/worker/operations.py";,>
line 146, in receive
self.consumer.process(windowed_value)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/worker/operations.py";,>
line 597, in process
delayed_application = self.dofn_receiver.receive(o)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/common.py";,>
line 809, in receive
self.process(windowed_value)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/common.py";,>
line 815, in process
self._reraise_augmented(exn)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/common.py";,>
line 882, in _reraise_augmented
raise_with_traceback(new_exn)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/future/utils/__init__.py";,>
line 446, in raise_with_traceback
raise exc.with_traceback(traceback)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/common.py";,>
line 813, in process
return self.do_fn_invoker.invoke_process(windowed_value)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/common.py";,>
line 611, in invoke_process
windowed_value, additional_args, additional_kwargs, output_processor)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/common.py";,>
line 685, in _invoke_process_per_window
windowed_value, self.process_method(*args_for_process))
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/iobase.py";,>
line 1052, in _pre_finalize
return sink.pre_finalize(init_result, write_results)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/filebasedsink.py";,>
line 218, in pre_finalize
FileSystems.delete(dst_glob_files)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/filesystems.py";,>
line 318, in delete
return filesystem.delete(paths)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/localfilesystem.py";,>
line 335, in delete
raise BeamIOError("Delete operation failed", exceptions)
apache_beam.io.filesystem.BeamIOError: Delete operation failed with exceptions
{'/tmp/py-wordcount-direct-00001-of-00004': OSError('No files found to delete
under: /tmp/py-wordcount-direct-00001-of-00004')} [while running
'write/Write/WriteImpl/PreFinalize'] with exceptions None
19/12/11 02:38:56 INFO
org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService:
Manifest at
/tmp/beam-tempt4pocngu/artifactspn0efcbg/job_8fef7e4f-1018-4a94-b291-545f7b8fd675/MANIFEST
has 1 artifact locations
19/12/11 02:38:56 INFO
org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactStagingService:
Removed dir
/tmp/beam-tempt4pocngu/artifactspn0efcbg/job_8fef7e4f-1018-4a94-b291-545f7b8fd675/
INFO:apache_beam.runners.portability.portable_runner:Job state changed to FAILED
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/examples/wordcount.py";,>
line 138, in <module>
run()
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/examples/wordcount.py";,>
line 117, in run
result = p.run()
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/pipeline.py";,>
line 417, in run
self._options).run(False)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/pipeline.py";,>
line 430, in run
return self.runner.run_pipeline(self, self._options)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/portability/spark_runner.py";,>
line 41, in run_pipeline
return super(SparkRunner, self).run_pipeline(pipeline, options)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/portability/portable_runner.py";,>
line 320, in run_pipeline
result.wait_until_finish()
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/portability/portable_runner.py";,>
line 438, in wait_until_finish
self._job_id, self._state, self._last_error_message()))
RuntimeError: Pipeline
BeamApp-jenkins-1211023835-6a3c3791_1c02c43e-0094-4ede-a23c-3a41e7ef6934 failed
in state FAILED: java.lang.RuntimeException: Error received from SDK harness
for instruction 62: Traceback (most recent call last):
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/common.py";,>
line 813, in process
return self.do_fn_invoker.invoke_process(windowed_value)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/common.py";,>
line 611, in invoke_process
windowed_value, additional_args, additional_kwargs, output_processor)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/common.py";,>
line 685, in _invoke_process_per_window
windowed_value, self.process_method(*args_for_process))
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/iobase.py";,>
line 1052, in _pre_finalize
return sink.pre_finalize(init_result, write_results)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/filebasedsink.py";,>
line 218, in pre_finalize
FileSystems.delete(dst_glob_files)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/filesystems.py";,>
line 318, in delete
return filesystem.delete(paths)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/localfilesystem.py";,>
line 335, in delete
raise BeamIOError("Delete operation failed", exceptions)
apache_beam.io.filesystem.BeamIOError: Delete operation failed with exceptions
{'/tmp/py-wordcount-direct-00001-of-00004': OSError('No files found to delete
under: /tmp/py-wordcount-direct-00001-of-00004')}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker.py";,>
line 138, in _execute
response = task()
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker.py";,>
line 188, in <lambda>
lambda: self.create_worker().do_instruction(request), request)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker.py";,>
line 286, in do_instruction
request.instruction_id)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker.py";,>
line 312, in process_bundle
bundle_processor.process_bundle(instruction_id))
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/worker/bundle_processor.py";,>
line 675, in process_bundle
data.transform_id].process_encoded(data.data)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/worker/bundle_processor.py";,>
line 146, in process_encoded
self.output(decoded_value)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/worker/operations.py";,>
line 259, in output
cython.cast(Receiver, self.receivers[output_index]).receive(windowed_value)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/worker/operations.py";,>
line 146, in receive
self.consumer.process(windowed_value)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/worker/operations.py";,>
line 597, in process
delayed_application = self.dofn_receiver.receive(o)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/common.py";,>
line 809, in receive
self.process(windowed_value)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/common.py";,>
line 815, in process
self._reraise_augmented(exn)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/common.py";,>
line 882, in _reraise_augmented
raise_with_traceback(new_exn)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/future/utils/__init__.py";,>
line 446, in raise_with_traceback
raise exc.with_traceback(traceback)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/common.py";,>
line 813, in process
return self.do_fn_invoker.invoke_process(windowed_value)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/common.py";,>
line 611, in invoke_process
windowed_value, additional_args, additional_kwargs, output_processor)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/common.py";,>
line 685, in _invoke_process_per_window
windowed_value, self.process_method(*args_for_process))
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/iobase.py";,>
line 1052, in _pre_finalize
return sink.pre_finalize(init_result, write_results)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/filebasedsink.py";,>
line 218, in pre_finalize
FileSystems.delete(dst_glob_files)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/filesystems.py";,>
line 318, in delete
return filesystem.delete(paths)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/io/localfilesystem.py";,>
line 335, in delete
raise BeamIOError("Delete operation failed", exceptions)
apache_beam.io.filesystem.BeamIOError: Delete operation failed with exceptions
{'/tmp/py-wordcount-direct-00001-of-00004': OSError('No files found to delete
under: /tmp/py-wordcount-direct-00001-of-00004')} [while running
'write/Write/WriteImpl/PreFinalize'] with exceptions None
ERROR:apache_beam.runners.worker.data_plane:Failed to read inputs in the data
plane.
Traceback (most recent call last):
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py";,>
line 272, in _read_inputs
for elements in elements_iterator:
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py";,>
line 392, in __next__
return self._next()
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py";,>
line 561, in _next
raise self
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Socket closed"
debug_error_string =
"{"created":"@1576031936.799950592","description":"Error received from peer
ipv4:127.0.0.1:38493","file":"src/core/lib/surface/call.cc","file_line":1055,"grpc_message":"Socket
closed","grpc_status":14}"
>
Exception in thread read_grpc_client_inputs:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py";,>
line 286, in <lambda>
target=lambda: self._read_inputs(elements_iterator),
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/worker/data_plane.py";,>
line 272, in _read_inputs
for elements in elements_iterator:
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py";,>
line 392, in __next__
return self._next()
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py";,>
line 561, in _next
raise self
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Socket closed"
debug_error_string =
"{"created":"@1576031936.799950592","description":"Error received from peer
ipv4:127.0.0.1:38493","file":"src/core/lib/surface/call.cc","file_line":1055,"grpc_message":"Socket
closed","grpc_status":14}"
>
Exception in thread run_worker_1-1:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker.py";,>
line 112, in run
for work_request in control_stub.Control(get_responses()):
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py";,>
line 392, in __next__
return self._next()
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py";,>
line 561, in _next
raise self
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Socket closed"
debug_error_string =
"{"created":"@1576031936.799996283","description":"Error received from peer
ipv4:127.0.0.1:38417","file":"src/core/lib/surface/call.cc","file_line":1055,"grpc_message":"Socket
closed","grpc_status":14}"
>
Exception in thread read_state:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker.py";,>
line 532, in pull_responses
for response in responses:
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py";,>
line 392, in __next__
return self._next()
File
"<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py";,>
line 561, in _next
raise self
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Socket closed"
debug_error_string =
"{"created":"@1576031936.799986897","description":"Error received from peer
ipv4:127.0.0.1:41819","file":"src/core/lib/surface/call.cc","file_line":1055,"grpc_message":"Socket
closed","grpc_status":14}"
>
> Task :sdks:python:test-suites:portable:py37:portableWordCountSparkRunnerBatch
> FAILED
FAILURE: Build completed with 2 failures.
1: Task failed with an exception.
-----------
* Where:
Build file
'<https://builds.apache.org/job/beam_PostCommit_Python37/ws/src/sdks/python/test-suites/direct/py37/build.gradle'>
line: 61
* What went wrong:
Execution failed for task
':sdks:python:test-suites:direct:py37:hdfsIntegrationTest'.
> Process 'command 'sh'' finished with non-zero exit value 255
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task
':sdks:python:test-suites:portable:py37:portableWordCountSparkRunnerBatch'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 59m 26s
84 actionable tasks: 63 executed, 21 from cache
Publishing build scan...
https://scans.gradle.com/s/x5b6urddfhfx6
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
