[
https://issues.apache.org/jira/browse/BEAM-3956?focusedWorklogId=87415&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87415
]
ASF GitHub Bot logged work on BEAM-3956:
----------------------------------------
Author: ASF GitHub Bot
Created on: 04/Apr/18 03:46
Start Date: 04/Apr/18 03:46
Worklog Time Spent: 10m
Work Description: shoyer commented on issue #4959: [BEAM-3956] Preserve
stacktraces for Python exceptions
URL: https://github.com/apache/beam/pull/4959#issuecomment-378471464
OK, so the stacktrace version is definitely prettier (see above), but my new
tests now fail when using GRPC (e.g., `test_error_traceback_includes_user_code
(apache_beam.runners.portability.fn_api_runner_test.FnApiRunnerTestWithGrpc)`).
I'm not quite sure why -- the stacktrace can still be found in the error log
when I run the test:
<details>
```
$ python setup.py test -s
apache_beam.runners.portability.fn_api_runner_test.FnApiRunnerTestWithGrpc.test_error_traceback_includes_user_code
/usr/local/google/home/shoyer/virtual-envs/beam-dev/local/lib/python2.7/site-packages/setuptools/dist.py:397:
UserWarning: Normalizing '2.5.0.dev' to '2.5.0.dev0'
normalized_version,
running test
/usr/local/google/home/shoyer/open-source/beam/sdks/python/gen_protos.py:50:
UserWarning: Installing grpcio-tools is recommended for development.
warnings.warn('Installing grpcio-tools is recommended for development.')
Searching for futures<4.0.0,>=3.1.1
Best match: futures 3.2.0
Processing futures-3.2.0-py2.7.egg
Using
/usr/local/google/home/shoyer/open-source/beam/sdks/python/.eggs/futures-3.2.0-py2.7.egg
Searching for typing<3.7.0,>=3.6.0
Best match: typing 3.6.4
Processing typing-3.6.4-py2.7.egg
Using
/usr/local/google/home/shoyer/open-source/beam/sdks/python/.eggs/typing-3.6.4-py2.7.egg
Searching for pyvcf<0.7.0,>=0.6.8
Best match: PyVCF 0.6.8
Processing PyVCF-0.6.8-py2.7.egg
Using
/usr/local/google/home/shoyer/open-source/beam/sdks/python/.eggs/PyVCF-0.6.8-py2.7.egg
Searching for pyyaml<4.0.0,>=3.12
Best match: PyYAML 3.12
Processing PyYAML-3.12-py2.7-linux-x86_64.egg
Using
/usr/local/google/home/shoyer/open-source/beam/sdks/python/.eggs/PyYAML-3.12-py2.7-linux-x86_64.egg
Searching for pytz>=2018.3
Best match: pytz 2018.3
Processing pytz-2018.3-py2.7.egg
Using
/usr/local/google/home/shoyer/open-source/beam/sdks/python/.eggs/pytz-2018.3-py2.7.egg
Searching for protobuf<4,>=3.5.0.post1
Best match: protobuf 3.5.2.post1
Processing protobuf-3.5.2.post1-py2.7.egg
Using
/usr/local/google/home/shoyer/open-source/beam/sdks/python/.eggs/protobuf-3.5.2.post1-py2.7.egg
Searching for oauth2client<5,>=2.0.1
Best match: oauth2client 4.1.2
Processing oauth2client-4.1.2-py2.7.egg
Using
/usr/local/google/home/shoyer/open-source/beam/sdks/python/.eggs/oauth2client-4.1.2-py2.7.egg
Searching for mock<3.0.0,>=1.0.1
Best match: mock 2.0.0
Processing mock-2.0.0-py2.7.egg
Using
/usr/local/google/home/shoyer/open-source/beam/sdks/python/.eggs/mock-2.0.0-py2.7.egg
Searching for httplib2<0.10,>=0.8
Best match: httplib2 0.9.2
Processing httplib2-0.9.2-py2.7.egg
Using
/usr/local/google/home/shoyer/open-source/beam/sdks/python/.eggs/httplib2-0.9.2-py2.7.egg
Searching for hdfs<3.0.0,>=2.1.0
Best match: hdfs 2.1.0
Processing hdfs-2.1.0-py2.7.egg
Using
/usr/local/google/home/shoyer/open-source/beam/sdks/python/.eggs/hdfs-2.1.0-py2.7.egg
Searching for grpcio<2,>=1.8
Best match: grpcio 1.10.1rc1
Processing grpcio-1.10.1rc1-py2.7-linux-x86_64.egg
Using
/usr/local/google/home/shoyer/open-source/beam/sdks/python/.eggs/grpcio-1.10.1rc1-py2.7-linux-x86_64.egg
Searching for dill==0.2.6
Best match: dill 0.2.6
Processing dill-0.2.6-py2.7.egg
Using
/usr/local/google/home/shoyer/open-source/beam/sdks/python/.eggs/dill-0.2.6-py2.7.egg
Searching for crcmod<2.0,>=1.7
Best match: crcmod 1.7
Processing crcmod-1.7-py2.7-linux-x86_64.egg
Using
/usr/local/google/home/shoyer/open-source/beam/sdks/python/.eggs/crcmod-1.7-py2.7-linux-x86_64.egg
Searching for avro<2.0.0,>=1.8.1
Best match: avro 1.8.2
Processing avro-1.8.2-py2.7.egg
Using
/usr/local/google/home/shoyer/open-source/beam/sdks/python/.eggs/avro-1.8.2-py2.7.egg
Searching for rsa>=3.1.4
Best match: rsa 3.4.2
Processing rsa-3.4.2-py2.7.egg
Using
/usr/local/google/home/shoyer/open-source/beam/sdks/python/.eggs/rsa-3.4.2-py2.7.egg
Searching for pyasn1>=0.1.7
Best match: pyasn1 0.4.2
Processing pyasn1-0.4.2-py2.7.egg
Using
/usr/local/google/home/shoyer/open-source/beam/sdks/python/.eggs/pyasn1-0.4.2-py2.7.egg
Searching for pyasn1-modules>=0.0.5
Best match: pyasn1-modules 0.2.1
Processing pyasn1_modules-0.2.1-py2.7.egg
Using
/usr/local/google/home/shoyer/open-source/beam/sdks/python/.eggs/pyasn1_modules-0.2.1-py2.7.egg
Searching for pbr>=0.11
Best match: pbr 4.0.1
Processing pbr-4.0.1-py2.7.egg
Using
/usr/local/google/home/shoyer/open-source/beam/sdks/python/.eggs/pbr-4.0.1-py2.7.egg
Searching for funcsigs>=1
Best match: funcsigs 1.0.2
Processing funcsigs-1.0.2-py2.7.egg
Using
/usr/local/google/home/shoyer/open-source/beam/sdks/python/.eggs/funcsigs-1.0.2-py2.7.egg
Searching for requests>=2.7.0
Best match: requests 2.18.4
Processing requests-2.18.4-py2.7.egg
Using
/usr/local/google/home/shoyer/open-source/beam/sdks/python/.eggs/requests-2.18.4-py2.7.egg
Searching for docopt
Best match: docopt 0.6.2
Processing docopt-0.6.2-py2.7.egg
Using
/usr/local/google/home/shoyer/open-source/beam/sdks/python/.eggs/docopt-0.6.2-py2.7.egg
Searching for urllib3<1.23,>=1.21.1
Best match: urllib3 1.22
Processing urllib3-1.22-py2.7.egg
Using
/usr/local/google/home/shoyer/open-source/beam/sdks/python/.eggs/urllib3-1.22-py2.7.egg
Searching for idna<2.7,>=2.5
Best match: idna 2.6
Processing idna-2.6-py2.7.egg
Using
/usr/local/google/home/shoyer/open-source/beam/sdks/python/.eggs/idna-2.6-py2.7.egg
Searching for chardet<3.1.0,>=3.0.2
Best match: chardet 3.0.4
Processing chardet-3.0.4-py2.7.egg
Using
/usr/local/google/home/shoyer/open-source/beam/sdks/python/.eggs/chardet-3.0.4-py2.7.egg
Searching for certifi>=2017.4.17
Best match: certifi 2018.1.18
Processing certifi-2018.1.18-py2.7.egg
Using
/usr/local/google/home/shoyer/open-source/beam/sdks/python/.eggs/certifi-2018.1.18-py2.7.egg
Searching for pyhamcrest<2.0,>=1.9
Best match: PyHamcrest 1.9.0
Processing PyHamcrest-1.9.0-py2.7.egg
Using
/usr/local/google/home/shoyer/open-source/beam/sdks/python/.eggs/PyHamcrest-1.9.0-py2.7.egg
running egg_info
writing requirements to apache_beam.egg-info/requires.txt
writing apache_beam.egg-info/PKG-INFO
writing top-level names to apache_beam.egg-info/top_level.txt
writing dependency_links to apache_beam.egg-info/dependency_links.txt
writing entry points to apache_beam.egg-info/entry_points.txt
reading manifest file 'apache_beam.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'README.md'
warning: no files found matching 'NOTICE'
warning: no files found matching 'LICENSE'
writing manifest file 'apache_beam.egg-info/SOURCES.txt'
running build_ext
test_error_traceback_includes_user_code
(apache_beam.runners.portability.fn_api_runner_test.FnApiRunnerTestWithGrpc)
... Traceback (most recent call last):
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/worker/sdk_worker.py",
line 119, in _execute
response = task()
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/worker/sdk_worker.py",
line 155, in <lambda>
self._execute(lambda: worker.do_instruction(work), work)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/worker/sdk_worker.py",
line 201, in do_instruction
request.instruction_id)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/worker/sdk_worker.py",
line 218, in process_bundle
processor.process_bundle(instruction_id)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/worker/bundle_processor.py",
line 286, in process_bundle
op.start()
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/worker/operations.py",
line 238, in start
self.output(windowed_value)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/worker/operations.py",
line 159, in output
cython.cast(Receiver,
self.receivers[output_index]).receive(windowed_value)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/worker/operations.py",
line 85, in receive
cython.cast(Operation, consumer).process(windowed_value)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/worker/operations.py",
line 392, in process
self.dofn_receiver.receive(o)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/common.py",
line 488, in receive
self.process(windowed_value)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/common.py",
line 496, in process
self._reraise_augmented(exn)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/common.py",
line 547, in _reraise_augmented
six.reraise(type(new_exn), new_exn, original_traceback)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/common.py",
line 494, in process
self.do_fn_invoker.invoke_process(windowed_value)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/common.py",
line 284, in invoke_process
windowed_value, self.process_method(windowed_value.value))
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/transforms/core.py",
line 972, in <lambda>
wrapper = lambda x: [fn(x)]
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py",
line 287, in first
return second(x)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py",
line 290, in second
return third(x)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py",
line 293, in third
raise ValueError('x')
ValueError: x [while running 'Map(first)']
ERROR:root:Error processing instruction bundle_1. Original traceback is
Traceback (most recent call last):
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/worker/sdk_worker.py",
line 119, in _execute
response = task()
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/worker/sdk_worker.py",
line 155, in <lambda>
self._execute(lambda: worker.do_instruction(work), work)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/worker/sdk_worker.py",
line 201, in do_instruction
request.instruction_id)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/worker/sdk_worker.py",
line 218, in process_bundle
processor.process_bundle(instruction_id)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/worker/bundle_processor.py",
line 286, in process_bundle
op.start()
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/worker/operations.py",
line 238, in start
self.output(windowed_value)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/worker/operations.py",
line 159, in output
cython.cast(Receiver,
self.receivers[output_index]).receive(windowed_value)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/worker/operations.py",
line 85, in receive
cython.cast(Operation, consumer).process(windowed_value)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/worker/operations.py",
line 392, in process
self.dofn_receiver.receive(o)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/common.py",
line 488, in receive
self.process(windowed_value)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/common.py",
line 496, in process
self._reraise_augmented(exn)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/common.py",
line 547, in _reraise_augmented
six.reraise(type(new_exn), new_exn, original_traceback)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/common.py",
line 494, in process
self.do_fn_invoker.invoke_process(windowed_value)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/common.py",
line 284, in invoke_process
windowed_value, self.process_method(windowed_value.value))
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/transforms/core.py",
line 972, in <lambda>
wrapper = lambda x: [fn(x)]
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py",
line 287, in first
return second(x)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py",
line 290, in second
return third(x)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py",
line 293, in third
raise ValueError('x')
ValueError: x [while running 'Map(first)']
Traceback (most recent call last):
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/worker/sdk_worker.py",
line 119, in _execute
response = task()
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/worker/sdk_worker.py",
line 155, in <lambda>
self._execute(lambda: worker.do_instruction(work), work)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/worker/sdk_worker.py",
line 201, in do_instruction
request.instruction_id)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/worker/sdk_worker.py",
line 218, in process_bundle
processor.process_bundle(instruction_id)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/worker/bundle_processor.py",
line 286, in process_bundle
op.start()
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/worker/operations.py",
line 238, in start
self.output(windowed_value)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/worker/operations.py",
line 159, in output
cython.cast(Receiver,
self.receivers[output_index]).receive(windowed_value)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/worker/operations.py",
line 85, in receive
cython.cast(Operation, consumer).process(windowed_value)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/worker/operations.py",
line 392, in process
self.dofn_receiver.receive(o)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/common.py",
line 488, in receive
self.process(windowed_value)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/common.py",
line 496, in process
self._reraise_augmented(exn)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/common.py",
line 547, in _reraise_augmented
six.reraise(type(new_exn), new_exn, original_traceback)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/common.py",
line 494, in process
self.do_fn_invoker.invoke_process(windowed_value)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/common.py",
line 284, in invoke_process
windowed_value, self.process_method(windowed_value.value))
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/transforms/core.py",
line 972, in <lambda>
wrapper = lambda x: [fn(x)]
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py",
line 287, in first
return second(x)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py",
line 290, in second
return third(x)
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py",
line 293, in third
raise ValueError('x')
ValueError: x [while running 'Map(first)']
FAIL
======================================================================
FAIL: test_error_traceback_includes_user_code
(apache_beam.runners.portability.fn_api_runner_test.FnApiRunnerTestWithGrpc)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py",
line 304, in test_error_traceback_includes_user_code
self.assertIn('second', message)
AssertionError: 'second' not found in 'Traceback (most recent call last):\n
File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py",
line 297, in test_error_traceback_includes_user_code\n p | beam.Create([0])
| beam.Map(first) # pylint: disable=expression-not-assigned\n File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/pipeline.py",
line 409, in __exit__\n self.run().wait_until_finish()\n File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/pipeline.py",
line 389, in run\n self.to_runner_api(), self.runner,
self._options).run(False)\n File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/pipeline.py",
line 402, in run\n return self.runner.run_pipeline(self)\n File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
line 215, in run_pipeline\n return
self.run_via_runner_api(pipeline.to_runner_api())\n File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
line 218, in run_via_runner_api\n return
self.run_stages(*self.create_stages(pipeline_proto))\n File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
line 837, in run_stages\n pcoll_buffers,
safe_coders).process_bundle.metrics\n File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
line 938, in run_stage\n
self._progress_frequency).process_bundle(data_input, data_output)\n File
"/usr/local/google/home/shoyer/open-source/beam/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
line 1131, in process_bundle\n raise
RuntimeError(result.error)\nRuntimeError: x [while running \'Map(first)\']\n'
----------------------------------------------------------------------
Ran 1 test in 0.049s
FAILED (failures=1)
Test failed: <unittest.runner.TextTestResult run=1 errors=0 failures=1>
error: Test failed: <unittest.runner.TextTestResult run=1 errors=0
failures=1>
```
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 87415)
Time Spent: 3h 50m (was: 3h 40m)
> Stacktraces from exceptions in user code should be preserved in the Python SDK
> ------------------------------------------------------------------------------
>
> Key: BEAM-3956
> URL: https://issues.apache.org/jira/browse/BEAM-3956
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Reporter: Stephan Hoyer
> Priority: Major
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> Currently, Beam's Python SDK loses stacktraces for exceptions. It does
> helpfully add a tag like "[while running StageA]" to exception error
> messages, but that doesn't include the stacktrace of Python functions being
> called.
> Including the full stacktraces would make a big difference for the ease of
> debugging Beam pipelines when things go wrong.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
