[
https://issues.apache.org/jira/browse/BEAM-8397?focusedWorklogId=334469&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-334469
]
ASF GitHub Bot logged work on BEAM-8397:
----------------------------------------
Author: ASF GitHub Bot
Created on: 26/Oct/19 00:41
Start Date: 26/Oct/19 00:41
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #9881: [BEAM-8397]
Fix infinite recursion errors in test_remote_runner_display_data_test.
URL: https://github.com/apache/beam/pull/9881#discussion_r339277535
##########
File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py
##########
@@ -235,40 +259,19 @@ def test_remote_runner_display_data(self):
p = Pipeline(remote_runner,
options=PipelineOptions(self.default_properties))
- # TODO: Should not subclass ParDo. Switch to PTransform as soon as
- # composite transforms support display data.
- class SpecialParDo(beam.ParDo):
- def __init__(self, fn, now):
- super(SpecialParDo, self).__init__(fn)
- self.fn = fn
- self.now = now
-
- # Make this a list to be accessible within closure
- def display_data(self):
- return {'asubcomponent': self.fn,
- 'a_class': SpecialParDo,
- 'a_time': self.now}
-
- class SpecialDoFn(beam.DoFn):
- def display_data(self):
- return {'dofn_value': 42}
-
- def process(self):
- pass
-
now = datetime.now()
# pylint: disable=expression-not-assigned
(p | ptransform.Create([1, 2, 3, 4, 5])
| 'Do' >> SpecialParDo(SpecialDoFn(), now))
- p.run()
+ # TODO(BEAM-366) Enable runner API on this test.
+ p.run(test_runner_api=False)
Review comment:
As per BEAM-366, Runner API does not plumb through display_data for
composite transforms. With this change we no longer have recursion errors. The
recursion errors were caught in [1] and triggered the same codepath as when we
set test_runner_api=False in [2], that's why the test was passing. However
sometimes recursion error was not caught in [1], but was caught by Python
IDE/debugger, and sometimes [3] Python 3.7 does emit/catch RecursionError. In
these scenarios, the test used to fail.
[1]
https://github.com/apache/beam/blob/c3c5999e3a82d865810f799767c179e2c17d304b/sdks/python/apache_beam/pipeline.py#L599
[2]
https://github.com/apache/beam/blob/c3c5999e3a82d865810f799767c179e2c17d304b/sdks/python/apache_beam/pipeline.py#L402
[3] https://bugs.python.org/issue38593
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 334469)
Time Spent: 2h (was: 1h 50m)
> DataflowRunnerTest.test_remote_runner_display_data fails due to infinite
> recursion during pickling.
> ---------------------------------------------------------------------------------------------------
>
> Key: BEAM-8397
> URL: https://issues.apache.org/jira/browse/BEAM-8397
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
> Reporter: Valentyn Tymofieiev
> Assignee: Valentyn Tymofieiev
> Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
> `python ./setup.py test -s
> apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest.test_remote_runner_display_data`
> passes.
> `tox -e py37-gcp` passes if Beam depends on dill==0.3.0, but fails if Beam
> depends on dill==0.3.1.1.`python ./setup.py nosetests --tests
> 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data`
> fails currently if run on master.
> The failure indicates infinite recursion during pickling:
> {noformat}
> test_remote_runner_display_data
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) ...
> Fatal Python error: Cannot recover from stack overflow.
> Current thread 0x00007f9d700ed740 (most recent call first):
> File "/usr/lib/python3.7/pickle.py", line 479 in get
> File "/usr/lib/python3.7/pickle.py", line 497 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1394 in save_function
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 882 in _batch_setitems
> File "/usr/lib/python3.7/pickle.py", line 856 in save_dict
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 910 in save_module_dict
> File
> "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py",
> line 198 in new_save_module_dict
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py",
> line 114 in wrapper
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 771 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1137 in save_cell
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 771 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1394 in save_function
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 882 in _batch_setitems
> File "/usr/lib/python3.7/pickle.py", line 856 in save_dict
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 910 in save_module_dict
> File
> "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py",
> line 198 in new_save_module_dict
> ...
> {noformat}
> cc: [~yoshiki.obata]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
