[
https://issues.apache.org/jira/browse/BEAM-8397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16958417#comment-16958417
]
Valentyn Tymofieiev commented on BEAM-8397:
-------------------------------------------
[~udim] pointed out "asubcomponent" entry is associated with an instance of
HasDisplayData, so instead of adding "asubcomponent" entry, we accumulate
associated display_data values as per [1], and thus populate the dofn_value
key. so the scenario of the existing test makes sense.
However, when we move SpecialDoFn, SpecialParDo out of
test_remote_runner_display_data method, the display_data suddenly becomes:
{noformat}
[{'key': 'dofn_value', 'namespace':
'apache_beam.runners.dataflow.dataflow_runner_test.SpecialDoFn', 'type':
'INTEGER', 'value': 42},
{'key': 'fn', 'namespace': 'apache_beam.transforms.core.ParDo', 'shortValue':
'SpecialDoFn', 'label': 'Transform Function', 'type': 'STRING', 'value':
'apache_beam.runners.dataflow.dataflow_runner_test.SpecialDoFn'}]
{noformat}
While the expected display_data is
{noformat}
[{'key': 'a_time, 'namespace':
'apache_beam.runners.dataflow.dataflow_runner_test.SpecialParDo', 'type':
'TIMESTAMP', 'value': 1571829781919, '},
{'key': 'a_class', 'shortValue': 'SpecialParDo', 'namespace':
'apache_beam.runners.dataflow.dataflow_runner_test.SpecialParDo', 'type':
'STRING', 'value':
'apache_beam.runners.dataflow.dataflow_runner_test.SpecialParDo'},
{'key': 'dofn_value', 'namespace':
'apache_beam.runners.dataflow.dataflow_runner_test.SpecialDoFn', 'type':
'INTEGER', 'value': 42,}]
{noformat}
It appears that display_data method of SpecialParDo is no longer called, and
instead display_data is populated from display_data of ParDo class, note the
namespace is transforms.core.ParDo. This rootcause of this behavior is unclear
so far.
[1]
[https://github.com/apache/beam/blob/8e574a177794aa04345cf8cd0c2ce1717ee40ec6/sdks/python/apache_beam/transforms/display.py#L100]
> DataflowRunnerTest.test_remote_runner_display_data fails due to infinite
> recursion during pickling.
> ---------------------------------------------------------------------------------------------------
>
> Key: BEAM-8397
> URL: https://issues.apache.org/jira/browse/BEAM-8397
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
> Reporter: Valentyn Tymofieiev
> Assignee: Valentyn Tymofieiev
> Priority: Major
>
> `python ./setup.py test -s
> apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest.test_remote_runner_display_data`
> passes.
> `tox -e py37-gcp` passes if Beam depends on dill==0.3.0, but fails if Beam
> depends on dill==0.3.1.1.`python ./setup.py nosetests --tests
> 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data`
> fails currently if run on master.
> The failure indicates infinite recursion during pickling:
> {noformat}
> test_remote_runner_display_data
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) ...
> Fatal Python error: Cannot recover from stack overflow.
> Current thread 0x00007f9d700ed740 (most recent call first):
> File "/usr/lib/python3.7/pickle.py", line 479 in get
> File "/usr/lib/python3.7/pickle.py", line 497 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1394 in save_function
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 882 in _batch_setitems
> File "/usr/lib/python3.7/pickle.py", line 856 in save_dict
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 910 in save_module_dict
> File
> "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py",
> line 198 in new_save_module_dict
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py",
> line 114 in wrapper
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 771 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1137 in save_cell
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 771 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1394 in save_function
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 882 in _batch_setitems
> File "/usr/lib/python3.7/pickle.py", line 856 in save_dict
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 910 in save_module_dict
> File
> "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py",
> line 198 in new_save_module_dict
> ...
> {noformat}
> cc: [~yoshiki.obata]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
