[jira] [Work logged] (BEAM-8397) DataflowRunnerTest.test_remote_runner_display_data fails due to infinite recursion during pickling.
[
https://issues.apache.org/jira/browse/BEAM-8397?focusedWorklogId=334470=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-334470
]
ASF GitHub Bot logged work on BEAM-8397:
Author: ASF GitHub Bot
Created on: 26/Oct/19 00:41
Start Date: 26/Oct/19 00:41
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #9881: [BEAM-8397]
Fix infinite recursion errors in test_remote_runner_display_data_test.
URL: https://github.com/apache/beam/pull/9881#discussion_r339277535
##
File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py
##
@@ -235,40 +259,19 @@ def test_remote_runner_display_data(self):
p = Pipeline(remote_runner,
options=PipelineOptions(self.default_properties))
-# TODO: Should not subclass ParDo. Switch to PTransform as soon as
-# composite transforms support display data.
-class SpecialParDo(beam.ParDo):
- def __init__(self, fn, now):
-super(SpecialParDo, self).__init__(fn)
-self.fn = fn
-self.now = now
-
- # Make this a list to be accessible within closure
- def display_data(self):
-return {'asubcomponent': self.fn,
-'a_class': SpecialParDo,
-'a_time': self.now}
-
-class SpecialDoFn(beam.DoFn):
- def display_data(self):
-return {'dofn_value': 42}
-
- def process(self):
-pass
-
now = datetime.now()
# pylint: disable=expression-not-assigned
(p | ptransform.Create([1, 2, 3, 4, 5])
| 'Do' >> SpecialParDo(SpecialDoFn(), now))
-p.run()
+# TODO(BEAM-366) Enable runner API on this test.
+p.run(test_runner_api=False)
Review comment:
As per BEAM-366, Runner API does not plumb through display_data for
composite transforms. With this change we no longer have recursion errors. The
recursion errors were caught in [1] and triggered the same codepath as when we
set test_runner_api=False in [2], that's why the test was passing. However
sometimes recursion error was not caught in [1], but was caught by Python
IDE/debugger, and sometimes Python 3.7 does emit/catch RecursionError [3]. In
these scenarios, the test used to fail.
[1]
https://github.com/apache/beam/blob/c3c5999e3a82d865810f799767c179e2c17d304b/sdks/python/apache_beam/pipeline.py#L599
[2]
https://github.com/apache/beam/blob/c3c5999e3a82d865810f799767c179e2c17d304b/sdks/python/apache_beam/pipeline.py#L402
[3] https://bugs.python.org/issue38593
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 334470)
Time Spent: 2h 10m (was: 2h)
> DataflowRunnerTest.test_remote_runner_display_data fails due to infinite
> recursion during pickling.
> ---
>
> Key: BEAM-8397
> URL: https://issues.apache.org/jira/browse/BEAM-8397
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> `python ./setup.py test -s
> apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest.test_remote_runner_display_data`
> passes.
> `tox -e py37-gcp` passes if Beam depends on dill==0.3.0, but fails if Beam
> depends on dill==0.3.1.1.`python ./setup.py nosetests --tests
> 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data`
> fails currently if run on master.
> The failure indicates infinite recursion during pickling:
> {noformat}
> test_remote_runner_display_data
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) ...
> Fatal Python error: Cannot recover from stack overflow.
> Current thread 0x7f9d700ed740 (most recent call first):
> File "/usr/lib/python3.7/pickle.py", line 479 in get
> File "/usr/lib/python3.7/pickle.py", line 497 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1394 in save_function
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 882 in _batch_setitems
> File "/usr/lib/python3.7/pickle.py", line 856 in
[jira] [Work logged] (BEAM-8397) DataflowRunnerTest.test_remote_runner_display_data fails due to infinite recursion during pickling.
[
https://issues.apache.org/jira/browse/BEAM-8397?focusedWorklogId=334469=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-334469
]
ASF GitHub Bot logged work on BEAM-8397:
Author: ASF GitHub Bot
Created on: 26/Oct/19 00:41
Start Date: 26/Oct/19 00:41
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #9881: [BEAM-8397]
Fix infinite recursion errors in test_remote_runner_display_data_test.
URL: https://github.com/apache/beam/pull/9881#discussion_r339277535
##
File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py
##
@@ -235,40 +259,19 @@ def test_remote_runner_display_data(self):
p = Pipeline(remote_runner,
options=PipelineOptions(self.default_properties))
-# TODO: Should not subclass ParDo. Switch to PTransform as soon as
-# composite transforms support display data.
-class SpecialParDo(beam.ParDo):
- def __init__(self, fn, now):
-super(SpecialParDo, self).__init__(fn)
-self.fn = fn
-self.now = now
-
- # Make this a list to be accessible within closure
- def display_data(self):
-return {'asubcomponent': self.fn,
-'a_class': SpecialParDo,
-'a_time': self.now}
-
-class SpecialDoFn(beam.DoFn):
- def display_data(self):
-return {'dofn_value': 42}
-
- def process(self):
-pass
-
now = datetime.now()
# pylint: disable=expression-not-assigned
(p | ptransform.Create([1, 2, 3, 4, 5])
| 'Do' >> SpecialParDo(SpecialDoFn(), now))
-p.run()
+# TODO(BEAM-366) Enable runner API on this test.
+p.run(test_runner_api=False)
Review comment:
As per BEAM-366, Runner API does not plumb through display_data for
composite transforms. With this change we no longer have recursion errors. The
recursion errors were caught in [1] and triggered the same codepath as when we
set test_runner_api=False in [2], that's why the test was passing. However
sometimes recursion error was not caught in [1], but was caught by Python
IDE/debugger, and sometimes [3] Python 3.7 does emit/catch RecursionError. In
these scenarios, the test used to fail.
[1]
https://github.com/apache/beam/blob/c3c5999e3a82d865810f799767c179e2c17d304b/sdks/python/apache_beam/pipeline.py#L599
[2]
https://github.com/apache/beam/blob/c3c5999e3a82d865810f799767c179e2c17d304b/sdks/python/apache_beam/pipeline.py#L402
[3] https://bugs.python.org/issue38593
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 334469)
Time Spent: 2h (was: 1h 50m)
> DataflowRunnerTest.test_remote_runner_display_data fails due to infinite
> recursion during pickling.
> ---
>
> Key: BEAM-8397
> URL: https://issues.apache.org/jira/browse/BEAM-8397
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
> `python ./setup.py test -s
> apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest.test_remote_runner_display_data`
> passes.
> `tox -e py37-gcp` passes if Beam depends on dill==0.3.0, but fails if Beam
> depends on dill==0.3.1.1.`python ./setup.py nosetests --tests
> 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data`
> fails currently if run on master.
> The failure indicates infinite recursion during pickling:
> {noformat}
> test_remote_runner_display_data
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) ...
> Fatal Python error: Cannot recover from stack overflow.
> Current thread 0x7f9d700ed740 (most recent call first):
> File "/usr/lib/python3.7/pickle.py", line 479 in get
> File "/usr/lib/python3.7/pickle.py", line 497 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1394 in save_function
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 882 in _batch_setitems
> File "/usr/lib/python3.7/pickle.py", line 856 in save_dict
>
[jira] [Work logged] (BEAM-8397) DataflowRunnerTest.test_remote_runner_display_data fails due to infinite recursion during pickling.
[
https://issues.apache.org/jira/browse/BEAM-8397?focusedWorklogId=334467=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-334467
]
ASF GitHub Bot logged work on BEAM-8397:
Author: ASF GitHub Bot
Created on: 26/Oct/19 00:40
Start Date: 26/Oct/19 00:40
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #9881: [BEAM-8397]
Fix infinite recursion errors in test_remote_runner_display_data_test.
URL: https://github.com/apache/beam/pull/9881#discussion_r339277943
##
File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py
##
@@ -279,8 +282,7 @@ def process(self):
{'type': 'INTEGER', 'namespace': nspace+'SpecialDoFn',
'value': 42, 'key': 'dofn_value'}]
expected_data = sorted(expected_data, key=lambda x:
x['namespace']+x['key'])
Review comment:
ah good catch, not required. PTAL
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 334467)
Time Spent: 1h 50m (was: 1h 40m)
> DataflowRunnerTest.test_remote_runner_display_data fails due to infinite
> recursion during pickling.
> ---
>
> Key: BEAM-8397
> URL: https://issues.apache.org/jira/browse/BEAM-8397
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> `python ./setup.py test -s
> apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest.test_remote_runner_display_data`
> passes.
> `tox -e py37-gcp` passes if Beam depends on dill==0.3.0, but fails if Beam
> depends on dill==0.3.1.1.`python ./setup.py nosetests --tests
> 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data`
> fails currently if run on master.
> The failure indicates infinite recursion during pickling:
> {noformat}
> test_remote_runner_display_data
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) ...
> Fatal Python error: Cannot recover from stack overflow.
> Current thread 0x7f9d700ed740 (most recent call first):
> File "/usr/lib/python3.7/pickle.py", line 479 in get
> File "/usr/lib/python3.7/pickle.py", line 497 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1394 in save_function
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 882 in _batch_setitems
> File "/usr/lib/python3.7/pickle.py", line 856 in save_dict
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 910 in save_module_dict
> File
> "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py",
> line 198 in new_save_module_dict
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py",
> line 114 in wrapper
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 771 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1137 in save_cell
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 771 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1394 in save_function
> File "/usr/lib/python3.7/pickle.py",
[jira] [Work logged] (BEAM-8397) DataflowRunnerTest.test_remote_runner_display_data fails due to infinite recursion during pickling.
[
https://issues.apache.org/jira/browse/BEAM-8397?focusedWorklogId=334465=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-334465
]
ASF GitHub Bot logged work on BEAM-8397:
Author: ASF GitHub Bot
Created on: 26/Oct/19 00:36
Start Date: 26/Oct/19 00:36
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #9881: [BEAM-8397]
Fix infinite recursion errors in test_remote_runner_display_data_test.
URL: https://github.com/apache/beam/pull/9881#discussion_r339277698
##
File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py
##
@@ -235,40 +259,19 @@ def test_remote_runner_display_data(self):
p = Pipeline(remote_runner,
options=PipelineOptions(self.default_properties))
-# TODO: Should not subclass ParDo. Switch to PTransform as soon as
-# composite transforms support display data.
-class SpecialParDo(beam.ParDo):
- def __init__(self, fn, now):
-super(SpecialParDo, self).__init__(fn)
-self.fn = fn
-self.now = now
-
- # Make this a list to be accessible within closure
- def display_data(self):
-return {'asubcomponent': self.fn,
-'a_class': SpecialParDo,
-'a_time': self.now}
-
-class SpecialDoFn(beam.DoFn):
- def display_data(self):
-return {'dofn_value': 42}
-
- def process(self):
-pass
-
now = datetime.now()
# pylint: disable=expression-not-assigned
(p | ptransform.Create([1, 2, 3, 4, 5])
| 'Do' >> SpecialParDo(SpecialDoFn(), now))
-p.run()
+# TODO(BEAM-366) Enable runner API on this test.
+p.run(test_runner_api=False)
Review comment:
Overall, I think BEAM-366 should capture the underlying reason here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 334465)
Time Spent: 1h 40m (was: 1.5h)
> DataflowRunnerTest.test_remote_runner_display_data fails due to infinite
> recursion during pickling.
> ---
>
> Key: BEAM-8397
> URL: https://issues.apache.org/jira/browse/BEAM-8397
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> `python ./setup.py test -s
> apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest.test_remote_runner_display_data`
> passes.
> `tox -e py37-gcp` passes if Beam depends on dill==0.3.0, but fails if Beam
> depends on dill==0.3.1.1.`python ./setup.py nosetests --tests
> 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data`
> fails currently if run on master.
> The failure indicates infinite recursion during pickling:
> {noformat}
> test_remote_runner_display_data
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) ...
> Fatal Python error: Cannot recover from stack overflow.
> Current thread 0x7f9d700ed740 (most recent call first):
> File "/usr/lib/python3.7/pickle.py", line 479 in get
> File "/usr/lib/python3.7/pickle.py", line 497 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1394 in save_function
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 882 in _batch_setitems
> File "/usr/lib/python3.7/pickle.py", line 856 in save_dict
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 910 in save_module_dict
> File
> "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py",
> line 198 in new_save_module_dict
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py",
> line 114 in wrapper
> File "/usr/lib/python3.7/pickle.py", line 504
[jira] [Work logged] (BEAM-8397) DataflowRunnerTest.test_remote_runner_display_data fails due to infinite recursion during pickling.
[
https://issues.apache.org/jira/browse/BEAM-8397?focusedWorklogId=334464=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-334464
]
ASF GitHub Bot logged work on BEAM-8397:
Author: ASF GitHub Bot
Created on: 26/Oct/19 00:36
Start Date: 26/Oct/19 00:36
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #9881: [BEAM-8397]
Fix infinite recursion errors in test_remote_runner_display_data_test.
URL: https://github.com/apache/beam/pull/9881#discussion_r339277535
##
File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py
##
@@ -235,40 +259,19 @@ def test_remote_runner_display_data(self):
p = Pipeline(remote_runner,
options=PipelineOptions(self.default_properties))
-# TODO: Should not subclass ParDo. Switch to PTransform as soon as
-# composite transforms support display data.
-class SpecialParDo(beam.ParDo):
- def __init__(self, fn, now):
-super(SpecialParDo, self).__init__(fn)
-self.fn = fn
-self.now = now
-
- # Make this a list to be accessible within closure
- def display_data(self):
-return {'asubcomponent': self.fn,
-'a_class': SpecialParDo,
-'a_time': self.now}
-
-class SpecialDoFn(beam.DoFn):
- def display_data(self):
-return {'dofn_value': 42}
-
- def process(self):
-pass
-
now = datetime.now()
# pylint: disable=expression-not-assigned
(p | ptransform.Create([1, 2, 3, 4, 5])
| 'Do' >> SpecialParDo(SpecialDoFn(), now))
-p.run()
+# TODO(BEAM-366) Enable runner API on this test.
+p.run(test_runner_api=False)
Review comment:
As per BEAM-366, Runner API does not plumb through display_data for
composite transforms. With this change we no longer have recursion errors. The
recursion errors are caught in [1] and triggered the same codepath as when we
set test_runner_api=False in [2], that's why the test was passing. However
sometimes recursion error was not caught in [1], but was caught by Python
IDE/debugger, and sometimes [3] Python 3.7 does emit/catch RecursionError. In
these scenarios, the test used to fail.
[1]
https://github.com/apache/beam/blob/c3c5999e3a82d865810f799767c179e2c17d304b/sdks/python/apache_beam/pipeline.py#L599
[2]
https://github.com/apache/beam/blob/c3c5999e3a82d865810f799767c179e2c17d304b/sdks/python/apache_beam/pipeline.py#L402
[3] https://bugs.python.org/issue38593
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 334464)
Time Spent: 1.5h (was: 1h 20m)
> DataflowRunnerTest.test_remote_runner_display_data fails due to infinite
> recursion during pickling.
> ---
>
> Key: BEAM-8397
> URL: https://issues.apache.org/jira/browse/BEAM-8397
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> `python ./setup.py test -s
> apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest.test_remote_runner_display_data`
> passes.
> `tox -e py37-gcp` passes if Beam depends on dill==0.3.0, but fails if Beam
> depends on dill==0.3.1.1.`python ./setup.py nosetests --tests
> 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data`
> fails currently if run on master.
> The failure indicates infinite recursion during pickling:
> {noformat}
> test_remote_runner_display_data
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) ...
> Fatal Python error: Cannot recover from stack overflow.
> Current thread 0x7f9d700ed740 (most recent call first):
> File "/usr/lib/python3.7/pickle.py", line 479 in get
> File "/usr/lib/python3.7/pickle.py", line 497 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1394 in save_function
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 882 in _batch_setitems
> File "/usr/lib/python3.7/pickle.py", line 856 in
[jira] [Work logged] (BEAM-8397) DataflowRunnerTest.test_remote_runner_display_data fails due to infinite recursion during pickling.
[
https://issues.apache.org/jira/browse/BEAM-8397?focusedWorklogId=334463=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-334463
]
ASF GitHub Bot logged work on BEAM-8397:
Author: ASF GitHub Bot
Created on: 26/Oct/19 00:35
Start Date: 26/Oct/19 00:35
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #9881: [BEAM-8397]
Fix infinite recursion errors in test_remote_runner_display_data_test.
URL: https://github.com/apache/beam/pull/9881#discussion_r339277535
##
File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py
##
@@ -235,40 +259,19 @@ def test_remote_runner_display_data(self):
p = Pipeline(remote_runner,
options=PipelineOptions(self.default_properties))
-# TODO: Should not subclass ParDo. Switch to PTransform as soon as
-# composite transforms support display data.
-class SpecialParDo(beam.ParDo):
- def __init__(self, fn, now):
-super(SpecialParDo, self).__init__(fn)
-self.fn = fn
-self.now = now
-
- # Make this a list to be accessible within closure
- def display_data(self):
-return {'asubcomponent': self.fn,
-'a_class': SpecialParDo,
-'a_time': self.now}
-
-class SpecialDoFn(beam.DoFn):
- def display_data(self):
-return {'dofn_value': 42}
-
- def process(self):
-pass
-
now = datetime.now()
# pylint: disable=expression-not-assigned
(p | ptransform.Create([1, 2, 3, 4, 5])
| 'Do' >> SpecialParDo(SpecialDoFn(), now))
-p.run()
+# TODO(BEAM-366) Enable runner API on this test.
+p.run(test_runner_api=False)
Review comment:
As per BEAM-366, Runner API does not plumb through display_data for
composite transforms. With this change we no longer have recursion errors. The
recursion errors are caught in [1] and triggered the same codepath as when we
set test_runner_api=False in [2], that's why the test was passing. However
sometimes recursion error was not caught in [1], but was caught by Python
IDE/debugger, and sometimes [3] Python 3.7 does not catch it. In these
scenarios, the test used to fail.
[1]
https://github.com/apache/beam/blob/c3c5999e3a82d865810f799767c179e2c17d304b/sdks/python/apache_beam/pipeline.py#L599
[2]
https://github.com/apache/beam/blob/c3c5999e3a82d865810f799767c179e2c17d304b/sdks/python/apache_beam/pipeline.py#L402
[3] https://bugs.python.org/issue38593
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 334463)
Time Spent: 1h 20m (was: 1h 10m)
> DataflowRunnerTest.test_remote_runner_display_data fails due to infinite
> recursion during pickling.
> ---
>
> Key: BEAM-8397
> URL: https://issues.apache.org/jira/browse/BEAM-8397
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> `python ./setup.py test -s
> apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest.test_remote_runner_display_data`
> passes.
> `tox -e py37-gcp` passes if Beam depends on dill==0.3.0, but fails if Beam
> depends on dill==0.3.1.1.`python ./setup.py nosetests --tests
> 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data`
> fails currently if run on master.
> The failure indicates infinite recursion during pickling:
> {noformat}
> test_remote_runner_display_data
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) ...
> Fatal Python error: Cannot recover from stack overflow.
> Current thread 0x7f9d700ed740 (most recent call first):
> File "/usr/lib/python3.7/pickle.py", line 479 in get
> File "/usr/lib/python3.7/pickle.py", line 497 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1394 in save_function
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 882 in _batch_setitems
> File "/usr/lib/python3.7/pickle.py", line 856 in save_dict
>
[jira] [Work logged] (BEAM-8397) DataflowRunnerTest.test_remote_runner_display_data fails due to infinite recursion during pickling.
[
https://issues.apache.org/jira/browse/BEAM-8397?focusedWorklogId=334462=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-334462
]
ASF GitHub Bot logged work on BEAM-8397:
Author: ASF GitHub Bot
Created on: 26/Oct/19 00:34
Start Date: 26/Oct/19 00:34
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #9881: [BEAM-8397]
Fix infinite recursion errors in test_remote_runner_display_data_test.
URL: https://github.com/apache/beam/pull/9881#discussion_r339277535
##
File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py
##
@@ -235,40 +259,19 @@ def test_remote_runner_display_data(self):
p = Pipeline(remote_runner,
options=PipelineOptions(self.default_properties))
-# TODO: Should not subclass ParDo. Switch to PTransform as soon as
-# composite transforms support display data.
-class SpecialParDo(beam.ParDo):
- def __init__(self, fn, now):
-super(SpecialParDo, self).__init__(fn)
-self.fn = fn
-self.now = now
-
- # Make this a list to be accessible within closure
- def display_data(self):
-return {'asubcomponent': self.fn,
-'a_class': SpecialParDo,
-'a_time': self.now}
-
-class SpecialDoFn(beam.DoFn):
- def display_data(self):
-return {'dofn_value': 42}
-
- def process(self):
-pass
-
now = datetime.now()
# pylint: disable=expression-not-assigned
(p | ptransform.Create([1, 2, 3, 4, 5])
| 'Do' >> SpecialParDo(SpecialDoFn(), now))
-p.run()
+# TODO(BEAM-366) Enable runner API on this test.
+p.run(test_runner_api=False)
Review comment:
As per BEAM-366, Runner API does not plumb through display_data for
composite transforms. With this change we no longer have recursion errors. The
recursion errors are caught in [1] and triggered the same codepath as when we
set test_runner_api=False in [2], that's why the test was passing. However
sometimes recursion error was not caught in [1], but was caught by Python
IDE/debugger, and sometimes [3] Python 3.7 does not catch it. In these
scenarios, the test used to fail.
[1]
https://github.com/apache/beam/blob/c3c5999e3a82d865810f799767c179e2c17d304b/sdks/python/apache_beam/pipeline.py#L402
[2]
https://github.com/apache/beam/blob/c3c5999e3a82d865810f799767c179e2c17d304b/sdks/python/apache_beam/pipeline.py#L599
[3] https://bugs.python.org/issue38593
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 334462)
Time Spent: 1h 10m (was: 1h)
> DataflowRunnerTest.test_remote_runner_display_data fails due to infinite
> recursion during pickling.
> ---
>
> Key: BEAM-8397
> URL: https://issues.apache.org/jira/browse/BEAM-8397
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> `python ./setup.py test -s
> apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest.test_remote_runner_display_data`
> passes.
> `tox -e py37-gcp` passes if Beam depends on dill==0.3.0, but fails if Beam
> depends on dill==0.3.1.1.`python ./setup.py nosetests --tests
> 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data`
> fails currently if run on master.
> The failure indicates infinite recursion during pickling:
> {noformat}
> test_remote_runner_display_data
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) ...
> Fatal Python error: Cannot recover from stack overflow.
> Current thread 0x7f9d700ed740 (most recent call first):
> File "/usr/lib/python3.7/pickle.py", line 479 in get
> File "/usr/lib/python3.7/pickle.py", line 497 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1394 in save_function
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 882 in _batch_setitems
> File "/usr/lib/python3.7/pickle.py", line 856 in save_dict
> File
>
[jira] [Work logged] (BEAM-8397) DataflowRunnerTest.test_remote_runner_display_data fails due to infinite recursion during pickling.
[
https://issues.apache.org/jira/browse/BEAM-8397?focusedWorklogId=334448=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-334448
]
ASF GitHub Bot logged work on BEAM-8397:
Author: ASF GitHub Bot
Created on: 26/Oct/19 00:11
Start Date: 26/Oct/19 00:11
Worklog Time Spent: 10m
Work Description: udim commented on pull request #9881: [BEAM-8397] Fix
infinite recursion errors in test_remote_runner_display_data_test.
URL: https://github.com/apache/beam/pull/9881#discussion_r339275364
##
File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py
##
@@ -235,40 +259,19 @@ def test_remote_runner_display_data(self):
p = Pipeline(remote_runner,
options=PipelineOptions(self.default_properties))
-# TODO: Should not subclass ParDo. Switch to PTransform as soon as
-# composite transforms support display data.
-class SpecialParDo(beam.ParDo):
- def __init__(self, fn, now):
-super(SpecialParDo, self).__init__(fn)
-self.fn = fn
-self.now = now
-
- # Make this a list to be accessible within closure
- def display_data(self):
-return {'asubcomponent': self.fn,
-'a_class': SpecialParDo,
-'a_time': self.now}
-
-class SpecialDoFn(beam.DoFn):
- def display_data(self):
-return {'dofn_value': 42}
-
- def process(self):
-pass
-
now = datetime.now()
# pylint: disable=expression-not-assigned
(p | ptransform.Create([1, 2, 3, 4, 5])
| 'Do' >> SpecialParDo(SpecialDoFn(), now))
-p.run()
+# TODO(BEAM-366) Enable runner API on this test.
+p.run(test_runner_api=False)
Review comment:
Could you add the reason why? IIRC, it's because Runner API does a
pickle-unpickle test which can crash under certain combinations of IDE/debugger
settings and Python versions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 334448)
Time Spent: 1h (was: 50m)
> DataflowRunnerTest.test_remote_runner_display_data fails due to infinite
> recursion during pickling.
> ---
>
> Key: BEAM-8397
> URL: https://issues.apache.org/jira/browse/BEAM-8397
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> `python ./setup.py test -s
> apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest.test_remote_runner_display_data`
> passes.
> `tox -e py37-gcp` passes if Beam depends on dill==0.3.0, but fails if Beam
> depends on dill==0.3.1.1.`python ./setup.py nosetests --tests
> 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data`
> fails currently if run on master.
> The failure indicates infinite recursion during pickling:
> {noformat}
> test_remote_runner_display_data
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) ...
> Fatal Python error: Cannot recover from stack overflow.
> Current thread 0x7f9d700ed740 (most recent call first):
> File "/usr/lib/python3.7/pickle.py", line 479 in get
> File "/usr/lib/python3.7/pickle.py", line 497 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1394 in save_function
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 882 in _batch_setitems
> File "/usr/lib/python3.7/pickle.py", line 856 in save_dict
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 910 in save_module_dict
> File
> "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py",
> line 198 in new_save_module_dict
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
>
[jira] [Work logged] (BEAM-8397) DataflowRunnerTest.test_remote_runner_display_data fails due to infinite recursion during pickling.
[
https://issues.apache.org/jira/browse/BEAM-8397?focusedWorklogId=334449=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-334449
]
ASF GitHub Bot logged work on BEAM-8397:
Author: ASF GitHub Bot
Created on: 26/Oct/19 00:11
Start Date: 26/Oct/19 00:11
Worklog Time Spent: 10m
Work Description: udim commented on pull request #9881: [BEAM-8397] Fix
infinite recursion errors in test_remote_runner_display_data_test.
URL: https://github.com/apache/beam/pull/9881#discussion_r339274728
##
File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py
##
@@ -279,8 +282,7 @@ def process(self):
{'type': 'INTEGER', 'namespace': nspace+'SpecialDoFn',
'value': 42, 'key': 'dofn_value'}]
expected_data = sorted(expected_data, key=lambda x:
x['namespace']+x['key'])
Review comment:
Is this invocation of sorted still necessary, since you've removed the other
one?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 334449)
Time Spent: 1h (was: 50m)
> DataflowRunnerTest.test_remote_runner_display_data fails due to infinite
> recursion during pickling.
> ---
>
> Key: BEAM-8397
> URL: https://issues.apache.org/jira/browse/BEAM-8397
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> `python ./setup.py test -s
> apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest.test_remote_runner_display_data`
> passes.
> `tox -e py37-gcp` passes if Beam depends on dill==0.3.0, but fails if Beam
> depends on dill==0.3.1.1.`python ./setup.py nosetests --tests
> 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data`
> fails currently if run on master.
> The failure indicates infinite recursion during pickling:
> {noformat}
> test_remote_runner_display_data
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) ...
> Fatal Python error: Cannot recover from stack overflow.
> Current thread 0x7f9d700ed740 (most recent call first):
> File "/usr/lib/python3.7/pickle.py", line 479 in get
> File "/usr/lib/python3.7/pickle.py", line 497 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1394 in save_function
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 882 in _batch_setitems
> File "/usr/lib/python3.7/pickle.py", line 856 in save_dict
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 910 in save_module_dict
> File
> "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py",
> line 198 in new_save_module_dict
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py",
> line 114 in wrapper
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 771 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1137 in save_cell
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 771 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1394 in save_function
> File
[jira] [Work logged] (BEAM-8397) DataflowRunnerTest.test_remote_runner_display_data fails due to infinite recursion during pickling.
[
https://issues.apache.org/jira/browse/BEAM-8397?focusedWorklogId=334405=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-334405
]
ASF GitHub Bot logged work on BEAM-8397:
Author: ASF GitHub Bot
Created on: 25/Oct/19 22:31
Start Date: 25/Oct/19 22:31
Worklog Time Spent: 10m
Work Description: tvalentyn commented on issue #9881: [BEAM-8397] Fix
infinite recursion errors in test_remote_runner_display_data_test.
URL: https://github.com/apache/beam/pull/9881#issuecomment-546534027
All tests passed, this is ready for review.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 334405)
Time Spent: 50m (was: 40m)
> DataflowRunnerTest.test_remote_runner_display_data fails due to infinite
> recursion during pickling.
> ---
>
> Key: BEAM-8397
> URL: https://issues.apache.org/jira/browse/BEAM-8397
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> `python ./setup.py test -s
> apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest.test_remote_runner_display_data`
> passes.
> `tox -e py37-gcp` passes if Beam depends on dill==0.3.0, but fails if Beam
> depends on dill==0.3.1.1.`python ./setup.py nosetests --tests
> 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data`
> fails currently if run on master.
> The failure indicates infinite recursion during pickling:
> {noformat}
> test_remote_runner_display_data
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) ...
> Fatal Python error: Cannot recover from stack overflow.
> Current thread 0x7f9d700ed740 (most recent call first):
> File "/usr/lib/python3.7/pickle.py", line 479 in get
> File "/usr/lib/python3.7/pickle.py", line 497 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1394 in save_function
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 882 in _batch_setitems
> File "/usr/lib/python3.7/pickle.py", line 856 in save_dict
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 910 in save_module_dict
> File
> "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py",
> line 198 in new_save_module_dict
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py",
> line 114 in wrapper
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 771 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1137 in save_cell
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 771 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1394 in save_function
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 882 in _batch_setitems
> File "/usr/lib/python3.7/pickle.py", line 856 in save_dict
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 910 in save_module_dict
> File
>
[jira] [Work logged] (BEAM-8397) DataflowRunnerTest.test_remote_runner_display_data fails due to infinite recursion during pickling.
[
https://issues.apache.org/jira/browse/BEAM-8397?focusedWorklogId=334348=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-334348
]
ASF GitHub Bot logged work on BEAM-8397:
Author: ASF GitHub Bot
Created on: 25/Oct/19 20:15
Start Date: 25/Oct/19 20:15
Worklog Time Spent: 10m
Work Description: tvalentyn commented on issue #9881: [BEAM-8397] Fix
infinite recursion errors in test_remote_runner_display_data_test.
URL: https://github.com/apache/beam/pull/9881#issuecomment-546496935
Run Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 334348)
Time Spent: 40m (was: 0.5h)
> DataflowRunnerTest.test_remote_runner_display_data fails due to infinite
> recursion during pickling.
> ---
>
> Key: BEAM-8397
> URL: https://issues.apache.org/jira/browse/BEAM-8397
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> `python ./setup.py test -s
> apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest.test_remote_runner_display_data`
> passes.
> `tox -e py37-gcp` passes if Beam depends on dill==0.3.0, but fails if Beam
> depends on dill==0.3.1.1.`python ./setup.py nosetests --tests
> 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data`
> fails currently if run on master.
> The failure indicates infinite recursion during pickling:
> {noformat}
> test_remote_runner_display_data
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) ...
> Fatal Python error: Cannot recover from stack overflow.
> Current thread 0x7f9d700ed740 (most recent call first):
> File "/usr/lib/python3.7/pickle.py", line 479 in get
> File "/usr/lib/python3.7/pickle.py", line 497 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1394 in save_function
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 882 in _batch_setitems
> File "/usr/lib/python3.7/pickle.py", line 856 in save_dict
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 910 in save_module_dict
> File
> "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py",
> line 198 in new_save_module_dict
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py",
> line 114 in wrapper
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 771 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1137 in save_cell
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 771 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1394 in save_function
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 882 in _batch_setitems
> File "/usr/lib/python3.7/pickle.py", line 856 in save_dict
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 910 in save_module_dict
> File
> "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py",
> line 198
[jira] [Work logged] (BEAM-8397) DataflowRunnerTest.test_remote_runner_display_data fails due to infinite recursion during pickling.
[
https://issues.apache.org/jira/browse/BEAM-8397?focusedWorklogId=334194=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-334194
]
ASF GitHub Bot logged work on BEAM-8397:
Author: ASF GitHub Bot
Created on: 25/Oct/19 15:22
Start Date: 25/Oct/19 15:22
Worklog Time Spent: 10m
Work Description: tvalentyn commented on issue #9881: [BEAM-8397] Fix
infinite recursion errors in test_remote_runner_display_data_test.
URL: https://github.com/apache/beam/pull/9881#issuecomment-546397903
Run Python PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 334194)
Time Spent: 0.5h (was: 20m)
> DataflowRunnerTest.test_remote_runner_display_data fails due to infinite
> recursion during pickling.
> ---
>
> Key: BEAM-8397
> URL: https://issues.apache.org/jira/browse/BEAM-8397
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> `python ./setup.py test -s
> apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest.test_remote_runner_display_data`
> passes.
> `tox -e py37-gcp` passes if Beam depends on dill==0.3.0, but fails if Beam
> depends on dill==0.3.1.1.`python ./setup.py nosetests --tests
> 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data`
> fails currently if run on master.
> The failure indicates infinite recursion during pickling:
> {noformat}
> test_remote_runner_display_data
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) ...
> Fatal Python error: Cannot recover from stack overflow.
> Current thread 0x7f9d700ed740 (most recent call first):
> File "/usr/lib/python3.7/pickle.py", line 479 in get
> File "/usr/lib/python3.7/pickle.py", line 497 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1394 in save_function
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 882 in _batch_setitems
> File "/usr/lib/python3.7/pickle.py", line 856 in save_dict
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 910 in save_module_dict
> File
> "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py",
> line 198 in new_save_module_dict
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py",
> line 114 in wrapper
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 771 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1137 in save_cell
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 771 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1394 in save_function
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 882 in _batch_setitems
> File "/usr/lib/python3.7/pickle.py", line 856 in save_dict
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 910 in save_module_dict
> File
> "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py",
> line
[jira] [Work logged] (BEAM-8397) DataflowRunnerTest.test_remote_runner_display_data fails due to infinite recursion during pickling.
[ https://issues.apache.org/jira/browse/BEAM-8397?focusedWorklogId=333811=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-333811 ] ASF GitHub Bot logged work on BEAM-8397: Author: ASF GitHub Bot Created on: 25/Oct/19 01:13 Start Date: 25/Oct/19 01:13 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #9881: [BEAM-8397] Fix infinite recursion errors in test_remote_runner_display_data_test. URL: https://github.com/apache/beam/pull/9881 test_remote_runner_display_data_test triggers an infinite recursion in apache_beam.internal.pickler module. Beam catches RecursionError, and condsiders the pipeline not suitable for RunnerApi, and the test passes on Direct runner without further pickling. However on some versions of Python interpreter, under certain conditions, the Python interpreter fails to raise RecursionError, which unexpectedly causes the test to fail. This PR removes the recursion error, and explicitly disables RunnerAPI in the test. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [![Build
[jira] [Work logged] (BEAM-8397) DataflowRunnerTest.test_remote_runner_display_data fails due to infinite recursion during pickling.
[
https://issues.apache.org/jira/browse/BEAM-8397?focusedWorklogId=333812=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-333812
]
ASF GitHub Bot logged work on BEAM-8397:
Author: ASF GitHub Bot
Created on: 25/Oct/19 01:13
Start Date: 25/Oct/19 01:13
Worklog Time Spent: 10m
Work Description: tvalentyn commented on issue #9881: [BEAM-8397] Fix
infinite recursion errors in test_remote_runner_display_data_test.
URL: https://github.com/apache/beam/pull/9881#issuecomment-546160444
R: @udim
cc: @robertwb @lazylynx @KevinGG
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 333812)
Time Spent: 20m (was: 10m)
> DataflowRunnerTest.test_remote_runner_display_data fails due to infinite
> recursion during pickling.
> ---
>
> Key: BEAM-8397
> URL: https://issues.apache.org/jira/browse/BEAM-8397
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> `python ./setup.py test -s
> apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest.test_remote_runner_display_data`
> passes.
> `tox -e py37-gcp` passes if Beam depends on dill==0.3.0, but fails if Beam
> depends on dill==0.3.1.1.`python ./setup.py nosetests --tests
> 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data`
> fails currently if run on master.
> The failure indicates infinite recursion during pickling:
> {noformat}
> test_remote_runner_display_data
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) ...
> Fatal Python error: Cannot recover from stack overflow.
> Current thread 0x7f9d700ed740 (most recent call first):
> File "/usr/lib/python3.7/pickle.py", line 479 in get
> File "/usr/lib/python3.7/pickle.py", line 497 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1394 in save_function
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 882 in _batch_setitems
> File "/usr/lib/python3.7/pickle.py", line 856 in save_dict
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 910 in save_module_dict
> File
> "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py",
> line 198 in new_save_module_dict
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py",
> line 114 in wrapper
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 771 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1137 in save_cell
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 771 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 1394 in save_function
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 882 in _batch_setitems
> File "/usr/lib/python3.7/pickle.py", line 856 in save_dict
> File
> "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py",
> line 910 in save_module_dict
> File
>
