[
https://issues.apache.org/jira/browse/BEAM-10260?focusedWorklogId=446353&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-446353
]
ASF GitHub Bot logged work on BEAM-10260:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 16/Jun/20 08:15
Start Date: 16/Jun/20 08:15
Worklog Time Spent: 10m
Work Description: mxm commented on a change in pull request #12012:
URL: https://github.com/apache/beam/pull/12012#discussion_r440644321
##########
File path: sdks/python/apache_beam/runners/worker/sdk_worker.py
##########
@@ -941,15 +942,29 @@ def extend(self,
is_cached=False
):
# type: (...) -> _Future
+ # Make sure the input is a list of elements
+ elements = list(elements)
Review comment:
Yes, though we will iterate twice over `elements`. 1) for the cache 2)
for serializing the elements for the runner. Materializing to a list ensures
that the results will be the same. An Iterable could theoretically return
different results.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 446353)
Time Spent: 1h (was: 50m)
> Nested iterable in state cache exceeds Python recursion limit
> -------------------------------------------------------------
>
> Key: BEAM-10260
> URL: https://issues.apache.org/jira/browse/BEAM-10260
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-harness
> Reporter: Maximilian Michels
> Assignee: Maximilian Michels
> Priority: P2
> Time Spent: 1h
> Remaining Estimate: 0h
>
> With state caching enabled, the changes in BEAM-8298 lead to an unbounded
> nesting of state iterables when appending to a bag state, quickly resulting
> in an exception:
> {noformat}
> events = list(event_bag.read())
> File
> "/code/venvs/venv/lib/python3.6/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 488, in __iter__ for elem in self.first:
> File
> "/code/venvs/venv/lib/python3.6/site-packages/apache_beam/runners/worker/statecache.py",
> line 186, in __iter__ for item in value:
> File
> "/code/venvs/venv/lib/python3.6/site-packages/apache_beam/runners/worker/statecache.py",
> line 186, in __iter__ for item in value:
> File
> "/code/venvs/venv/lib/python3.6/site-packages/apache_beam/runners/worker/statecache.py",
> line 186, in __iter__ for item in value: [Previous line repeated 977
> more times]
> RecursionError: maximum recursion depth exceeded while calling a Python
> object
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
