Robert Bradshaw created BEAM-12204:
--------------------------------------
Summary: Portable Java caches entirety of iterable side inputs.
Key: BEAM-12204
URL: https://issues.apache.org/jira/browse/BEAM-12204
Project: Beam
Issue Type: Bug
Components: java-fn-execution
Reporter: Robert Bradshaw
Assignee: Robert Bradshaw
Fix For: 2.30.0
Though the data is read over the state channel in a paginated way, it is
wrapped in LazyCachingIteratorToIterable which forces the entire iterable to be
cached. On the other hand, not caching at all for re-iteration could be
expensive. We should do what Python does and cache a bit (e.g. the first page)
and still support re-iteration by re-requesting the data over the state
channel.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
