[
https://issues.apache.org/jira/browse/BEAM-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17145322#comment-17145322
]
Beam JIRA Bot commented on BEAM-3290:
-------------------------------------
This issue was marked "stale-P2" and has not received a public comment in 14
days. It is now automatically moved to P3. If you are still affected by it, you
can comment and move it back to P2.
> Construct iterators directly if possible to allow spilling to disk
> ------------------------------------------------------------------
>
> Key: BEAM-3290
> URL: https://issues.apache.org/jira/browse/BEAM-3290
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
> Reporter: Holden Karau
> Priority: P3
>
> When you construct a collection first and convert it to an iterator you force
> Spark to evaluate the entire input partition before it can get the first
> element off the output. This breaks some of the spilling to disk Spark can do
> otherwise. Instead chain operations on Iterators.
> This is only possible in the Java API for Spark 2 and above (and that's my
> fault from back in my work in the Spark project).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
