[
https://issues.apache.org/jira/browse/BEAM-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17546683#comment-17546683
]
Kenneth Knowles commented on BEAM-3290:
---------------------------------------
This issue has been migrated to https://github.com/apache/beam/issues/18529
> Construct iterators directly if possible to allow spilling to disk
> ------------------------------------------------------------------
>
> Key: BEAM-3290
> URL: https://issues.apache.org/jira/browse/BEAM-3290
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
> Reporter: Holden Karau
> Priority: P3
>
> When you construct a collection first and convert it to an iterator you force
> Spark to evaluate the entire input partition before it can get the first
> element off the output. This breaks some of the spilling to disk Spark can do
> otherwise. Instead chain operations on Iterators.
> This is only possible in the Java API for Spark 2 and above (and that's my
> fault from back in my work in the Spark project).
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
