Bryan Cutler created SPARK-27660:
------------------------------------
Summary: Allow PySpark toLocalIterator to pre-fetch data
Key: SPARK-27660
URL: https://issues.apache.org/jira/browse/SPARK-27660
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 3.0.0
Reporter: Bryan Cutler
In SPARK-23961, data was no longer prefetched so that the local iterator could
close cleanly in the case of not consuming all of the data. If the user intends
to iterate over all elements, then prefetching data could bring back any lost
performance. We would need to run some tests to see if the performance is worth
it due to additional complexity and extra usage of memory. The option to
prefetch could adding as a user conf, and it's possible this could improve the
Scala toLocalIterator also.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
