Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/242#issuecomment-39372516
Hey @holdenk, I actually thought of a problem with using Iterator here --
users won't be able to cache() an RDD of Iterators in a meaningful way. (After
the first time you read it, all iterators will be exhausted). We should use
Iterable instead, both in Scala and in Java. This will make the implementation
tougher when we do go to on-disk storage, and we may still have to do special
tricks for caching (cause we don't just want to cache an object that knows how
to read data across the network), but it's a more appropriate API.
You should do something equivalent in Python; it already has a concept of
iterables
(http://www.shutupandship.com/2012/01/understanding-python-iterables-and.html).
CC @rxin, @aarondav in case there are other ideas.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
