Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/11578#issuecomment-199137357
I think this is still useful, if not for getting rid of the iterator
interface. Pull-based iterators are slow in general and fundamentally at odds
with whole stage codegen, which is a push based model. The overhead is not just
an iterator call, but also pushing records generated by the previous operator
into some buffer. Having something that is not whole-stage codegened
artificially breaks the pipeline into two.
It looks like the computations done in sampling is very simple, and I can't
think of a reason why it should be slower than adding an iterator.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
