Github user viirya commented on the issue:
https://github.com/apache/spark/pull/16633
@rxin even it breaks the RDD job chain. I think it is still useful in some
cases, for example, the number of partitions is big and you only need to get
one or few partitions to satisfy the limit.
Broken RDD job chain means we do an extra scan of the few partitions. You
can save the time to scan all other partitions and shuffling.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
