Github user viirya commented on the issue:
https://github.com/apache/spark/pull/16633
we don't need accurate number. we can have a confident margin.
the bad with broken rdd chain is re-processing the rows. anything else?
I don't think it is worth changing core and scheduler for this purpose. too
risk and might introduce new bugs.
we still can avoid shuffling for 2. we don't need to shuffle those
partitions.
On Jan 20, 2017 11:08 AM, "Fei Wang" <[email protected]> wrote:
For 1, my idea is not use the proposal in this PR,
1. how you determine total rows in all partitions are (much) more than
limit number. and then go into this code path and how to decide the much
more than, i can not use cbo estimate stats here because the locallimit
plan maybe complex and we can not ensure the accuracy of the estimate row
number.
2 as @rxin <https://github.com/rxin> suggest, this break the rdd chain
So for 1, i think it need some improvement of spark core and scheduler as i
mentioned above
For 2 it is ok to me, the solution is the same with i described above(still
shuffle +shuffle to multi partition + modified mapoutput statistics), right?
â
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<https://github.com/apache/spark/pull/16633#issuecomment-273966102>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAEM92HSe9tYMjebITrGDZnSfeSDeIdbks5rUCVBgaJpZM4Lm-g1>
.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
