Github user scwf commented on the issue:
https://github.com/apache/spark/pull/16633
For 1, my idea is not use the proposal in this PR,
1. how you determine `total rows in all partitions are (much) more than
limit number.` and then go into this code path and how to decide the `much more
than`, i can not use cbo estimate stats here because the locallimit plan maybe
complex and we can not ensure the accuracy of the estimate row number.
2 as @rxin suggest, this break the rdd chain
So for 1, i think it need some improvement of spark core and scheduler as i
mentioned above
For 2 it is ok to me, the solution is the same with i described above(still
shuffle +shuffle to multi partition + modified mapoutput statistics), right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
