Github user SaintBacchus commented on the pull request:
https://github.com/apache/spark/pull/5152#issuecomment-85334365
@sryza it's very close to the 'coalesce', but 'coalesce' can't make sure
the next compution will be in the same executor and this cause network transfor
The advantage of 'processCoalesce' is : reduce the network transfor and
save the num of cores;
The disadvantage is also that: it almost only can be used in the small
cached rdd.
BTW, in this patch I had expose an interface of executor level task
scheduler.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
