Github user colorant commented on the pull request:
https://github.com/apache/spark/pull/907#issuecomment-44496929
And the implementation here is try to minimize the impact to current spark
code base. Especially the shuffle part of code as described in the jira. which
might be improved later together with shuffle framework itself. currently, Just
try to make the wheel runs.
And there are improvements needed on the quota control part of logic. Say
enhance the quota control to make it work on path/disk bases instead of store
bases as a whole to reduce the chance of out of disk space issue ( though as
whole the store's quotation is not overrun yet) etc. And would benefit from
catching varous out of disk space exception etc.
However, if the disk space is not a issue. I think this one just works
fine. And I also have tried to make sure that if not extra config is done on
diskstore. the general data flow path is almost identical as the current
approaching, overhead is neglec table.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
