Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/3861#issuecomment-74950927
We spoke a bit offline about this, but my feeling was that the best thing
here might be to add a way to launch the shuffle service as a standalone
application (initially, not one managed by Mesos) so that it can be shared
across Spark applications. That would involve writing some simple launching
scripts for it in a similar way to existing daemons we launch, and you'd ask
users to launch the shuffle service similar to other storage systems like HDFS.
That's very simple and would avoid diverging a lot between Mesos and the other
modes. And longer term we could actually have a single shared shuffle service
that is scheduled by mesos.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
