Github user tejasapatil commented on the issue:
https://github.com/apache/spark/pull/18209
Given that at Facebook we use our own in-house scheduler, I see why people
would want to see their scheduler impls added right in Spark codebase as a
first class citizen. Like @srowen said, this should not be part of Spark
codebase. There would be many schedulers that would be developed in future and
if we keep adding those to Spark, it will be lot of maintenance burden. Mesos
and YARN have their place in the codebase given that they were there in early
days of the project and are widely used in deployments (not saying that this is
good or bad). This PR is making things worse for future by taking the easier
route of adding yet another `if ..else` or `switch case` for a new scheduler
impl. One can blame that this should have been cleanly right in the start for
mesos / YARN but to give benefit of doubt to original authors, they didn't
expect new schedulers to come up in future.
I would recommend to add interfaces in Spark code to abstract out things so
that plugging in new schedulers would be as simple as adding a jar to the
classpath. As a matter of fact, we are currently able to use Spark with our own
scheduler by extending the existing interfaces. The only patching we have to do
is 3-4 lines which is not super pain during upgrades. Seems like you need more
than the current interfaces.
Having said this, I do appreciate the efforts put in this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
