Github user chicagobuss commented on the issue:
https://github.com/apache/spark/pull/18209
I don't see why spark should have native mesos support but not native nomad
support - it doesn't make sense.
We heavily rely on nomad for all ephemeral workload scheduling and if this
doesn't go into the trunk people will always have to follow a cumbersome
patching (or just "don't run the real thing, run this instead" process.
I think it's worth noting why we don't want to run Mesos - when we run a
spark workload in the cloud, we provision tens of thousands of VMs and
immediately want to start using them for a short burst of work. When we're
done, it all goes away. Mesos does not handle this sort of extreme elasticity
very well, but nomad excels at it.
It's a win for everyone when there's more choice - especially since I'm
sure hashicorp will support the ongoing efforts to support and maintain this
integration.
Thanks for the hard work on this, @barnardb. We tested it thoroughly and
found it great for our needs. Would love to see this pushed through.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
