Github user Zariel commented on the pull request:
https://github.com/apache/spark/pull/8358#issuecomment-138854320
I dont think there is an issue with running Mesos and wanting to run over
multiple disks, as it is the responsibility of whoever is managing Mesos to
setup this space, running Spark on Mesos should essentially be a black box.
Mesos recommends that to use multiple Mesos work dirs one should use something
like raid or lvm.
`MESOS_DIRECTORY` is a sandbox managed by Mesos itself, its is derived from
the framework id, executor and task and the Mesos `WORK_DIR` option which gets
set when Mesos launches a task.
As currently implemented `SPARK_LOCAL_DIRS` is the lowest priority and in
the if block, being overridden by both YARN and `SPARK_EXECUTOR_DIRS`, I think
it would make more sense to open a separate issue to solve this and make sure
that it does the correct thing with regards to the docs (or update the docs).
Regarding the dynamic allocation issues this can be changed to be disabled
when dynamic allocation is enabled, or a config option can be added to
explicitly enable this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
