Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/7966#issuecomment-129039966
> You shouldn't assume those directories are under the Mesos sandbox. They
are user configurable (in the Hadoop world there would be several of them on
different disks to allow parallel reads/writes).
Maybe that's how it works in mesos, and in such case, you should figure out
how to fix it (in Mesos or in Spark). Yes, in YARN you can configure multiple
of these directories but (i) Spark handles that just fine and (ii) YARN creates
per-*application* directories, where the shuffle files are written. When
containers (= executors) go away, those directories still remain. They're only
deleted after the app finished.
Nothing has changed here at all. You had to *add* this chmod, which means
that you intentionally tried to change the behavior. And that broke existing
code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
