Github user dragos commented on the pull request:
https://github.com/apache/spark/pull/7966#issuecomment-128689283
You shouldn't assume those directories are under the Mesos sandbox. They
are user configurable (in the Hadoop world there would be several of them on
different disks to allow parallel reads/writes). Also, shuffle files live there
so you can't delete them when the executor exits, or dynamic allocation will
break and Spark won't find them anymore. This is the bug that @tnachen reported
first on the PR to add dynamic allocation on Mesos.
That chmod was simply following what was done before, so I'm surprised
there's any regression in Yarn. As far as I could tell at the time, Yarn mode
took a different code path. Maybe some changes merged later unified some logic?
Given that standalone mode now also allows dynamic allocation...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
