Github user aarondav commented on the pull request:
https://github.com/apache/spark/pull/1274#issuecomment-47690237
This change seems reasonable because on large clusters, we occasionally see
a single disk on a single machine is failed, and this may cause the entire
application to crash because the executor will keep getting restarted until the
Master kills the application.
It also allows a more uniform configuration for heterogeneous cluster with
different numbers of disks.
The downside of this behavioral change is that a misconfiguration like
mistyping one of your local dirs may go unnoticed for a while, but this will
hopefully become apparent after a `df` or a look at any of the executor logs.
This fail-fast approach is generally better, but current Spark does not do a
good job communicating the reason for executors that crash immediately upon
startup.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
