Github user mridulm commented on the pull request:
https://github.com/apache/spark/pull/10045#issuecomment-161130625
@kayousterhout Permanently disabling task on an executor can be problematic
for a few reasons :
a) When locality timeout's are aggressively high (user config, so we dont
control it).
b) When it is very suboptimal to move task from the executor (usually (a))
- particularly relevant when data is stored off-VM, like in accelerator card,
gpu, etc.
c) Any infinite timeout is risky - a very aggresively high default value
which can be controlled by user would be my preference to permanently disabling
it.
Having said that, as I mentioned before, would prefer if we just eliminated
this - I was never expecting this to survive for this long.
(b) actually results in a lot of task failures due to transient issues
(particularly at dev time) - but the cost of moving data out is extremely
expensive.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
