Github user markhamstra commented on the issue:
https://github.com/apache/spark/pull/17113
@mridulm Correct, turning task interruption on by default is not so much a
matter of Spark itself handling it well as it is a possible (though not
completely known) issue with lower layer libraries not handling interruption
well. The original concern with HDFS is likely fixed now, but there are similar
concerns with Cassandra and other libraries. Logically, we'd like to interrupt
Tasks when associated Jobs or Stages are killed in the DAGScheduler. In
practice, nobody knows right now how to do that safely in all circumstances, so
the default is to not attempt to interrupt the tasks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
