[
https://issues.apache.org/jira/browse/SPARK-25569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-25569:
----------------------------------
Affects Version/s: (was: 3.0.0)
3.1.0
> Failing a Spark job when an accumulator cannot be updated
> ---------------------------------------------------------
>
> Key: SPARK-25569
> URL: https://issues.apache.org/jira/browse/SPARK-25569
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.1.0
> Reporter: Shixiong Zhu
> Priority: Major
>
> Currently, when Spark fails to merge an accumulator updates from a task, it
> will not fail the task. (See
> https://github.com/apache/spark/blob/b7d80349b0e367d78cab238e62c2ec353f0f12b3/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1266)
> So an accumulator update failure may be ignored silently. Some user may want
> to use accumulators in business critical things, and would like to fail a job
> when an accumulator is broken.
> We can add a flag to always fail a Spark job when hitting an accumulator
> failure. Or we can add a new property to an accumulator and only fail a spark
> job when such accumulator fails.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
