[
https://issues.apache.org/jira/browse/SPARK-20087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352750#comment-16352750
]
Imran Rashid commented on SPARK-20087:
--------------------------------------
cc [[email protected]]
I think this makes sense, as accumulators are already biased towards counting
based on the *computation* done, not based on the *data* itself, wrt retries,
failures, etc. But wanted to get some more thoughts, as this is, in a way, a
breaking change in the accumulator api. I'd consider the behavior before a bug
so its OK in my book.
> Include accumulators / taskMetrics when sending TaskKilled to onTaskEnd
> listeners
> ---------------------------------------------------------------------------------
>
> Key: SPARK-20087
> URL: https://issues.apache.org/jira/browse/SPARK-20087
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.1.0
> Reporter: Charles Lewis
> Priority: Major
>
> When tasks end due to an ExceptionFailure, subscribers to onTaskEnd receive
> accumulators / task metrics for that task, if they were still available.
> These metrics are not currently sent when tasks are killed intentionally,
> such as when a speculative retry finishes, and the original is killed (or
> vice versa). Since we're killing these tasks ourselves, these metrics should
> almost always exist, and we should treat them the same way as we treat
> ExceptionFailures.
> Sending these metrics with the TaskKilled end reason makes aggregation across
> all tasks in an app more accurate. This data can inform decisions about how
> to tune the speculation parameters in order to minimize duplicated work, and
> in general, the total cost of an app should include both successful and
> failed tasks, if that information exists.
> PR: https://github.com/apache/spark/pull/17422
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
