[
https://issues.apache.org/jira/browse/SPARK-20087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-20087:
------------------------------------
Assignee: Apache Spark
> Include accumulators / taskMetrics when sending TaskKilled to onTaskEnd
> listeners
> ---------------------------------------------------------------------------------
>
> Key: SPARK-20087
> URL: https://issues.apache.org/jira/browse/SPARK-20087
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.1.0
> Reporter: Charles Lewis
> Assignee: Apache Spark
>
> When tasks end due to an ExceptionFailure, subscribers to onTaskEnd receive
> accumulators / task metrics for that task, if they were still available.
> These metrics are not currently sent when tasks are killed intentionally,
> such as when a speculative retry finishes, and the original is killed (or
> vice versa). Since we're killing these tasks ourselves, these metrics should
> almost always exist, and we should treat them the same way as we treat
> ExceptionFailures.
> Sending these metrics with the TaskKilled end reason makes aggregation across
> all tasks in an app more accurate. This data can inform decisions about how
> to tune the speculation parameters in order to minimize duplicated work, and
> in general, the total cost of an app should include both successful and
> failed tasks, if that information exists.
> PR: https://github.com/apache/spark/pull/17422
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
