[ https://issues.apache.org/jira/browse/SPARK-20087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451732#comment-16451732 ]
Xianjin YE commented on SPARK-20087: ------------------------------------ cc [~jiangxb1987] [~irashid], I am going to send a new pr if you still think this is the desired behaviour. > Include accumulators / taskMetrics when sending TaskKilled to onTaskEnd > listeners > --------------------------------------------------------------------------------- > > Key: SPARK-20087 > URL: https://issues.apache.org/jira/browse/SPARK-20087 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 2.1.0 > Reporter: Charles Lewis > Priority: Major > > When tasks end due to an ExceptionFailure, subscribers to onTaskEnd receive > accumulators / task metrics for that task, if they were still available. > These metrics are not currently sent when tasks are killed intentionally, > such as when a speculative retry finishes, and the original is killed (or > vice versa). Since we're killing these tasks ourselves, these metrics should > almost always exist, and we should treat them the same way as we treat > ExceptionFailures. > Sending these metrics with the TaskKilled end reason makes aggregation across > all tasks in an app more accurate. This data can inform decisions about how > to tune the speculation parameters in order to minimize duplicated work, and > in general, the total cost of an app should include both successful and > failed tasks, if that information exists. > PR: https://github.com/apache/spark/pull/17422 -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org