GitHub user advancedxy opened a pull request:
https://github.com/apache/spark/pull/21165
Spark 20087: Attach accumulators / metrics to 'TaskKilled' end reason
## What changes were proposed in this pull request?
The ultimate goal is for listeners to onTaskEnd to receive metrics when a
task is killed intentionally, since the data is currently just thrown away.
This is already done for ExceptionFailure, so this just copies the same
approach.
## How was this patch tested?
Updated existing tests.
This is a rework of https://github.com/apache/spark/pull/17422, all credits
should go to @noodle-fb
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/advancedxy/spark SPARK-20087
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21165.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21165
----
commit 6262c52f7eb4abcf06742e9afd9d1454f06cdf1f
Author: Charles Lewis <noodle@...>
Date: 2017-03-22T20:33:55Z
report metrics for killed tasks
commit 18308895ad07c9c757e96e08f5c40b5dcaaf3455
Author: Charles Lewis <noodle@...>
Date: 2017-03-24T19:06:46Z
add task killed to exception accum test
commit cb276bc1f1583ca7dcf44b8e80a9fc2cd09953cf
Author: Charles Lewis <noodle@...>
Date: 2017-03-24T23:20:59Z
extra fixes for task killed reason merge
commit 30ae1457afd9899a5fa937c2fefac31f6a0752ed
Author: Xianjin YE <advancedxy@...>
Date: 2018-04-26T07:34:54Z
Fix merge conflict and semantic difference
commit 88b1cebd76d7414d4bbdb99e03fe10f74e25029a
Author: Xianjin YE <advancedxy@...>
Date: 2018-04-26T09:22:20Z
Make accums in TaskKilled immutable and extract
common logic in TaskRunner to reduce duplicate code
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
