GitHub user noodle-fb opened a pull request:
https://github.com/apache/spark/pull/17422
Attach accumulators / metrics to 'TaskKilled' end reason
## What changes were proposed in this pull request?
The ultimate goal is for listeners to `onTaskEnd` to receive metrics when a
task is killed intentionally, since the data is currently just thrown away.
This is already done for ExceptionFailure, so this just copies the same
approach.
## How was this patch tested?
The unit test in DAGSchedulerSuite that tests this for ExceptionFailure was
modified to test the same thing for TaskKilled. I also re-tested all the unit
tests modified by the [last change to
TaskKilled](https://github.com/apache/spark/commit/8e558041aa0c41ba9fb2ce242daaf6d6ed4d85b7),
and made sure they all still pass.
For integration tests, I ran a query that caused a speculative task retry
on our deployment, and verified that the metrics showed up in our logging for
that retry when it was killed.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/noodle-fb/spark task-killed-metrics
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17422.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17422
----
commit 0625dc3187c9c8fa8d507adc0da75747c30f0ebc
Author: Charles Lewis <[email protected]>
Date: 2017-03-22T20:33:55Z
report metrics for killed tasks
commit ee883b2f3da10a4e4a48f4a98910ccadceac461c
Author: Charles Lewis <[email protected]>
Date: 2017-03-24T19:06:46Z
add task killed to exception accum test
commit 25ffbf49b2779d4fa795d754ee20fbe3542dd57d
Author: Charles Lewis <[email protected]>
Date: 2017-03-24T23:20:59Z
extra fixes for task killed reason merge
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
