GitHub user devaraj-kavali opened a pull request:
https://github.com/apache/spark/pull/16801
[SPARK-13619] [WEBUI] [CORE] Jobs page UI shows wrong number of failed tasks
## What changes were proposed in this pull request?
When the Failed/Killed Task End events come after the Job End event then
these task events are simply getting ignored without considering/updating into
JobUIData. It is happening because jobId information is getting deleted from
stageIdToActiveJobIds during the Job End event and Task End event is not able
to find the Job information to update.
## How was this patch tested?
### Current behaviour of Spark Jobs page for Running Application and
History page,
#### Completed Jobs (1)
| Job Id | Description | Submitted | Duration | Stages: Succeeded/Total |
Tasks (for all stages): Succeeded/Total |
| --- | --- | --- | --- | --- | --- |
| 0 | saveAsTextFile at JavaWordCountWithSlowTask.java:49 | 2017/01/25
09:03:14 | 1.4 min | 2/2 | 400/400 (17 killed) |
#### Completed Stages (2)
| Stage Id | Description | Submitted | Duration | Tasks: Succeeded/Total |
Input | Output | Shuffle Read | Shuffle Write |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | saveAsTextFile at JavaWordCountWithSlowTask.java:49 +details |
2017/01/25 09:04:34 | 5 s | 200/200 (2 failed) (1 killed) | | 6.8 KB | 2.3 MB
| |
| 0 | mapToPair at JavaWordCountWithSlowTask.java:33 +details | 2017/01/25
09:03:15 | 1.3 min | 200/200 (16 killed) | 1915.5 MB | | | 2.3 MB |
### Behaviour of the Web Pages after applying the patch,
#### Completed Jobs (1)
| Job Id | Description | Submitted | Duration | Stages: Succeeded/Total |
Tasks (for all stages): Succeeded/Total |
| --- | --- | --- | --- | --- | --- |
| 0 | saveAsTextFile at JavaWordCountWithSlowTask.java:49 | 2017/01/25
09:03:14 | 1.4 min | 2/2 | 400/400 (2 failed) (17 killed) |
#### Completed Stages (2)
| Stage Id | Description | Submitted | Duration | Tasks: Succeeded/Total |
Input | Output | Shuffle Read | Shuffle Write |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | saveAsTextFile at JavaWordCountWithSlowTask.java:49 +details |
2017/01/25 09:04:34 | 5 s | 200/200 (2 failed) (1 killed) | | 6.8 KB |
2.3 MB |
| 0 | mapToPair at JavaWordCountWithSlowTask.java:33 +details | 2017/01/25
09:03:15 | 1.3 min | 200/200 (16 killed) | 1915.5 MB | | | 2.3 MB |
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/devaraj-kavali/spark SPARK-13619
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16801.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16801
----
commit 3b51ef0e5ddd58e0bd8f90a52ca08145e5cdef4d
Author: Devaraj K <deva...@apache.org>
Date: 2017-02-04T01:40:35Z
[SPARK-13619] [WEBUI] [CORE] Jobs page UI shows wrong number of failed
tasks
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
