GitHub user attilapiros opened a pull request:
https://github.com/apache/spark/pull/20408
[SPARK-23189][Core][UI] Reflect stage level blacklisting on executor tab
## What changes were proposed in this pull request?
The purpose of this PR to reflect the stage level blacklisting on the
executor tab for the currently active stages.
After this change in the executor tab at the Status column one of the
following label will be:
- "Blacklisted" when the executor is blacklisted application level (old
flag)
- "Dead" when the executor is not Blacklisted and not Active
- "Blacklisted in Stages: [...]" when the executor is Active but the
blacklistedInStages set is not empty where within the [] coma separated active
stageIDs are listed.
- "Active" when the executor is Active and blacklistedInStages set is empty
## How was this patch tested?
Both with unit tests and manually.
#### Manual test
Spark was started as:
```bash
bin/spark-shell --master "local-cluster[2,1,1024]" --conf
"spark.blacklist.enabled=true" --conf
"spark.blacklist.stage.maxFailedTasksPerExecutor=1" --conf
"spark.blacklist.application.maxFailedTasksPerExecutor=10"
```
And the job was:
```scala
import org.apache.spark.SparkEnv
val pairs = sc.parallelize(1 to 10000, 10).map { x =>
if (SparkEnv.get.executorId.toInt == 0) throw new RuntimeException("Bad
executor")
else {
Thread.sleep(10)
(x % 10, x)
}
}
val all = pairs.cogroup(pairs)
all.collect()
```
UI screenshots about the running:
- One executor is blacklisted in the two stages:
- One stage completes the other one is still running:
- Both stages are completed:
### Unit tests
In AppStatusListenerSuite.scala both node blacklisting for stage and
executor blacklisting for stage are tested.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/attilapiros/spark SPARK-23189
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20408.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20408
----
commit 4f5176be0d3da7794d20895d8a0bfce16d6b8e5c
Author: âattilapirosâ <piros.attila.zsolt@...>
Date: 2018-01-25T22:17:11Z
inital version
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
