GitHub user sandeep-katta opened a pull request:
https://github.com/apache/spark/pull/22571
[SPARK-25392][Spark Job History]Inconsistent behaviour for pool details in
spark web UI and history server page
## What changes were proposed in this pull request?
1. Added the pool information to the "Pool Information" field in the
environmentDetails by using the poolname as the key value, the xml string of
some other fields of the pool is Value.
This event is persist in event log dir
2.AppStatusListener::onEnvironmentUpdate will parse the xml String and
construct the Map[poolName,Pool] which will be stored in kvstore.
2, the page is divided into two types: one is AllStagesPage, and the other
is PoolPage,both will fetch the pool Details from kvstore and display
accordingly
In summary, modify the environmentDetails, save the pool information
through the SparkListenerEnvironmentUpdate event information; modify it
according to the modification scheme.
## How was this patch tested?
Added Testcases and also verified manually in the cluster
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sandeep-katta/spark historyserver
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22571.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22571
----
commit 65e3ca79c3ce8bcd16d9e92d7e9b5e69b532edef
Author: sandeep-katta <sandeep.katta2007@...>
Date: 2018-09-27T10:18:45Z
RootCause:History server does not maintain Pool Details as no event was
sent to maintain pool Details
Modification content:Pool Information is sent as a part of EnvironmentInfo
Event
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
