GitHub user smurakozi opened a pull request:
https://github.com/apache/spark/pull/20330
[SPARK-23121][core] Fix for ui becoming unaccessible for long running
streaming apps
## What changes were proposed in this pull request?
The allJobs and the job pages attempt to use stage attempt and DAG
visualization from the store, but for long running jobs they are not guaranteed
to be retained, leading to exceptions when these pages are rendered.
To fix it `store.lastStageAttempt(stageId)` and
`store.operationGraphForJob(jobId)` are wrapped in `store.asOption` and default
values are used if the info is missing.
## How was this patch tested?
Manual testing of the UI, also using the test command reported in
SPARK-23121:
./bin/spark-submit --class
org.apache.spark.examples.streaming.HdfsWordCount
./examples/jars/spark-examples_2.11-2.4.0-SNAPSHOT.jar /spark
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/smurakozi/spark SPARK-23121
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20330.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20330
----
commit 94d50b42d6bf233afd398049c95386920c21c252
Author: Sandor Murakozi <smurakozi@...>
Date: 2018-01-19T10:59:36Z
Fixed issue caused by the store cleaning up old stages
commit d60ae4f39337b91118324064c6a3dc58a3fc2832
Author: Sandor Murakozi <smurakozi@...>
Date: 2018-01-19T11:33:27Z
JobPage doesn't break if operationGraphForJob is not in the store for a
jobid
commit 832378d25245126c285e794fadcaea019b70a78a
Author: Sandor Murakozi <smurakozi@...>
Date: 2018-01-19T11:34:59Z
lastStageNameAndDescription uses store.lastStageAttempt
commit 6525ef4eda0bf65bbbcb842495341afc8c5971ad
Author: Sandor Murakozi <smurakozi@...>
Date: 2018-01-19T12:15:33Z
Changed message in case of missing DAG visualization info
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
