GitHub user shahidki31 opened a pull request:
https://github.com/apache/spark/pull/22645
[SPARK-25566][SPARK-25567][WEBUI][SQL]Support pagination for SQL tab to
avoid OOM
## What changes were proposed in this pull request?
Currently SQL tab in the WEBUI doesn't have pagination. Because of that
following issues happening.
1) For large number of executions, SQL page is throwing OOM exception
(around 40,000)
2) For large number of executions, loading SQL page is taking time.
3) Difficult to analyse the execution table for large number of execution.
[Note: spark.ui.retainedExecutions = 50000]
All the tabs, Jobs, Stages etc. supports pagination. So, to make it
consistent with other tabs
SQL tab also should support pagination.
I have followed the similar flow of the pagination code in the Jobs and
Stages page for SQL page.
Also, this patch doesn't make any behavior change for the SQL tab except
the pagination support.
## How was this patch tested?
bin/spark-shell --conf spark.ui.retainedExecutions=50000
Run 50,000 sql queries.
**Before this PR**
**After this PR**
Loading of the page is faster, and OOM issue doesn't happen.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shahidki31/spark SPARK-25566
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22645.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22645
----
commit 4dc1b225b4de9f16c920bd5aaaa97e3597d023f3
Author: Shahid <shahidki31@...>
Date: 2018-10-04T21:16:09Z
SPARK-25566
[Spark Job History] SQL UI Page does not support Pagination
commit e2b45d51fbb00eab2e7b2e2e2fe35d45ca3f424c
Author: Shahid <shahidki31@...>
Date: 2018-10-05T16:55:19Z
[SPARK-25566]SQL UI Page support Pagination to avoid OOM
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
