[GitHub] spark pull request #22645: [SPARK-25566][SPARK-25567][WEBUI][SQL]Support pag...

[shahidki31]

GitHub user shahidki31 opened a pull request:

    https://github.com/apache/spark/pull/22645

    [SPARK-25566][SPARK-25567][WEBUI][SQL]Support pagination for SQL tab to 
avoid OOM

    ## What changes were proposed in this pull request?
    Currently SQL tab in the WEBUI doesn't have pagination. Because of that 
following issues happening.
    1) For large number of executions, SQL page is throwing OOM exception 
(around 40,000)
    2) For large number of executions, loading SQL page is taking time.
    3) Difficult to analyse the execution table for large number of execution.
    [Note: spark.ui.retainedExecutions = 50000]
    
    All the tabs, Jobs, Stages etc. supports pagination. So, to make it 
consistent with other tabs
    SQL tab also should support pagination.
    
    I have followed the similar flow of the pagination code in the Jobs and 
Stages page for SQL page.
    Also, this patch doesn't make any behavior change for the SQL tab except 
the pagination support.
    
    ## How was this patch tested?
    bin/spark-shell --conf spark.ui.retainedExecutions=50000
    Run 50,000 sql queries.
    **Before this PR**
    ![screenshot from 2018-10-05 
22-58-11](https://user-images.githubusercontent.com/23054875/46550276-33b5e680-c8f2-11e8-9e32-9ae9c5b181e0.png)
    
    **After this PR**
    
    Loading of the page is faster, and OOM issue doesn't happen.
    ![screenshot from 2018-10-05 
23-14-29](https://user-images.githubusercontent.com/23054875/46551092-71b40a00-c8f4-11e8-93dd-ff5b99d44c54.png)
    
    
    
    
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/shahidki31/spark SPARK-25566

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/22645.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #22645
    
----
commit 4dc1b225b4de9f16c920bd5aaaa97e3597d023f3
Author: Shahid <shahidki31@...>
Date:   2018-10-04T21:16:09Z

        SPARK-25566
    
    [Spark Job History] SQL UI Page does not support Pagination

commit e2b45d51fbb00eab2e7b2e2e2fe35d45ca3f424c
Author: Shahid <shahidki31@...>
Date:   2018-10-05T16:55:19Z

    [SPARK-25566]SQL UI Page support Pagination to avoid OOM

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org