[GitHub] spark pull request #22689: [SPARK-25697][CORE]When zstd compression enabled ...

[shahidki31]

GitHub user shahidki31 opened a pull request:

    https://github.com/apache/spark/pull/22689

    [SPARK-25697][CORE]When zstd compression enabled in progress application is 
throwing Error is throwing in the history webui…

    ## What changes were proposed in this pull request?
    When we enable event log compression and compression codec as 'zstd', we 
are unable to open the webui of the running application from the history server 
page. 
    The reason is that, Replay listener was unable to read from the zstd 
compressed eventlog due to the zstd frame was not finished yet. This causes 
truncated error while reading the eventLog. 
    
    So, when we try to open the WebUI from the History server page, it throws 
"truncated error ", and we never able to open running application in the webui, 
when we enable zstd compression.
    
    In this PR, when the IO excpetion happens, and if it is a running 
application, we  log the error,
    "Failed to read Spark event log: evetLogDirAppName.inprogress", instead of 
throwing exception.
    
    ## How was this patch tested?
    Test steps:
    1)spark.eventLog.compress =  true
    2)spark.io.compression.codec = zstd 
    3)restart history server
    4) launch bin/spark-shell
    5) run some queries
    6) Open history server page
    7) click on the application
    Before fix:
    ![screenshot from 2018-10-10 
23-52-12](https://user-images.githubusercontent.com/23054875/46757387-9b4fa580-cce7-11e8-96ad-8938400483ed.png)
    
    ![screenshot from 2018-10-10 
23-52-28](https://user-images.githubusercontent.com/23054875/46757393-a0145980-cce7-11e8-8cb0-44b583dde648.png)
    
    After fix:
    
    ![screenshot from 2018-10-10 
23-43-49](https://user-images.githubusercontent.com/23054875/46756971-6858e200-cce6-11e8-946c-0bffebb2cfba.png)
    
    ![screenshot from 2018-10-10 
23-44-05](https://user-images.githubusercontent.com/23054875/46756981-6d1d9600-cce6-11e8-95ea-ff8339a2fdfd.png)
    
       
    
    (Please explain how this patch was tested. E.g. unit tests, integration 
tests, manual tests)
    (If this patch involves UI changes, please attach a screenshot; otherwise, 
remove this)
    
    Please review http://spark.apache.org/contributing.html before opening a 
pull request.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/shahidki31/spark SPARK-25697

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/22689.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #22689
    
----
commit f6e971f37d39642966689201868420da2441ffa0
Author: Shahid <shahidki31@...>
Date:   2018-10-10T17:50:48Z

    When zstd compression enabled in progress application is throwing Error in 
UI

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org