[ 
https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16124731#comment-16124731
 ] 

Marcelo Vanzin commented on SPARK-18085:
----------------------------------------

[~duyanghao] that should all be explained in the document attached to this bug. 
I encourage you to read it if you're looking for details, or take a look at the 
work-in-progress code linked in many comments above. You're also welcome to run 
the code against your event logs and report any problems.

Note that no part of this work is about speeding up the loading of logs; 
loading an event log from scratch will most probably become slower now, when 
writing data to disk. The goal here is to control memory usage of the SHS, and 
to only have to process event logs once.

> SPIP: Better History Server scalability for many / large applications
> ---------------------------------------------------------------------
>
>                 Key: SPARK-18085
>                 URL: https://issues.apache.org/jira/browse/SPARK-18085
>             Project: Spark
>          Issue Type: Umbrella
>          Components: Spark Core, Web UI
>    Affects Versions: 2.0.0
>            Reporter: Marcelo Vanzin
>              Labels: SPIP
>         Attachments: spark_hs_next_gen.pdf
>
>
> It's a known fact that the History Server currently has some annoying issues 
> when serving lots of applications, and when serving large applications.
> I'm filing this umbrella to track work related to addressing those issues. 
> I'll be attaching a document shortly describing the issues and suggesting a 
> path to how to solve them.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to