[
https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15609109#comment-15609109
]
Marcelo Vanzin commented on SPARK-18085:
----------------------------------------
It's a little bit of everything. The two biggest issues we see are UI load
times and OOMs, and they feed into each other. We've seen single applications
that take 10s of minutes to load and use a huge amount of memory. Then you load
another application that causes the SHS to OOM and restart, and now you're back
to loading the original large application again. It's a vicious cycle.
> Scalability enhancements for the History Server
> -----------------------------------------------
>
> Key: SPARK-18085
> URL: https://issues.apache.org/jira/browse/SPARK-18085
> Project: Spark
> Issue Type: Umbrella
> Components: Spark Core, Web UI
> Affects Versions: 2.0.0
> Reporter: Marcelo Vanzin
> Attachments: spark_hs_next_gen.pdf
>
>
> It's a known fact that the History Server currently has some annoying issues
> when serving lots of applications, and when serving large applications.
> I'm filing this umbrella to track work related to addressing those issues.
> I'll be attaching a document shortly describing the issues and suggesting a
> path to how to solve them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
