Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/204#issuecomment-39919830
Also @tgraves, I have not tried to render thousands of apps, but from what
I've experienced on the SparkUI's stage page, which may display thousands of
tasks, loading the HTML becomes much slower. In other words, as you suspected,
the UI doesn't scale super well. I think the correct thing to do here is to add
pagination (which may or may not go out with this PR).
We can set the retained applications higher. I started it at 20 just
because I was testing locally. The thing is each ApplicationHistoryInfo may
very widely across applications, because each of them contains a SparkUI, which
holds various data structures whose sizes scale with the number of tasks. For
now, however, we will rotate on the granularity of applications, because this
is much simpler.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
