Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/8744#issuecomment-151935694
> BTW, do we need to take special care with #9297 when integrated with ATS?
1. an older history server will fail when it gets a newer event. The
current code attaches the `spark.SPARK_VERSION` field to histories; some
pattern magic logic could perhaps fail fast if there's a different version
(requirement: no 1.6.x patch adds a new event). If added across providers, even
the UI could detect version mismatch and refuse to load newer histories.
1. Memory consumption on ATS: more stuff. Maybe it could be configurable
about which events to persist?
1. Memory consumption on history server from expanded UIs. Adding metrics
for the spark history server will help identify those issues, but not address
it. This is independent of how history is persisted.
1. Note that you could do event filtering on playback too, if you had full
histories but wanted to strip out some events.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
