GitHub user vanzin opened a pull request:
https://github.com/apache/spark/pull/19751
[SPARK-20653][core] Add cleaning of old elements from the status store.
This change restores the functionality that keeps a limited number of
different types (jobs, stages, etc) depending on configuration, to avoid
the store growing indefinitely over time.
The feature is implemented by creating a new type (ElementTrackingStore)
that wraps a KVStore and allows triggers to be set up for when elements
of a certain type meet a certain threshold. Triggers don't need to
necessarily only delete elements, but the current API is set up in a way
that makes that use case easier.
The new store also has a trigger for the "close" call, which makes it
easier for listeners to register code for cleaning things up and flushing
partial state to the store.
The old configurations for cleaning up the stored elements from the core
and SQL UIs are now active again, and the old unit tests are re-enabled.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vanzin/spark SPARK-20653
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19751.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19751
----
commit 8c346a148d7be78b0f53aadb9c8ca78098b0ea6c
Author: Marcelo Vanzin <[email protected]>
Date: 2017-04-18T20:38:10Z
[SPARK-20653][core] Add cleaning of old elements from the status store.
This change restores the functionality that keeps a limited number of
different types (jobs, stages, etc) depending on configuration, to avoid
the store growing indefinitely over time.
The feature is implemented by creating a new type (ElementTrackingStore)
that wraps a KVStore and allows triggers to be set up for when elements
of a certain type meet a certain threshold. Triggers don't need to
necessarily only delete elements, but the current API is set up in a way
that makes that use case easier.
The new store also has a trigger for the "close" call, which makes it
easier for listeners to register code for cleaning things up and flushing
partial state to the store.
The old configurations for cleaning up the stored elements from the core
and SQL UIs are now active again, and the old unit tests are re-enabled.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
