GitHub user vanzin opened a pull request:
https://github.com/apache/spark/pull/18887
[SPARK-20642][core] Store FsHistoryProvider listing data in a KVStore.
The application listing is still generated from event logs, but is now
stored
in a KVStore instance. By default an in-memory store is used, but a new
config
allows setting a local disk path to store the data, in which case a LevelDB
store will be created.
The provider stores things internally using the public REST API types; I
believe
this is better going forward since it will make it easier to get rid of the
internal history server API which is mostly redundant at this point.
I also added a finalizer to LevelDBIterator, to make sure that resources are
eventually released. This helps when code iterates but does not exhaust the
iterator, thus not triggering the auto-close code.
HistoryServerSuite was modified to not re-start the history server
unnecessarily;
this makes the json validation tests run more quickly.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vanzin/spark SPARK-20642
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18887.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18887
----
commit 842589d34a2c2b97180d392d35eaa985edbfd671
Author: Marcelo Vanzin <[email protected]>
Date: 2016-10-13T22:30:59Z
SHS-NG M2: Store FsHistoryProvider listing data in a KVStore.
The application listing is still generated from event logs, but is now
stored
in a KVStore instance. By default an in-memory store is used, but a new
config
allows setting a local disk path to store the data, in which case a LevelDB
store will be created.
The provider stores things internally using the public REST API types; I
believe
this is better going forward since it will make it easier to get rid of the
internal history server API which is mostly redundant at this point.
I also added a finalizer to LevelDBIterator, to make sure that resources are
eventually released. This helps when code iterates but does not exhaust the
iterator, thus not triggering the auto-close code.
HistoryServerSuite was modified to not re-start the history server
unnecessarily;
this makes the json validation tests run more quickly.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
