GitHub user vanzin opened a pull request:

    https://github.com/apache/spark/pull/18887

    [SPARK-20642][core] Store FsHistoryProvider listing data in a KVStore.

    The application listing is still generated from event logs, but is now 
stored
    in a KVStore instance. By default an in-memory store is used, but a new 
config
    allows setting a local disk path to store the data, in which case a LevelDB
    store will be created.
    
    The provider stores things internally using the public REST API types; I 
believe
    this is better going forward since it will make it easier to get rid of the
    internal history server API which is mostly redundant at this point.
    
    I also added a finalizer to LevelDBIterator, to make sure that resources are
    eventually released. This helps when code iterates but does not exhaust the
    iterator, thus not triggering the auto-close code.
    
    HistoryServerSuite was modified to not re-start the history server 
unnecessarily;
    this makes the json validation tests run more quickly.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/vanzin/spark SPARK-20642

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/18887.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #18887
    
----
commit 842589d34a2c2b97180d392d35eaa985edbfd671
Author: Marcelo Vanzin <van...@cloudera.com>
Date:   2016-10-13T22:30:59Z

    SHS-NG M2: Store FsHistoryProvider listing data in a KVStore.
    
    The application listing is still generated from event logs, but is now 
stored
    in a KVStore instance. By default an in-memory store is used, but a new 
config
    allows setting a local disk path to store the data, in which case a LevelDB
    store will be created.
    
    The provider stores things internally using the public REST API types; I 
believe
    this is better going forward since it will make it easier to get rid of the
    internal history server API which is mostly redundant at this point.
    
    I also added a finalizer to LevelDBIterator, to make sure that resources are
    eventually released. This helps when code iterates but does not exhaust the
    iterator, thus not triggering the auto-close code.
    
    HistoryServerSuite was modified to not re-start the history server 
unnecessarily;
    this makes the json validation tests run more quickly.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to