[ https://issues.apache.org/jira/browse/SPARK-31608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jungtaek Lim resolved SPARK-31608. ---------------------------------- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 28412 [https://github.com/apache/spark/pull/28412] > Add a hybrid KVStore to make UI loading faster > ---------------------------------------------- > > Key: SPARK-31608 > URL: https://issues.apache.org/jira/browse/SPARK-31608 > Project: Spark > Issue Type: Story > Components: Web UI > Affects Versions: 3.0.1 > Reporter: Baohe Zhang > Assignee: Baohe Zhang > Priority: Major > Fix For: 3.1.0 > > > This is a follow-up for the work done by Hieu Huynh in 2019. > Add a new class HybridKVStore to make the history server faster when loading > event files. When rebuilding the application state from event logs, > HybridKVStore will first write data to an in-memory store and having a > background thread that keeps pushing the change to levelDB. > I ran some tests on 3.0.1 on mac os: > ||kvstore type / log size||100m||200m||500m||1g||2g|| > |HybridKVStore|5s to parse, 7s(include the parsing time) to switch to > leveldb|6s to parse, 10s to switch to leveldb|15s to parse, 23s to switch to > leveldb|23s to parse, 40s to switch to leveldb|37s to parse, 73s to switch to > leveldb| > |LevelDB|12s to parse|19s to parse|43s to parse|69s to parse|124s to parse| > For example when loading a 1g file, HybridKVStore takes 23s to parse (that > means, users only need to wait for 23s to see the UI), the background thread > will still run 17s to copy data to leveldb. And after that, the in memory > store can be closed, the entire store now moves to leveldb. So in general, it > has 3x - 4x UI loading speed improvement. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org