[
https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188755#comment-14188755
]
Jason Lowe commented on YARN-2765:
----------------------------------
I agree that the timeline server seems like a worthy candidate for rocksdb.
IIUC rocksdb's main use-case over leveldb is better performance when the
database is larger than the node's RAM, which is likely in the case of the
timeline server.
bq. And one other merit I've heard about rocksdb is that it can ride on HDFS.
This is news to me. I knew rocksdb could be used as a cache of data that came
from HDFS or could be backed-up to HDFS, but I didn't think it could read/write
directly to it as part of normal operations.
bq. There's a rocksdb jni which seems to have windows support:
https://github.com/fusesource/rocksdbjni
Awesome, thanks for finding that. I was looking at the standard org.rocksdb
package. Only concern with the fusesource option would be if it starts to
diverge significantly from the standard one. The API is already slightly
different between the two, and the fusesource one hasn't been touched in a year
while the org.rocksdb package was updated just last week.
Probably best to continue this conversation in a separate JIRA proposing we
consider rocksdb for the timeline server. If it works well there it should be
very straightforward to provide store backends for the RM, NM, and JHS if it
makes sense for them as well.
> Add leveldb-based implementation for RMStateStore
> -------------------------------------------------
>
> Key: YARN-2765
> URL: https://issues.apache.org/jira/browse/YARN-2765
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Attachments: YARN-2765.patch, YARN-2765v2.patch
>
>
> It would be nice to have a leveldb option to the resourcemanager recovery
> store. Leveldb would provide some benefits over the existing filesystem store
> such as better support for atomic operations, fewer I/O ops per state update,
> and far fewer total files on the filesystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
