Zhijie Shen commented on YARN-2765:

bq. This should work if the leveldb database is on a network store like a filer.

Thanks for sharing. This is an interesting use case that I'm not aware of 

bq. I briefly considered using rocksdb for this but decided against it for a 
couple of reasons:

It's not particularly related to this Jira, but I just want to think it out 
loudly. It seems that rocksdb claims to have better performance in terms of I/O 
than leveldb, while their APIs are very similar to each other. After we have 
the leveldb impl, it shouldn't be that difficult to make a rocksdb impl. 
Probably leveldb is enough to serve as the state store for RM/NM/JHS, but the 
timeline server may want a stronger one. Rocksdb may be a compromise before 
migrating to fully distributed storage solution based on HBase. And one other 
merit I've heard about rocksdb is that it can ride on HDFS. Correct me if I'm 
wrong, but it seems that rocksdb can also help to scale out the storage problem 
as well as support RM HA deployment in a shared nothing environment (e.g. 
without a network storage).

I'm not saying we should go with rocksdb now instead of leveldb, as we know it 
has been used for other components already. I'm trying to propose if we can 
think of rocksdb, which looks stronger but still reasonably simple alternate.

> Add leveldb-based implementation for RMStateStore
> -------------------------------------------------
>                 Key: YARN-2765
>                 URL: https://issues.apache.org/jira/browse/YARN-2765
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>         Attachments: YARN-2765.patch, YARN-2765v2.patch
> It would be nice to have a leveldb option to the resourcemanager recovery 
> store. Leveldb would provide some benefits over the existing filesystem store 
> such as better support for atomic operations, fewer I/O ops per state update, 
> and far fewer total files on the filesystem.

This message was sent by Atlassian JIRA

Reply via email to