Jason Lowe commented on YARN-2765:

I agree that the timeline server seems like a worthy candidate for rocksdb.  
IIUC rocksdb's main use-case over leveldb is better performance when the 
database is larger than the node's RAM, which is likely in the case of the 
timeline server.

bq. And one other merit I've heard about rocksdb is that it can ride on HDFS.

This is news to me.  I knew rocksdb could be used as a cache of data that came 
from HDFS or could be backed-up to HDFS, but I didn't think it could read/write 
directly to it as part of normal operations.

bq. There's a rocksdb jni which seems to have windows support: 

Awesome, thanks for finding that.  I was looking at the standard org.rocksdb 
package.  Only concern with the fusesource option would be if it starts to 
diverge significantly from the standard one.  The API is already slightly 
different between the two, and the fusesource one hasn't been touched in a year 
while the org.rocksdb package was updated just last week.

Probably best to continue this conversation in a separate JIRA proposing we 
consider rocksdb for the timeline server.  If it works well there it should be 
very straightforward to provide store backends for the RM, NM, and JHS if it 
makes sense for them as well.

> Add leveldb-based implementation for RMStateStore
> -------------------------------------------------
>                 Key: YARN-2765
>                 URL: https://issues.apache.org/jira/browse/YARN-2765
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>         Attachments: YARN-2765.patch, YARN-2765v2.patch
> It would be nice to have a leveldb option to the resourcemanager recovery 
> store. Leveldb would provide some benefits over the existing filesystem store 
> such as better support for atomic operations, fewer I/O ops per state update, 
> and far fewer total files on the filesystem.

This message was sent by Atlassian JIRA

Reply via email to