Jason Lowe updated YARN-2765:
    Attachment: YARN-2765v2.patch

Thanks for the review, Tsuyoshi!

bq. How about adding helper methods like getKeyPrefix/getNodePath for getting 
key prefix and node path?

Sure, added some helper methods to compute leveldb keys for various things.

bq. I found that the patch includes lots hard-coded "/". I think it's better to 
have private field SEPARATOR = "/". 

IMHO this makes the code less readable, similar to a code style like {{final 
int ONE = 1}}.  But I don't care too strongly about it and changed all 
occurrences to SEPARATOR.

For Zhijie's comments:

bq. One drawback I can think of is that while LeveldbRMStateStore is 
lightweight for single RM restarting, multiple RMs of HA are not able to share 
this single-host database.

This should work if the leveldb database is on a network store like a filer.  
Leveldb uses locks to prevent multiple processes from trying to access the 
database simultaneously, so there's a little bit of help for the fencing 
scenarios.  However the fencing script actions would have to do some extra work 
to force a poorly-behaving resourcemanager to let go of the locks so a standby 
RM can open the store and become active.

bq. Did you have a chance to think of an enhanced k/v db: rocksdb?

I briefly considered using rocksdb for this but decided against it for a couple 
of reasons:

* leveldb is already used by the timeline server and nodemanager, and I would 
rather avoid adding yet another new dependency for this
* leveldb supports win32/win64, but it doesn't appear that the standard 
rocksdbjni distribution has support for Windows.

> Add leveldb-based implementation for RMStateStore
> -------------------------------------------------
>                 Key: YARN-2765
>                 URL: https://issues.apache.org/jira/browse/YARN-2765
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>         Attachments: YARN-2765.patch, YARN-2765v2.patch
> It would be nice to have a leveldb option to the resourcemanager recovery 
> store. Leveldb would provide some benefits over the existing filesystem store 
> such as better support for atomic operations, fewer I/O ops per state update, 
> and far fewer total files on the filesystem.

This message was sent by Atlassian JIRA

Reply via email to