[
https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188595#comment-14188595
]
Zhijie Shen edited comment on YARN-2765 at 10/29/14 5:22 PM:
-------------------------------------------------------------
bq. This should work if the leveldb database is on a network store like a filer.
Thanks for sharing. This is an interesting use case that I'm not aware of
before.
bq. I briefly considered using rocksdb for this but decided against it for a
couple of reasons:
It's not particularly related to this Jira, but I just want to think it out
loudly. It seems that rocksdb claims to have better performance in terms of I/O
than leveldb, while their APIs are very similar to each other. After we have
the leveldb impl, it shouldn't be that difficult to make a rocksdb impl.
Probably leveldb is enough to serve as the state store for RM/NM/JHS, but the
timeline server may want a stronger one. Rocksdb may be a compromise before
migrating to fully distributed storage solution based on HBase. And one other
merit I've heard about rocksdb is that it can ride on HDFS. Correct me if I'm
wrong, but it seems that rocksdb can also help to scale out the storage problem
as well as support RM HA deployment in a shared nothing environment (e.g.
without a network storage).
I'm not saying we should go with rocksdb now instead of leveldb, as we know it
has been used for other components already. I'm trying to propose if we can
think of rocksdb, which looks stronger but still reasonably simple alternate.
There's a rocksdb jni which seems to have windows support:
https://github.com/fusesource/rocksdbjni
It should be the same org whose leveldbjni is currently used by us.
was (Author: zjshen):
bq. This should work if the leveldb database is on a network store like a filer.
Thanks for sharing. This is an interesting use case that I'm not aware of
before.
bq. I briefly considered using rocksdb for this but decided against it for a
couple of reasons:
It's not particularly related to this Jira, but I just want to think it out
loudly. It seems that rocksdb claims to have better performance in terms of I/O
than leveldb, while their APIs are very similar to each other. After we have
the leveldb impl, it shouldn't be that difficult to make a rocksdb impl.
Probably leveldb is enough to serve as the state store for RM/NM/JHS, but the
timeline server may want a stronger one. Rocksdb may be a compromise before
migrating to fully distributed storage solution based on HBase. And one other
merit I've heard about rocksdb is that it can ride on HDFS. Correct me if I'm
wrong, but it seems that rocksdb can also help to scale out the storage problem
as well as support RM HA deployment in a shared nothing environment (e.g.
without a network storage).
I'm not saying we should go with rocksdb now instead of leveldb, as we know it
has been used for other components already. I'm trying to propose if we can
think of rocksdb, which looks stronger but still reasonably simple alternate.
> Add leveldb-based implementation for RMStateStore
> -------------------------------------------------
>
> Key: YARN-2765
> URL: https://issues.apache.org/jira/browse/YARN-2765
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Attachments: YARN-2765.patch, YARN-2765v2.patch
>
>
> It would be nice to have a leveldb option to the resourcemanager recovery
> store. Leveldb would provide some benefits over the existing filesystem store
> such as better support for atomic operations, fewer I/O ops per state update,
> and far fewer total files on the filesystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
