[
https://issues.apache.org/jira/browse/HDDS-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17136010#comment-17136010
]
Jitendra Nath Pandey edited comment on HDDS-3630 at 6/15/20, 4:52 PM:
----------------------------------------------------------------------
It is expected that majority of the containers on a datanode would be closed,
idle and not being accessed.
Following considerations are relevant:
* What are the in-memory and thread count characteristics for closed container
vs open containers? We might be able to optimize the closed containers, they
need only read cache/threads.
* We could attempt to shut down rocksdb instance or tune it down (if possible)
when container is not being accessed.
* Since closed containers are immutable, we could consider implementing a
combined cache for closed container rocksDBs, and not rely on rocksDB cache.
was (Author: jnp):
It is expected that majority of the containers on a datanode would be closed,
idle and not being accessed.
Following considerations are relevant:
* What are the in-memory and thread count characteristics for closed container
vs open containers? We might be able to optimize the closed containers, they
need only read cache/threads.
* We could attempt to shut down rocksdb instance or tune it down (if possible)
when container is not being accessed.
> Merge rocksdb in datanode
> -------------------------
>
> Key: HDDS-3630
> URL: https://issues.apache.org/jira/browse/HDDS-3630
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Reporter: runzhiwang
> Assignee: runzhiwang
> Priority: Major
> Attachments: Merge RocksDB in Datanode-v1.pdf, Merge RocksDB in
> Datanode-v2.pdf
>
>
> Currently, one rocksdb for one container. one container has 5GB capacity.
> 10TB data need more than 2000 rocksdb in one datanode. It's difficult to
> limit the memory of 2000 rocksdb. So maybe we should limited instance of
> rocksdb for each disk.
> The design of improvement is in the follow link, but still is a draft.
> TODO:
> 1. compatibility with current logic i.e. one rocksdb for each container
> 2. measure the memory usage before and after improvement
> 3. effect on efficiency of read and write.
> https://docs.google.com/document/d/18Ybg-NjyU602c-MYXaJHP6yrg-dVMZKGyoK5C_pp1mM/edit#
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
