[
https://issues.apache.org/jira/browse/HDDS-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17118639#comment-17118639
]
Istvan Fajth commented on HDDS-3630:
------------------------------------
One important thing that came out from HDDS-3267 is that the ContainerCache
implementation is used to cache RocksDB connections inside the DataNode. If we
merge the RocksDB instances, we most probably should drop the ContainerCache
and the underlying RocksDB connection caching/pooling implemented there as it
will not be needed anymore.
NOTE
If for some reason we do not drop the implementation, then please file a follow
up JIRA to fix the following flaw in that code before closing this one:
The ContainerCache code uses this.put() to put the value to the LRUMap, but in
LRUMap put is not overridden, it provides addMapping() which checks properly
for boundaries, and since the ContainerCache code does not check for
boundaries, basically in the current implementation we have an unlimited cache
which lies about itself and says it is limited by maxSize.
> Merge rocksdb in datanode
> -------------------------
>
> Key: HDDS-3630
> URL: https://issues.apache.org/jira/browse/HDDS-3630
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Reporter: runzhiwang
> Assignee: runzhiwang
> Priority: Major
> Attachments: Merge RocksDB in Datanode-v1.pdf
>
>
> Currently, one rocksdb for one container. one container has 5GB capacity.
> 10TB data need more than 2000 rocksdb in one datanode. It's difficult to
> limit the memory of 2000 rocksdb. So maybe we should limited instance of
> rocksdb for each disk.
> The design of improvement is in the follow link, but still is a draft.
> TODO:
> 1. compatibility with current logic i.e. one rocksdb for each container
> 2. measure the memory usage before and after improvement
> 3. effect on efficiency of read and write.
> https://docs.google.com/document/d/18Ybg-NjyU602c-MYXaJHP6yrg-dVMZKGyoK5C_pp1mM/edit#
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
