[
https://issues.apache.org/jira/browse/HDDS-4370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17253325#comment-17253325
]
Lokesh Jain commented on HDDS-4370:
-----------------------------------
[~arp] Currently for every deleted block there is a separate key in RocksDB. If
we consider file size of 1 MB then a 5 GB container would have 5000 blocks. If
we consider on average 2500 blocks deleted and 10000 containers with such
number then there are 25 million deleted blocks. For 100000 containers there
would be 250 million deleted blocks. As cluster ages the metadata could keep
increasing.
I understand the debugging issue. We probably need a mechanism to consolidate
these blocks in a single file, probably having a file in container where we
keep appending the blocks deleted rather than storing them in rocksDB?
FYI, with HDDS-4023, we also delete the closed containers and rocksdb once all
blocks for that container have been deleted. In such a case all the block
information will be lost.
> Datanode deletion service can avoid storing deleted blocks
> ----------------------------------------------------------
>
> Key: HDDS-4370
> URL: https://issues.apache.org/jira/browse/HDDS-4370
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Datanode
> Reporter: Lokesh Jain
> Assignee: Aryan Gupta
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.1.0
>
>
> Currently BlockDeletingService in datanode processes the blocks to be deleted
> and then stores them in a deleted blocks table. We can avoid storing deleted
> blocks in the container rocksDB.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
