[
https://issues.apache.org/jira/browse/HDDS-1234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799260#comment-16799260
]
Aravindan Vijayan commented on HDDS-1234:
-----------------------------------------
{quote}
Yiqun Lin The count is actually not significant for the API. The idea was to
actually to store the count of keys to get an estimate of how many keys would
be read using the read-only iterator on the OM's RocksDB.
The main idea is to be able to use a prefix iterator on the (ContainerId,
keyPrefix) -> (Key count), for efficient lookup based on ContainerId.
We probably did miss this point in the design doc; but a Container can have
millions of keys and so a storage model like this with prefix iterator, we
think will work well. The value is effectively a placeholder. Aravindan Vijayan
Please add if I missed anything.
{quote}
Adding to this, we can support up to 1B keys in Ozone Manager & Recon's RockDB
using the current setup. 1B is a defensive estimate for the RocksDb's 8GB key
limit. When we move to an ozone cluster with more than 1B keys, we can change
Recon's containerDB storage to a (containerId, Key Prefix) -> Count model.
Currently, it is a degenerate case for that (prefix length = actual length).
Using a prefix seek on Recon container DB first and then on OM RocksDB we can
answer the question on what are the keys in a container (In other words, to
answer the question "If I lose a container, which keys' data have I lost?").
> Iterate the OM DB snapshot and populate the recon container DB.
> ----------------------------------------------------------------
>
> Key: HDDS-1234
> URL: https://issues.apache.org/jira/browse/HDDS-1234
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Recon
> Reporter: Aravindan Vijayan
> Assignee: Aravindan Vijayan
> Priority: Major
> Fix For: 0.5.0
>
> Attachments: HDDS-1234-000.patch, HDDS-1234-001.patch,
> HDDS-1234-002.patch, HDDS-1234-003.patch, HDDS-1234-004.patch
>
>
> * OM DB snapshot contains the Key->ContainerId + BlockId information.
> * Iterate the OM snapshot DB and create the reverse map of (ContainerId, Key
> prefix) -> Key count to be stored in the Recon container DB.
> * Use a codec to store data into Recon container DB.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
