[
https://issues.apache.org/jira/browse/HDDS-816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16688482#comment-16688482
]
Bharat Viswanadham commented on HDDS-816:
-----------------------------------------
Hi [~elek]
The reason why we have not gone with rocksdb estimate number of keys is, like
if we take this value during OM start, and DB compaction is not done, still we
see the wrong value. Suppose, if we have a background thread, which takes this
values for every configured interval, it causes an impression to admin that key
count is varying continously somehow, without any user action (like deletion,
creation), and it will cause a panic to admin. And also the estimates key count
can also be an issue with this approach because the calculation wibe
estimatedate when below are happening on rocksdb. (comments from code of
rocksdb)
(1) there exist merge keys
(2) keys are directly overwritten
(3) deletion on non-existing keys
So, on our OM rocksb 2 and 3 can happen, so the estimate will be mostly wrong.
Even if we think after compaction we get this value, but after some time
background thread gets the estimate count value again and we will be wrong. And
if we don't have background thread to update at the frequent interval the
behavior is unpredictable and it depends on rocksdb compaction timing.
So, I think missing the interval if OM is having unclean shutdown will be okay.
And atleast we shall be accurate in the happy scenario.
> Create OM metrics for bucket, volume, keys
> ------------------------------------------
>
> Key: HDDS-816
> URL: https://issues.apache.org/jira/browse/HDDS-816
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
> Reporter: Bharat Viswanadham
> Assignee: Bharat Viswanadham
> Priority: Major
> Attachments: HDDS-816.00.patch, HDDS-816.01.patch, HDDS-816.03.patch,
> Metrics for number of volumes, buckets, keys.pdf, Proposed Approach.pdf
>
>
> This Jira is used to create the following metrics in Ozone manager.
> # number of volumes
> # number of buckets
> # number of keys
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
