[
https://issues.apache.org/jira/browse/HDDS-13841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18033445#comment-18033445
]
Devesh Kumar Singh commented on HDDS-13841:
-------------------------------------------
As per initial analysis, below is the cause of this issue:
The issue was in the
{code:java}
handlePutKeyEvent{code}
method in NSSummaryTaskDbEventHandler.java When a file is created, if its
parent directory's NSSummary doesn't exist in the in-memory map or database, the
code creates a new NSSummary with default values, including parentId = 0.
This breaks the propagation chain, preventing key counts from being propagated
to ancestor directories and the bucket.
This can occur when:
1. Directory creation events and file creation events are processed in
different batches
2. Directory NSSummaries haven't been persisted to the DB when file events
arrive
3. This can happen when directory creation events and file creation events
are processed in different batches. Could be because of recent optimizations
done at OM for batch processing.
When the propagation chain is broken (parentId = 0), the propagateSizeUpwards
method stops propagating counts upward, resulting in incorrect total key counts
at the bucket level.
> Namespace summary API gives wrong count of directories and keys
> ---------------------------------------------------------------
>
> Key: HDDS-13841
> URL: https://issues.apache.org/jira/browse/HDDS-13841
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Recon
> Reporter: Abhishek Pal
> Assignee: Arafat Khan
> Priority: Major
>
> The backend API:
> {code:java}
> api/v1/namespace/summary?path=<path to location> {code}
> gives us wrong metadata details for the number of directories and keys.
> Here is a sample command output for key list and bucket info:
> {code:java}
> ozone sh bucket info volume1/fso-bucket
> {
> "metadata" : { },
> "volumeName" : "volume1",
> "name" : "fso-bucket",
> "storageType" : "DISK",
> "versioning" : false,
> "listCacheSize" : 1000,
> "usedBytes" : 188743680,
> "usedNamespace" : 9,
> "creationTime" : "2025-10-23T14:26:43.493Z",
> "modificationTime" : "2025-10-23T14:26:43.493Z",
> "sourcePathExist" : true,
> "quotaInBytes" : -1,
> "quotaInNamespace" : -1,
> "bucketLayout" : "FILE_SYSTEM_OPTIMIZED",
> "owner" : "om",
> "link" : false
> }{code}
>
> {code:java}
> ozone sh key list volume1/fso-bucket
> [ {
> "volumeName" : "volume1",
> "bucketName" : "fso-bucket",
> "name" : "dir1/",
> "owner" : "",
> "dataSize" : 0,
> "creationTime" : "2025-10-23T14:26:43.521Z",
> "modificationTime" : "2025-10-23T14:27:31.830Z",
> "replicationConfig" : {
> "replicationFactor" : "THREE",
> "requiredNodes" : 3,
> "minimumNodes" : 1,
> "replicationType" : "RATIS"
> },
> "metadata" : { },
> "tags" : { },
> "file" : false
> }, {
> "volumeName" : "volume1",
> "bucketName" : "fso-bucket",
> "name" : "dir1/dir2/",
> "owner" : "",
> "dataSize" : 0,
> "creationTime" : "2025-10-23T14:26:43.521Z",
> "modificationTime" : "2025-10-23T14:27:41.078Z",
> "replicationConfig" : {
> "replicationFactor" : "THREE",
> "requiredNodes" : 3,
> "minimumNodes" : 1,
> "replicationType" : "RATIS"
> },
> "metadata" : { },
> "tags" : { },
> "file" : false
> }, {
> "volumeName" : "volume1",
> "bucketName" : "fso-bucket",
> "name" : "dir1/dir2/dir3/",
> "owner" : "",
> "dataSize" : 0,
> "creationTime" : "2025-10-23T14:26:43.521Z",
> "modificationTime" : "2025-10-23T14:27:56.584Z",
> "replicationConfig" : {
> "replicationFactor" : "THREE",
> "requiredNodes" : 3,
> "minimumNodes" : 1,
> "replicationType" : "RATIS"
> },
> "metadata" : { },
> "tags" : { },
> "file" : false
> }, {
> "volumeName" : "volume1",
> "bucketName" : "fso-bucket",
> "name" : "dir1/dir2/dir3/file1",
> "owner" : "om",
> "dataSize" : 10485760,
> "creationTime" : "2025-10-23T14:27:54.747Z",
> "modificationTime" : "2025-10-23T14:27:56.534Z",
> "replicationConfig" : {
> "replicationFactor" : "THREE",
> "requiredNodes" : 3,
> "minimumNodes" : 1,
> "replicationType" : "RATIS"
> },
> "metadata" : { },
> "tags" : { },
> "file" : true
> }, {
> "volumeName" : "volume1",
> "bucketName" : "fso-bucket",
> "name" : "dir1/dir2/dir3/testfile",
> "owner" : "om",
> "dataSize" : 10485760,
> "creationTime" : "2025-10-23T14:27:43.942Z",
> "modificationTime" : "2025-10-23T14:27:45.590Z",
> "replicationConfig" : {
> "replicationFactor" : "THREE",
> "requiredNodes" : 3,
> "minimumNodes" : 1,
> "replicationType" : "RATIS"
> },
> "metadata" : { },
> "tags" : { },
> "file" : true
> }, {
> "volumeName" : "volume1",
> "bucketName" : "fso-bucket",
> "name" : "dir1/dir2/file1",
> "owner" : "om",
> "dataSize" : 10485760,
> "creationTime" : "2025-10-23T14:27:39.443Z",
> "modificationTime" : "2025-10-23T14:27:41.034Z",
> "replicationConfig" : {
> "replicationFactor" : "THREE",
> "requiredNodes" : 3,
> "minimumNodes" : 1,
> "replicationType" : "RATIS"
> },
> "metadata" : { },
> "tags" : { },
> "file" : true
> }, {
> "volumeName" : "volume1",
> "bucketName" : "fso-bucket",
> "name" : "dir1/dir2/testfile",
> "owner" : "om",
> "dataSize" : 10485760,
> "creationTime" : "2025-10-23T14:27:34.768Z",
> "modificationTime" : "2025-10-23T14:27:36.524Z",
> "replicationConfig" : {
> "replicationFactor" : "THREE",
> "requiredNodes" : 3,
> "minimumNodes" : 1,
> "replicationType" : "RATIS"
> },
> "metadata" : { },
> "tags" : { },
> "file" : true
> }, {
> "volumeName" : "volume1",
> "bucketName" : "fso-bucket",
> "name" : "dir1/file1",
> "owner" : "om",
> "dataSize" : 10485760,
> "creationTime" : "2025-10-23T14:27:30.153Z",
> "modificationTime" : "2025-10-23T14:27:31.789Z",
> "replicationConfig" : {
> "replicationFactor" : "THREE",
> "requiredNodes" : 3,
> "minimumNodes" : 1,
> "replicationType" : "RATIS"
> },
> "metadata" : { },
> "tags" : { },
> "file" : true
> }, {
> "volumeName" : "volume1",
> "bucketName" : "fso-bucket",
> "name" : "dir1/testfile",
> "owner" : "om",
> "dataSize" : 10485760,
> "creationTime" : "2025-10-23T14:27:23.159Z",
> "modificationTime" : "2025-10-23T14:27:24.946Z",
> "replicationConfig" : {
> "replicationFactor" : "THREE",
> "requiredNodes" : 3,
> "minimumNodes" : 1,
> "replicationType" : "RATIS"
> },
> "metadata" : { },
> "tags" : { },
> "file" : true
> } ] {code}
> However the API returns 3 keys and 3 directories only.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
