[
https://issues.apache.org/jira/browse/HDDS-13841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18033795#comment-18033795
]
Arafat Khan edited comment on HDDS-13841 at 10/29/25 11:08 AM:
---------------------------------------------------------------
h2. How We Uncovered This Issue
We created the following directory tree:
{code:java}
volume1/fso-bucket/
└── dir1/
├── file1
├── file2
└── dir2/
├── file1
├── file2
└── dir3/
├── file1
└── file2 {code}
h2. Commands Executed
Using {{ozone fs}}
{code:java}
fallocate -l 10M testfile
ozone fs -mkdir -p ofs://om/volume2/fso-bucket1/dir1/dir2/dir3
ozone fs -put -f testfile ofs://om/volume2/fso-bucket1/dir1/file1
ozone fs -put -f testfile ofs://om/volume2/fso-bucket1/dir1/file2
ozone fs -put -f testfile ofs://om/volume2/fso-bucket1/dir1/dir2/file3
ozone fs -put -f testfile ofs://om/volume2/fso-bucket1/dir1/dir2/file4
ozone fs -put -f testfile ofs://om/volume2/fso-bucket1/dir1/dir2/dir3/file5
ozone fs -put -f testfile ofs://om/volume2/fso-bucket1/dir1/dir2/dir3/file6
{code}
h3. Using {{ozone sh}}
{code:java}
ozone sh volume create /volume2
ozone sh bucket create --layout=FILE_SYSTEM_OPTIMIZED /volume2/fso-bucket2
fallocate -l 10M /tmp/samplefileozone sh key put
/volume2/fso-bucket2/dir1/testfile1 /tmp/samplefile
ozone sh key put /volume2/fso-bucket2/dir1/testfile2 /tmp/samplefile
ozone sh key put /volume2/fso-bucket2/dir1/dir2/testfile1 /tmp/samplefile
ozone sh key put /volume2/fso-bucket2/dir1/dir2/testfile2 /tmp/samplefile
ozone sh key put /volume2/fso-bucket2/dir1/dir2/dir3/testfile1 /tmp/samplefile
ozone sh key put /volume2/fso-bucket2/dir1/dir2/dir3/testfile2 /tmp/samplefile
{code}
h3. Observations
We noticed that namespace summaries for volumes and buckets created using the
{{ozone fs}} command showed inconsistencies.
In contrast, those created using the {{ozone sh}} command behaved correctly.
Both sets of operations generate events that are applied to the namespace
summary tree in Recon. The difference lies in how these events are interpreted.
h2. Event Comparison
h3. Using {{ozone fs -}}
{code:java}
Filtered (only fileTable and directoryTable events, in order):
#8 → directoryTable → PUT → /.../dir1
#9 → directoryTable → PUT → /.../dir2
#10 → directoryTable → PUT → /.../dir3
...
#38 → directoryTable → UPDATE → /.../dir1
#67 → directoryTable → UPDATE → /.../dir1
#96 → directoryTable → UPDATE → /.../dir2
#125 → directoryTable → UPDATE → /.../dir2
#154 → directoryTable → UPDATE → /.../dir3
#183 → directoryTable → UPDATE → /.../dir3 {code}
h3. Using {{ozone sh}}
{code:java}
Filtered (only fileTable and directoryTable events, in order):
#5 → directoryTable → PUT → /.../dir1
#11 → fileTable → PUT → /.../testfile1
#18 → fileTable → PUT → /.../testfile2
#20 → directoryTable → PUT → /.../dir2
#26 → fileTable → PUT → /.../testfile1
#33 → fileTable → PUT → /.../testfile2
#35 → directoryTable → PUT → /.../dir3
#41 → fileTable → PUT → /.../testfile1
#48 → fileTable → PUT → /.../testfile2 {code}
As shown, {{ozone sh}} produces only {{PUT}} events, whereas {{ozone fs}}
produces additional {{UPDATE}} and some {{DELETE}} events.
The {{DELETE}} events are harmless, but the {{UPDATE}} events from {{ozone fs}}
are the root cause of the issue.
h3. Root Cause
When Recon processes a directory {{UPDATE}} event, it deletes the old directory
record and replaces it with the updated version.
This triggers both {{deleteDir}} and {{putDirectory}} operations.
The problem arises because during {{{}putDirectory{}}}, the propagation logic
passes an incorrect object ID when the directory already contains files or
subdirectories.
This causes incorrect size and count propagation in the namespace summary.
The bug did not surface earlier because our tests used empty directories, where
size propagation was trivial and unaffected.
h3. Conclusion
This issue highlights a *testing gap* in our Recon integration tests.
We plan to create a parent JIRA to address this by improving test coverage for:
* Namespace summary propagation
* Other Recon endpoints and tasks
This will ensure more validation across all event scenarios.
was (Author: JIRAUSER284839):
h3. How We Uncovered This Issue
We created the following directory tree:
{{}}
{code:java}
{code}
{{volume1/fso-bucket/
└── dir1/
├── file1
├── file2
└── dir2/
├── file1
├── file2
└── dir3/
├── file1
└── file2 }}
{code:java}
{code}
{{We built this tree using both the }}{{ozone sh}}{{{color:#172b4d} and
{color}}}{{ozone fs}}{{{color:#172b4d} commands.{color}}}
h3. Commands Executed
h3. Using {{ozone fs}}
{code:java}
fallocate -l 10M testfile
ozone fs -mkdir -p ofs://om/volume2/fso-bucket1/dir1/dir2/dir3
ozone fs -put -f testfile ofs://om/volume2/fso-bucket1/dir1/file1
ozone fs -put -f testfile ofs://om/volume2/fso-bucket1/dir1/file2
ozone fs -put -f testfile ofs://om/volume2/fso-bucket1/dir1/dir2/file3
ozone fs -put -f testfile ofs://om/volume2/fso-bucket1/dir1/dir2/file4
ozone fs -put -f testfile ofs://om/volume2/fso-bucket1/dir1/dir2/dir3/file5
ozone fs -put -f testfile ofs://om/volume2/fso-bucket1/dir1/dir2/dir3/file6
{code}
h3. Using {{ozone sh}}
{code:java}
ozone sh volume create /volume2
ozone sh bucket create --layout=FILE_SYSTEM_OPTIMIZED /volume2/fso-bucket2
fallocate -l 10M /tmp/samplefileozone sh key put
/volume2/fso-bucket2/dir1/testfile1 /tmp/samplefile
ozone sh key put /volume2/fso-bucket2/dir1/testfile2 /tmp/samplefile
ozone sh key put /volume2/fso-bucket2/dir1/dir2/testfile1 /tmp/samplefile
ozone sh key put /volume2/fso-bucket2/dir1/dir2/testfile2 /tmp/samplefile
ozone sh key put /volume2/fso-bucket2/dir1/dir2/dir3/testfile1 /tmp/samplefile
ozone sh key put /volume2/fso-bucket2/dir1/dir2/dir3/testfile2 /tmp/samplefile
{code}
h3. Observations
We noticed that namespace summaries for volumes and buckets created using the
{{ozone fs}} command showed inconsistencies.
In contrast, those created using the {{ozone sh}} command behaved correctly.
Both sets of operations generate events that are applied to the namespace
summary tree in Recon. The difference lies in how these events are interpreted.
h3. Event Comparison
h3. Using {{ozone fs}}
{code:java}
Filtered (only fileTable and directoryTable events, in order):
#8 → directoryTable → PUT → /.../dir1
#9 → directoryTable → PUT → /.../dir2
#10 → directoryTable → PUT → /.../dir3
...
#38 → directoryTable → UPDATE → /.../dir1
#67 → directoryTable → UPDATE → /.../dir1
#96 → directoryTable → UPDATE → /.../dir2
#125 → directoryTable → UPDATE → /.../dir2
#154 → directoryTable → UPDATE → /.../dir3
#183 → directoryTable → UPDATE → /.../dir3 {code}
h3. Using {{ozone sh}}
{code:java}
Filtered (only fileTable and directoryTable events, in order):
#5 → directoryTable → PUT → /.../dir1
#11 → fileTable → PUT → /.../testfile1
#18 → fileTable → PUT → /.../testfile2
#20 → directoryTable → PUT → /.../dir2
#26 → fileTable → PUT → /.../testfile1
#33 → fileTable → PUT → /.../testfile2
#35 → directoryTable → PUT → /.../dir3
#41 → fileTable → PUT → /.../testfile1
#48 → fileTable → PUT → /.../testfile2 {code}
As shown, {{ozone sh}} produces only {{PUT}} events, whereas {{ozone fs}}
produces additional {{UPDATE}} and some {{DELETE}} events.
The {{DELETE}} events are harmless, but the {{UPDATE}} events from {{ozone fs}}
are the root cause of the issue.
h3. Root Cause
When Recon processes a directory {{UPDATE}} event, it deletes the old directory
record and replaces it with the updated version.
This triggers both {{deleteDir}} and {{putDirectory}} operations.
The problem arises because during {{{}putDirectory{}}}, the propagation logic
passes an incorrect object ID when the directory already contains files or
subdirectories.
This causes incorrect size and count propagation in the namespace summary.
The bug did not surface earlier because our tests used empty directories, where
size propagation was trivial and unaffected.
h3. Conclusion
This issue highlights a *testing gap* in our Recon integration tests.
We plan to create a parent JIRA to address this by improving test coverage for:
* Namespace summary propagation
* Other Recon endpoints and tasks
This will ensure more validation across all event scenarios.
> Namespace summary API gives wrong count of directories and keys
> ---------------------------------------------------------------
>
> Key: HDDS-13841
> URL: https://issues.apache.org/jira/browse/HDDS-13841
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Recon
> Reporter: Abhishek Pal
> Assignee: Arafat Khan
> Priority: Major
> Labels: pull-request-available
>
> The backend API:
> {code:java}
> api/v1/namespace/summary?path=<path to location> {code}
> gives us wrong metadata details for the number of directories and keys.
> Here is a sample command output for key list and bucket info:
> {code:java}
> ozone sh bucket info volume1/fso-bucket
> {
> "metadata" : { },
> "volumeName" : "volume1",
> "name" : "fso-bucket",
> "storageType" : "DISK",
> "versioning" : false,
> "listCacheSize" : 1000,
> "usedBytes" : 188743680,
> "usedNamespace" : 9,
> "creationTime" : "2025-10-23T14:26:43.493Z",
> "modificationTime" : "2025-10-23T14:26:43.493Z",
> "sourcePathExist" : true,
> "quotaInBytes" : -1,
> "quotaInNamespace" : -1,
> "bucketLayout" : "FILE_SYSTEM_OPTIMIZED",
> "owner" : "om",
> "link" : false
> }{code}
>
> {code:java}
> ozone sh key list volume1/fso-bucket
> [ {
> "volumeName" : "volume1",
> "bucketName" : "fso-bucket",
> "name" : "dir1/",
> "owner" : "",
> "dataSize" : 0,
> "creationTime" : "2025-10-23T14:26:43.521Z",
> "modificationTime" : "2025-10-23T14:27:31.830Z",
> "replicationConfig" : {
> "replicationFactor" : "THREE",
> "requiredNodes" : 3,
> "minimumNodes" : 1,
> "replicationType" : "RATIS"
> },
> "metadata" : { },
> "tags" : { },
> "file" : false
> }, {
> "volumeName" : "volume1",
> "bucketName" : "fso-bucket",
> "name" : "dir1/dir2/",
> "owner" : "",
> "dataSize" : 0,
> "creationTime" : "2025-10-23T14:26:43.521Z",
> "modificationTime" : "2025-10-23T14:27:41.078Z",
> "replicationConfig" : {
> "replicationFactor" : "THREE",
> "requiredNodes" : 3,
> "minimumNodes" : 1,
> "replicationType" : "RATIS"
> },
> "metadata" : { },
> "tags" : { },
> "file" : false
> }, {
> "volumeName" : "volume1",
> "bucketName" : "fso-bucket",
> "name" : "dir1/dir2/dir3/",
> "owner" : "",
> "dataSize" : 0,
> "creationTime" : "2025-10-23T14:26:43.521Z",
> "modificationTime" : "2025-10-23T14:27:56.584Z",
> "replicationConfig" : {
> "replicationFactor" : "THREE",
> "requiredNodes" : 3,
> "minimumNodes" : 1,
> "replicationType" : "RATIS"
> },
> "metadata" : { },
> "tags" : { },
> "file" : false
> }, {
> "volumeName" : "volume1",
> "bucketName" : "fso-bucket",
> "name" : "dir1/dir2/dir3/file1",
> "owner" : "om",
> "dataSize" : 10485760,
> "creationTime" : "2025-10-23T14:27:54.747Z",
> "modificationTime" : "2025-10-23T14:27:56.534Z",
> "replicationConfig" : {
> "replicationFactor" : "THREE",
> "requiredNodes" : 3,
> "minimumNodes" : 1,
> "replicationType" : "RATIS"
> },
> "metadata" : { },
> "tags" : { },
> "file" : true
> }, {
> "volumeName" : "volume1",
> "bucketName" : "fso-bucket",
> "name" : "dir1/dir2/dir3/testfile",
> "owner" : "om",
> "dataSize" : 10485760,
> "creationTime" : "2025-10-23T14:27:43.942Z",
> "modificationTime" : "2025-10-23T14:27:45.590Z",
> "replicationConfig" : {
> "replicationFactor" : "THREE",
> "requiredNodes" : 3,
> "minimumNodes" : 1,
> "replicationType" : "RATIS"
> },
> "metadata" : { },
> "tags" : { },
> "file" : true
> }, {
> "volumeName" : "volume1",
> "bucketName" : "fso-bucket",
> "name" : "dir1/dir2/file1",
> "owner" : "om",
> "dataSize" : 10485760,
> "creationTime" : "2025-10-23T14:27:39.443Z",
> "modificationTime" : "2025-10-23T14:27:41.034Z",
> "replicationConfig" : {
> "replicationFactor" : "THREE",
> "requiredNodes" : 3,
> "minimumNodes" : 1,
> "replicationType" : "RATIS"
> },
> "metadata" : { },
> "tags" : { },
> "file" : true
> }, {
> "volumeName" : "volume1",
> "bucketName" : "fso-bucket",
> "name" : "dir1/dir2/testfile",
> "owner" : "om",
> "dataSize" : 10485760,
> "creationTime" : "2025-10-23T14:27:34.768Z",
> "modificationTime" : "2025-10-23T14:27:36.524Z",
> "replicationConfig" : {
> "replicationFactor" : "THREE",
> "requiredNodes" : 3,
> "minimumNodes" : 1,
> "replicationType" : "RATIS"
> },
> "metadata" : { },
> "tags" : { },
> "file" : true
> }, {
> "volumeName" : "volume1",
> "bucketName" : "fso-bucket",
> "name" : "dir1/file1",
> "owner" : "om",
> "dataSize" : 10485760,
> "creationTime" : "2025-10-23T14:27:30.153Z",
> "modificationTime" : "2025-10-23T14:27:31.789Z",
> "replicationConfig" : {
> "replicationFactor" : "THREE",
> "requiredNodes" : 3,
> "minimumNodes" : 1,
> "replicationType" : "RATIS"
> },
> "metadata" : { },
> "tags" : { },
> "file" : true
> }, {
> "volumeName" : "volume1",
> "bucketName" : "fso-bucket",
> "name" : "dir1/testfile",
> "owner" : "om",
> "dataSize" : 10485760,
> "creationTime" : "2025-10-23T14:27:23.159Z",
> "modificationTime" : "2025-10-23T14:27:24.946Z",
> "replicationConfig" : {
> "replicationFactor" : "THREE",
> "requiredNodes" : 3,
> "minimumNodes" : 1,
> "replicationType" : "RATIS"
> },
> "metadata" : { },
> "tags" : { },
> "file" : true
> } ] {code}
> However the API returns 3 keys and 3 directories only.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
