Arafat Khan created HDDS-13432:
----------------------------------
Summary: Accelerating Namespace Usage Calculation in Recon
Key: HDDS-13432
URL: https://issues.apache.org/jira/browse/HDDS-13432
Project: Apache Ozone
Issue Type: Improvement
Components: Ozone Recon
Reporter: Arafat Khan
Recon calculates disk usage by recursively traversing the entire directory tree
for a given bucket or volume, performing a RocksDB lookup for every directory.
This process involves millions of individual reads for large buckets with deep
and wide directory structures, causing:
* Extremely high latency (e.g., 54 seconds for 5.6 million directories).
* Poor performance due to millions of RocksDB Get operations, many hitting
disk rather than cache.
* Single-threaded traversal leading to underutilization of CPU resources.
* Poor RocksDB key locality causing inefficient disk seeks and block-cache
misses because directory summaries are stored with random keys.
These issues result in slow and inefficient disk usage calculations, especially
for very large and complex datasets.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
