priyeshkaratha commented on code in PR #8907: URL: https://github.com/apache/ozone/pull/8907#discussion_r2409354539
########## hadoop-hdds/docs/content/design/storage-distribution.md: ########## @@ -0,0 +1,180 @@ +--- +title: Storage Capacity Distribution Dashboard +summary: Proposal for introducing a comprehensive storage distribution dashboard in Recon. +date: 2025-08-05 +jira: HDDS-13177 +status: Under Review +author: Priyesh Karatha +--- + +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, + either express or implied. See the License for the specific + language governing permissions and limitations under the License. +--> + +# Abstract + +Ozone currently lacks a unified interface to monitor and analyze storage distribution across its cluster components. This makes it difficult to: + +- Understand data distribution across the cluster +- Debug storage reclamation issues +- Monitor pending deletion progress +- Analyze utilization patterns +- Identify potential bottlenecks and imbalances + +This proposal introduces a comprehensive **Storage Capacity Distribution Dashboard** in Recon to address these challenges. + +--- + +# Key Features + +## 1. Storage Distribution Analysis + +Detailed breakdown of storage usage across components: + +- **Global Used Space**: Represents the actual physical storage consumed on the DataNodes. +- **Global Namespace Space**: Logical size of the namespace, calculated as the sum of pendingDirectorySize + pendingKeySize + totalOpenKeySize + totalCommittedSize, multiplied by the replication factor. +- **Open Keys and Open Files**: Space occupied by data in open keys and files that are not yet finalized. +- **Committed Keys**: Space used by fully committed key–value pairs. +- **Component-wise Distribution**: Breakdown of metrics across OM, SCM, and DataNodes. + +## 2. Deletion Progress Monitoring + +Track pending deletions at various stages: + +- **OM Pending Deletions**: Keys marked for deletion at OM +- **SCM Pending Deletions**: Container-level deletions managed by SCM +- **DataNode Pending Deletions**: Block-level deletion metrics on each DataNode + +## 3. Cluster Overview Metrics + +Summarized cluster statistics: + +- Total capacity and used space +- Free space distribution across components + +--- + +# Implementation Approaches + +## Approach 1: Recon-based Implementation + +Leverage the existing Recon service to build the dashboard with centralized and efficient data collection. +Recon currently maintains synchronization with the OM database and constructs the NSSummary tree, providing established calculation logic for metrics such as openKeysBytes and committedBytes. +Additionally, Recon already possesses a comprehensive physical and logical capacity breakdown through its OM DB insights component. + +### Benefits + +- **Unified Data Source**: All metrics aggregated centrally in Recon +- **Performance Optimization**: Incremental sync reduces the load +- **Reduced Overhead**: Avoids redundant calculations across services +- **Code Reusability**: Built on top of existing Recon infrastructure and endpoints + +### Component-wise Enhancements + +#### **DataNodes (DN)** + +- **Current State**: DNs expose storage metrics in their reports +- **Enhancement**: + - Add `pending deletion byte counters` in container metadata + - Calculate total pending per DN from container metadata and publish metrics +- **Responsibilities**: + - Report actual and pending deletion usage per container + +#### **Storage Container Manager (SCM)** + +- **Current Gap**: No block size tracking in the block deletion process +- **Enhancement**: + - Track block sizes when OM issues a deletion request + - Send a deletion command to DN along with block size and replication factor + + ``` + OM → SCM: block deletion request + block size + SCM → DN: delete command + block size + replication factor + ``` + +- **Responsibilities**: + - Serve as the metadata bridge between logical keys and physical blocks + +#### **Ozone Manager (OM)** + +- **Enhancement**: + - Compute block sizes during deletion +- **Responsibilities**: + - Expose logical storage metrics: committed keys, open keys, namespace usage Review Comment: You’re right. The OM related calculations are handled in Recon. Since the OM database is already synced with Recon, that’s why I mentioned it that way. I’ll reword it to make it clearer if it’s confusing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
