errose28 commented on code in PR #8907: URL: https://github.com/apache/ozone/pull/8907#discussion_r2696243879
########## hadoop-hdds/docs/content/design/storage-distribution.md: ########## @@ -0,0 +1,396 @@ +--- +title: Storage Capacity Distribution Design +summary: Proposal for introducing a comprehensive storage distribution dashboard in Recon. +date: 2025-08-05 +jira: HDDS-13177 +status: Under Review +author: Priyesh Karatha +--- + +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, + either express or implied. See the License for the specific + language governing permissions and limitations under the License. +--> + +# Abstract +Ozone currently lacks a unified interface to monitor and analyze storage distribution across its cluster components. This makes it difficult to: + +- Understand data distribution across the cluster +- Debug storage reclamation issues +- Monitor pending deletion progress +- Analyze utilization patterns +- Identify potential bottlenecks and imbalances + +This proposal introduces a comprehensive **Storage Capacity Distribution Dashboard** in Recon to address these challenges. +This initiative significantly enhances operational insight, capacity planning accuracy, and system observability in large-scale Ozone deployments. + +## Approach 1: Recon-based + +### Architectural Overview + +The dashboard operates as an extension to **Recon**, leveraging existing telemetry from: + +- **Ozone Manager (OM)** - namespace-level and key-level metadata. +- **Storage Container Manager (SCM)** - container, block, and replication metrics. +- **DataNodes (DNs)** - raw storage usage and pending deletion data. + +All metrics are aggregated, and exposed through a **RESTful API (/storagedistribution)** for recon dashboard. + + +### Data Flow Summary + +- OM → Recon + - Periodic synchronization happens between OM db to Recon. +- OM -> SCM + - Block deletion request sent by OM to SCM carries block size and replicated size. +- SCM → Recon + - SCM aggregates the blocks in the block deletion request into DeletedBlocksTransaction with aggregated block size and replicated size, persisted into DB. + - SCM maintains and updates a deletion transaction summary for newly created/deleted transactions. Summary is exposed through metrics, API, and also persisted. + - Recon gets the deletion transaction summary using SCM client whenever required. +- SCM → Datanode + - Block deletion transaction sent by SCM to DN carries block size. +- DataNode → Recon + - Exposes TotalPendingDeletionBytes via JMX through BlockDeletingService. + - Recon fetches this data for node-level pending deletion tracking. +- Recon + - Consolidates all component-level data. + - Computes global storage, namespace usage, and pending deletion metrics. + - Exposes results via RestFull API /storagedistribution. + +### Recon API Design + +#### 1. Get Storage Distribution Information + +**Endpoint:** `GET /storageDistribution` + +**Description:** Retrieves comprehensive storage metrics including global capacity, namespace usage, and per-DataNode storage breakdown. + +**Parameters:** None + +**Responses:** 200 OK + +**Response Structure:** + +| Field | Type | Description | +|--------------------|------------------|----------------------------------------| +| globalStorage | Object | Overall storage capacity metrics. | +| globalNamespace | Object | Overall namespace (key) usage metrics. | +| usedSpaceBreakdown | Object | Detailed breakdown of used space. | +| dataNodeUsage | Array of Objects | Per-DataNode storage usage. | + +**Response Objects:** + +##### globalStorage Object +| Field | Type | Description | +|----------------|------|---------------------------------------------------| +| totalUsedSpace | Long | Total used space across all DataNodes in bytes. | +| totalFreeSpace | Long | Total free space across all DataNodes in bytes. | +| totalCapacity | Long | Total raw capacity across all DataNodes in bytes. | + +##### globalNamespace Object +| Field | Type | Description | +|----------------|------|---------------------------------------------------------| +| totalUsedSpace | Long | Total used space for namespace objects (keys) in bytes. | +| totalKeys | Long | Total number of keys in the namespace. | + +##### usedSpaceBreakdown Object +| Field | Type | Description | +|----------------------------|------|--------------------------------------------------------| +| openKeyBytes | Long | Bytes currently held by open keys (not yet committed). | +| committedKeyBytes | Long | Bytes committed to existing keys. | +| preAllocatedContainerBytes | Long | Pre-allocated space for open containers. | Review Comment: > However, if we split this into multiple APIs, the UI would need to fetch data from several endpoints and perform additional calculations on its side There are already two APIs declared in the doc, and one is broken down per component: > 1. Get Storage Distribution Information Endpoint: GET /storageDistribution > 2. Get Pending Deletion Bytes Endpoint: GET /pendingDeletion ... Parameters: component (required): Component type - om, scm, or dn The Recon UI I last saw for this feature had information from both endpoints aggregated. My proposal is also two APIs with one broken down by component, just with different information in each one so that they make sense in isolation. > Since this is currently used only by the Recon UI Changing APIs is hard, we need to think about the future, not just the present use case. Scripting REST calls to Recon is sometimes done currently when we need more detailed information. @devabhishekpal can you add some clarity on the frontend requirements here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
