[
https://issues.apache.org/jira/browse/HDDS-15594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krishna Kumar Asawa reassigned HDDS-15594:
------------------------------------------
Assignee: Gargi Jaiswal
> Rack Aware and Rack Scatter Policies to consider rack density/capacity
> ----------------------------------------------------------------------
>
> Key: HDDS-15594
> URL: https://issues.apache.org/jira/browse/HDDS-15594
> Project: Apache Ozone
> Issue Type: Task
> Reporter: Wei-Chiu Chuang
> Assignee: Gargi Jaiswal
> Priority: Major
>
> h1. Problem Statement
> The current Apache Ozone container placement policies
> (`SCMContainerPlacementRackAware` and `SCMContainerPlacementRackScatter`)
> select racks and nodes under those racks without considering the aggregate
> storage capacity or density of the racks.
> *(Note: Node-level capacity awareness is addressed separately; this proposal
> focuses strictly on rack-level capacity imbalances).*
> In a heterogeneous deployment (e.g., where Rack A has an aggregate capacity
> of 1PB across its datanodes, and Rack B has an aggregate capacity of 10PB):
> 1. **Uniform Rack Selection:** The placement policies select racks uniformly
> at random or in a round-robin/scatter manner to satisfy rack-level fault
> tolerance (e.g., 2 racks for 3-replica Ratis pipelines, or maximizing unique
> racks for Erasure Coding).
> 2. **Aggressive Depletion of Low-Capacity Racks:** Because the placement
> policy treats all racks as having equal capacity, the 1PB rack receives a
> similar number of container allocations as the 10PB rack. Consequently, the
> 1PB rack will reach capacity **10 times faster**.
> 3. **Loss of Rack-Level Fault Tolerance:** Once the 1PB rack is full, SCM can
> no longer allocate new containers that span that rack. This forces SCM to
> either fail new container allocations or fallback to placing multiple
> replicas on the same rack, violating the rack-safety policy.
> 4. **Sub-optimal Rebalancing:** Even if individual datanode capacity policies
> (or the `ContainerBalancer`) optimize node-level usage, the placement path
> lacks the global rack-level awareness needed to prevent entire low-capacity
> racks from filling up prematurely.
> h1. Proposed Improvement
> We should introduce rack capacity and density awareness into SCM placement
> policies to weight the selection of racks themselves:
> h2. Option 1: Rack-Weighted Selection
> * Modify the rack selection step in `SCMContainerPlacementRackAware` and
> `SCMContainerPlacementRackScatter`.
> * The probability of selecting a particular rack should be weighted by the
> aggregate capacity (or aggregate remaining capacity) of all healthy, active
> datanodes in that rack.
> h2. Option 2: Rack-Capacity aware Placement Constraints
> * When selecting target racks for container replicas (especially under
> Erasure Coding where spanning multiple racks is critical), the algorithm
> should optimize the distribution such that higher-density racks host a
> proportionally larger share of the replica load without violating fault
> tolerance bounds.
> h1. Benefits
> * Prevents low-capacity racks from filling up prematurely in heterogeneous
> rack environments.
> * Proactively preserves rack-level fault tolerance for the entire cluster by
> spreading the write load proportionally to each rack's storage footprint.
> * Reduces I/O and network overhead generated by reactive post-write balancing
> (`ContainerBalancer`).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
