devmadhuu opened a new pull request, #12079:
URL: https://github.com/apache/ozone/pull/12079

   ## What changes were proposed in this pull request?
   
   SCM currently logs tier-placement failures but does not expose counters 
showing why allocation on `SSD, DISK or ARCHIVE` failed. A write can succeed 
through fallback, leaving administrators without metrics explaining why the 
preferred tier could not be used.
   
   This PR adds the following counter to `SCMPipelineMetrics` for 
RATIS/standalone allocation:
   
   
`scm_pipeline_metrics_tier_placement_failures{requestedtier="SSD",reason="no_space"}`
   
   <html><head></head><body><p>The counter uses bounded labels:</p>
   
   Label | Values
   -- | --
   requestedtier | SSD, DISK, ARCHIVE, UNKNOWN
   reason | no_space, no_suitable_node, other
   
   
   <p>Reasons are classified using existing SCM result codes:</p>
   - no_space: insufficient eligible storage of the requested type. This 
includes space/type eligibility failures and does not necessarily mean the 
physical disks are full.
   - no_suitable_node: insufficient healthy/suitable nodes or placement 
restrictions such as pipeline limits.
   - other: unclassified failures, including ordinary I/O errors and readiness 
timeouts.
   
   A failure is counted once after allocation on that tier exhausts the 
existing retries. Successful recovery on the same tier does not increment the 
counter.
   
   For example, suppose ten HOT block allocations prefer `SSD`. Six succeed on 
`SSD`, three fail its space/type check, and one fails suitable-node selection. 
If all four unsuccessful `SSD` allocations succeed through DISK fallback, 
administrators see:
   
   ```
   
scm_pipeline_metrics_tier_placement_failures{requestedtier="SSD",reason="no_space"}
 3
   
scm_pipeline_metrics_tier_placement_failures{requestedtier="SSD",reason="no_suitable_node"}
 1
   
scm_pipeline_metrics_tier_placement_failures{requestedtier="DISK",reason="no_space"}
 0
   ```
   
   These counters measure failed tier allocations, rather than failed keys or 
bytes. They reset with SCM metrics registration. All tier/reason combinations 
are exported, including zeros.
   
   Existing aggregate metrics, allocation decisions, retry behavior, exception 
messages and fallback rules remain unchanged.  EC related tier allocation 
metrics will be done in follow up PR.
   
   ## What is the link to the Apache JIRA?
   
   [HDDS-17820 
(https://issues.apache.org/jira/browse/HDDS-17820)](https://issues.apache.org/jira/browse/HDDS-17820)
   
   ## How was this patch tested?
   
   - Added `TestSCMTierPlacementMetrics` for reason classification, tier 
separation, UNKNOWN handling, Prometheus output, concurrent updates and 
registration/reset behavior.
   - Extended `TestWritableRatisContainerProvider` to verify terminal failure 
counting, successful recovery, readiness failures and unchanged exceptions.
   - Extended `TestBlockManager` to verify permitted/disabled fallback and 
separate accounting when both preferred and fallback tiers fail.
   - SCM allocation, pipeline and placement regression run: 508 passed, 19 
existing skips, no failures or errors.
   - Existing pipeline-metrics mini-cluster tests: 3 passed. An existing 
pipeline-command test also passed.
   
   </body></html>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to