devmadhuu opened a new pull request, #12079:
URL: https://github.com/apache/ozone/pull/12079
## What changes were proposed in this pull request?
SCM currently logs tier-placement failures but does not expose counters
showing why allocation on `SSD, DISK or ARCHIVE` failed. A write can succeed
through fallback, leaving administrators without metrics explaining why the
preferred tier could not be used.
This PR adds the following counter to `SCMPipelineMetrics` for
RATIS/standalone allocation:
`scm_pipeline_metrics_tier_placement_failures{requestedtier="SSD",reason="no_space"}`
<html><head></head><body><p>The counter uses bounded labels:</p>
Label | Values
-- | --
requestedtier | SSD, DISK, ARCHIVE, UNKNOWN
reason | no_space, no_suitable_node, other
<p>Reasons are classified using existing SCM result codes:</p>
- no_space: insufficient eligible storage of the requested type. This
includes space/type eligibility failures and does not necessarily mean the
physical disks are full.
- no_suitable_node: insufficient healthy/suitable nodes or placement
restrictions such as pipeline limits.
- other: unclassified failures, including ordinary I/O errors and readiness
timeouts.
A failure is counted once after allocation on that tier exhausts the
existing retries. Successful recovery on the same tier does not increment the
counter.
For example, suppose ten HOT block allocations prefer `SSD`. Six succeed on
`SSD`, three fail its space/type check, and one fails suitable-node selection.
If all four unsuccessful `SSD` allocations succeed through DISK fallback,
administrators see:
```
scm_pipeline_metrics_tier_placement_failures{requestedtier="SSD",reason="no_space"}
3
scm_pipeline_metrics_tier_placement_failures{requestedtier="SSD",reason="no_suitable_node"}
1
scm_pipeline_metrics_tier_placement_failures{requestedtier="DISK",reason="no_space"}
0
```
These counters measure failed tier allocations, rather than failed keys or
bytes. They reset with SCM metrics registration. All tier/reason combinations
are exported, including zeros.
Existing aggregate metrics, allocation decisions, retry behavior, exception
messages and fallback rules remain unchanged. EC related tier allocation
metrics will be done in follow up PR.
## What is the link to the Apache JIRA?
[HDDS-17820
(https://issues.apache.org/jira/browse/HDDS-17820)](https://issues.apache.org/jira/browse/HDDS-17820)
## How was this patch tested?
- Added `TestSCMTierPlacementMetrics` for reason classification, tier
separation, UNKNOWN handling, Prometheus output, concurrent updates and
registration/reset behavior.
- Extended `TestWritableRatisContainerProvider` to verify terminal failure
counting, successful recovery, readiness failures and unchanged exceptions.
- Extended `TestBlockManager` to verify permitted/disabled fallback and
separate accounting when both preferred and fallback tiers fail.
- SCM allocation, pipeline and placement regression run: 508 passed, 19
existing skips, no failures or errors.
- Existing pipeline-metrics mini-cluster tests: 3 passed. An existing
pipeline-command test also passed.
</body></html>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]