Siyao Meng created HDDS-17558:
---------------------------------
Summary: EC reconstruct commands rejected by the global
reconstruction limit are counted in EcReconstructionCmdsDeferredTotal as target
datanode deferrals
Key: HDDS-17558
URL: https://issues.apache.org/jira/browse/HDDS-17558
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-9-separate-global-limit-reconstruction-deferred-metric.patch,
TestBugCR9GlobalLimitDeferredMetric.java
h3. Mechanism
When {{hdds.scm.replication.reconstruction.global.limit}} is reached,
{{GlobalReconstructionTracker.reserveAndSend}} in {{ReplicationManager}}
rejects the reconstruct command and calls
{{ReplicationManagerMetrics.incrECReconstructionCmdsDeferredTotal}}. That is
the counter {{ReplicationManager.sendThrottledReconstructionCommand}} also
increments when no target datanode has capacity. Its description says it counts
commands that could not be sent "due to the pending commands on the target
datanode", the EC metrics table on the Ozone website ([Erasure Coding
page|https://github.com/apache/ozone-site/blob/master/docs/03-core-concepts/02-replication/04-erasure-coding.md])
gives the same meaning, and the Grafana panel that plots it is titled
"Deferred Commands Rates (Overloaded Nodes)".
{{EcReconstructionGlobalLimitReachedTotal}} (HDDS-15075) is only incremented by
the decommission pre-check in {{ECUnderReplicationHandler}}, so global limit
rejections of missing index reconstructions are counted only as datanode
overload deferrals (the {{InflightEcReconstruction}} and
{{EcReconstructionGlobalLimit}} gauges show that the limit is saturated, but
not how many commands it rejected).
The PR that added the limit (HDDS-15071,
[https://github.com/apache/ozone/pull/11054]) noted this as a follow-up and
proposed a separate {{ec_reconstruction_cmds_deferred_global_limit_total}}
counter. The draft design doc for HDDS-15014
([https://github.com/apache/ozone/pull/10086]) instead says global limit
deferrals go into the existing counter, so the intended meaning is not settled
in writing.
h3. Trigger
# Set {{hdds.scm.replication.reconstruction.global.limit}} to a positive value
(the default 0 disables the limit, and then this cannot happen).
# More EC containers need reconstruction than the limit allows, with idle
target datanodes.
# The reconstruct commands beyond the limit are rejected and each adds 1 to
{{EcReconstructionCmdsDeferredTotal}}.
h3. Impact
Metrics only; scheduling is unaffected and the container is retried on the next
pass. Operators cannot tell cluster limit deferrals from datanode overload from
this counter, and the overload panel shows load that does not exist. No
released version is affected: the global limit was added by HDDS-15071 and is
not in 2.2.1 or earlier.
h3. Reproduction
[^TestBugCR9GlobalLimitDeferredMetric.java] ({{hadoop-hdds/server-scm}}, unit
test with a mocked {{NodeManager}}, no cluster). With the global limit set to 1
and idle targets it sends one reconstruct command, then a second that is
rejected with "Global reconstruction limit (1) reached", and asserts that
{{EcReconstructionCmdsDeferredTotal}} did not change. On the unfixed code it
fails with {{expected: <0> but was: <1>}}. A second test is the control: a
rejection caused by overloaded targets adds 1, as expected. Both pass with the
patch.
h3. Patch
[^CR-9-separate-global-limit-reconstruction-deferred-metric.patch], against
efe0c7eba1527530d1001f6517b733576fdc6d61. It also applies to master at
99182f55b04.
Adds {{EcReconstructionCmdsDeferredGlobalLimitTotal}} next to
{{EcReconstructionCmdsDeferredTotal}} and counts global limit rejections there
instead, so {{EcReconstructionCmdsDeferredTotal}} again counts only target
datanode deferrals, as its description and every released version do. It does
not reuse {{EcReconstructionGlobalLimitReachedTotal}} because that is one of
the four decommission fallback counters, so counting missing index deferrals
there would mix the two paths. This changes what
{{EcReconstructionCmdsDeferredTotal}} counts on master when a global limit is
configured. Covered by new assertions in
{{TestReplicationManager.testSendReconstructionCommandRejectedWhenGlobalLimitReached}}
(fails without the change) and
{{testSendThrottledReconstructionCommandThrowsWhenNoTargets}} (per target
rejections still go to the old counter only). The replication package unit
tests of {{hdds-server-scm}} (545 tests) pass and checkstyle is clean. Showing
the new counter in {{ozone insight}} ({{ReplicaManagerInsight}}) and in the SCM
overview Grafana dashboard is left for a follow-up. A row for the new counter
belongs in the website's EC metrics table, as a separate ozone-site change, and
the HDDS-15014 design draft would need its metrics section updated.
Found by code review of the SCM EC reconstruction global limit and decommission
reconstruction paths, as part of the TLA+ verification effort under HDDS-15926,
on commit efe0c7eba1527530d1001f6517b733576fdc6d61. Checked against HDDS issues
and apache/ozone pull requests for duplicates before filing. The attached patch
is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
