[
https://issues.apache.org/jira/browse/HDDS-17558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17558:
------------------------------
Affects Version/s: 2.3.0
> EC reconstruct commands rejected by the global reconstruction limit are
> counted in EcReconstructionCmdsDeferredTotal as target datanode deferrals
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17558
> URL: https://issues.apache.org/jira/browse/HDDS-17558
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.3.0
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> CR-9-separate-global-limit-reconstruction-deferred-metric.patch,
> TestBugCR9GlobalLimitDeferredMetric.java
>
>
> h3. Mechanism
> When {{hdds.scm.replication.reconstruction.global.limit}} is reached,
> {{GlobalReconstructionTracker.reserveAndSend}} in {{ReplicationManager}}
> rejects the reconstruct command and calls
> {{ReplicationManagerMetrics.incrECReconstructionCmdsDeferredTotal}}. That is
> the counter {{ReplicationManager.sendThrottledReconstructionCommand}} also
> increments when no target datanode has capacity. Its description says it
> counts commands that could not be sent "due to the pending commands on the
> target datanode", the EC metrics table on the Ozone website ([Erasure Coding
> page|https://github.com/apache/ozone-site/blob/master/docs/03-core-concepts/02-replication/04-erasure-coding.md])
> gives the same meaning, and the Grafana panel that plots it is titled
> "Deferred Commands Rates (Overloaded Nodes)".
> {{EcReconstructionGlobalLimitReachedTotal}} (HDDS-15075) is only incremented
> by the decommission pre-check in {{ECUnderReplicationHandler}}, so global
> limit rejections of missing index reconstructions are counted only as
> datanode overload deferrals (the {{InflightEcReconstruction}} and
> {{EcReconstructionGlobalLimit}} gauges show that the limit is saturated, but
> not how many commands it rejected).
> The PR that added the limit (HDDS-15071,
> [https://github.com/apache/ozone/pull/11054]) noted this as a follow-up and
> proposed a separate {{ec_reconstruction_cmds_deferred_global_limit_total}}
> counter. The draft design doc for HDDS-15014
> ([https://github.com/apache/ozone/pull/10086]) instead says global limit
> deferrals go into the existing counter, so the intended meaning is not
> settled in writing.
> h3. Trigger
> # Set {{hdds.scm.replication.reconstruction.global.limit}} to a positive
> value (the default 0 disables the limit, and then this cannot happen).
> # More EC containers need reconstruction than the limit allows, with idle
> target datanodes.
> # The reconstruct commands beyond the limit are rejected and each adds 1 to
> {{EcReconstructionCmdsDeferredTotal}}.
> h3. Impact
> Metrics only; scheduling is unaffected and the container is retried on the
> next pass. Operators cannot tell cluster limit deferrals from datanode
> overload from this counter, and the overload panel shows load that does not
> exist. No released version is affected: the global limit was added by
> HDDS-15071 and is not in 2.2.1 or earlier.
> h3. Reproduction
> [^TestBugCR9GlobalLimitDeferredMetric.java] ({{hadoop-hdds/server-scm}}, unit
> test with a mocked {{NodeManager}}, no cluster). With the global limit set to
> 1 and idle targets it sends one reconstruct command, then a second that is
> rejected with "Global reconstruction limit (1) reached", and asserts that
> {{EcReconstructionCmdsDeferredTotal}} did not change. On the unfixed code it
> fails with {{expected: <0> but was: <1>}}. A second test is the control: a
> rejection caused by overloaded targets adds 1, as expected. Both pass with
> the patch.
> h3. Patch
> [^CR-9-separate-global-limit-reconstruction-deferred-metric.patch], against
> efe0c7eba1527530d1001f6517b733576fdc6d61. It also applies to master at
> 99182f55b04.
> Adds {{EcReconstructionCmdsDeferredGlobalLimitTotal}} next to
> {{EcReconstructionCmdsDeferredTotal}} and counts global limit rejections
> there instead, so {{EcReconstructionCmdsDeferredTotal}} again counts only
> target datanode deferrals, as its description and every released version do.
> It does not reuse {{EcReconstructionGlobalLimitReachedTotal}} because that is
> one of the four decommission fallback counters, so counting missing index
> deferrals there would mix the two paths. This changes what
> {{EcReconstructionCmdsDeferredTotal}} counts on master when a global limit is
> configured. Covered by new assertions in
> {{TestReplicationManager.testSendReconstructionCommandRejectedWhenGlobalLimitReached}}
> (fails without the change) and
> {{testSendThrottledReconstructionCommandThrowsWhenNoTargets}} (per target
> rejections still go to the old counter only). The replication package unit
> tests of {{hdds-server-scm}} (545 tests) pass and checkstyle is clean.
> Showing the new counter in {{ozone insight}} ({{ReplicaManagerInsight}}) and
> in the SCM overview Grafana dashboard is left for a follow-up. A row for the
> new counter belongs in the website's EC metrics table, as a separate
> ozone-site change, and the HDDS-15014 design draft would need its metrics
> section updated.
> Found by code review of the SCM EC reconstruction global limit and
> decommission reconstruction paths, as part of the TLA+ verification effort
> under HDDS-15926, on commit efe0c7eba1527530d1001f6517b733576fdc6d61. Checked
> against HDDS issues and apache/ozone pull requests for duplicates before
> filing. The attached patch is a proposal for review. Generated with Specula
> (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
