Siyao Meng created HDDS-17064:
---------------------------------
Summary: Full container report carried by a failed heartbeat is
not resent until the next periodic report, hiding volume failure losses for up
to two hours
Key: HDDS-17064
URL: https://issues.apache.org/jira/browse/HDDS-17064
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-2-resend-container-report-after-failed-heartbeat.patch,
TestBugMC2LostFcrNotResent.java
h3. Mechanism
{{StateContext.getFullReports}} clears the per endpoint "ready to send" flag of
the container report as soon as it puts the report into a heartbeat request
(HDDS-5111). If that heartbeat fails with an {{IOException}},
{{HeartbeatEndpointTask.putBackIncrementalReports}} puts back only incremental
and command status reports; its comment says the full reports are kept because
only the latest is needed, but nothing sets the flag again. The container
report is therefore not sent until the next {{StateContext.refreshFullReport}},
which happens only at the next periodic container report or the next volume
failure.
Since HDDS-8448, {{ContainerSet.handleVolumeFailures}} removes the containers
of a failed volume without sending ICRs and relies on that full container
report (FCR) to tell SCM and Recon about the loss. If the one heartbeat
carrying it fails, the loss is not reported.
h3. Trigger
# A volume fails its health check; {{ContainerSet.handleVolumeFailures}}
removes its containers, refreshes the FCR and triggers a heartbeat.
# That heartbeat fails while SCM stays up, for example an RPC timeout
({{ozone.scm.heartbeat.rpc-timeout}}, 5s by default) or a reset of an open
connection. A refused connection is first retried by the RPC client (15 times,
1s apart, by default), and after an SCM restart the datanode re-registers and
sends a fresh report.
# Later heartbeats succeed, without the FCR.
The same happens to a periodic FCR, delaying the receiver's reconciliation by
one more period.
h3. Impact
* SCM keeps the replicas of every container on the failed volume until the next
periodic FCR from that datanode, up to one report period later
({{hdds.container.report.interval}} plus a random delay of up to the same
interval, chosen when the datanode starts, so a fixed period between 60 and 120
minutes by default). In that time ReplicationManager does not see these
containers as under replicated, so their re-replication is delayed by up to
that long (argued from the code, the SCM side was not run). SCM only asks a
datanode to re-register when it is not registered (as after an SCM restart,
which therefore does not hit this), so a reachable datanode is not asked to
send a fresh report. In SCM HA each SCM has its own flag, so only the SCMs
whose heartbeat failed miss the report, and ReplicationManager is affected when
the leader is one of them.
* Recon shows the same lost replicas in its replica and container health views
for the same time (run). Recon asks a datanode to re-register when its
heartbeats stop for 3 heartbeat intervals (180s by default), so at Recon only
failures shorter than that are affected.
h3. Reproduction
PASS, deterministic, unmodified source. [^TestBugMC2LostFcrNotResent.java]
({{hadoop-ozone/recon}}) runs the real datanode {{StateContext}},
{{ContainerSet.handleVolumeFailures}} and {{HeartbeatEndpointTask}} against a
stub RPC proxy that hands each heartbeat's FCR to a real
{{ReconContainerReportHandler}} over a real {{ReconContainerManager}} and
{{SCMNodeManager}}. The volume is a mock whose {{isFailed}} is switched to
true, and the stub fails one heartbeat with a {{ConnectException}}, as the RPC
client throws once its connection retries are used up. After the failed
heartbeat and 10 successful ones, Recon has received only the first FCR and
still lists the replica the datanode removed; a control run without the failed
heartbeat removes it at once. Heartbeats do not go through
{{ReconNodeManager.processHeartbeat}}, so the 180s re-register check was read,
not run. A passing test means the defect is present.
h3. Patch
[^MC-2-resend-container-report-after-failed-heartbeat.patch], against
916aa1d9949cbb10f2750a40c8704340893131a2. It also applies to master at
62d982ff2d9.
A new {{StateContext.putBackFullContainerReport}} sets the container report
flag of the endpoint again, and {{putBackIncrementalReports}} calls it when the
failed request carried a container report. The next heartbeat then sends the
stored container report, which is the one that failed or a newer one. Node and
pipeline reports are left as they are, since their publishers refresh them
every 60s by default. The container report now rides on every following
heartbeat until one succeeds. If the heartbeat failed after SCM processed it,
SCM receives the same report twice, which it handles as an ordinary repeated
report. A heartbeat that fails because it is too large (the situation of
HDDS-16723) keeps carrying the report and can keep failing longer than before,
when only the ICR backlog was retried.
Covered by {{testFailedHeartbeatResendsContainerReport}} in the existing
{{TestHeartbeatEndpointTask}}: after a failed heartbeat, the next one must
carry the container report again and the one after must not, and a later failed
heartbeat that carried no container report must not cause one to be sent.
Without the change it fails because the second heartbeat has no container
report, and without the check that the failed request carried the report it
fails because the fourth heartbeat carries one. With it
{{TestHeartbeatEndpointTask}}, {{TestStateContext}}, {{TestContainerSet}},
{{TestReportPublisher}}, {{TestVolumeSetDiskChecks}}, {{TestOzoneContainer}},
{{TestDatanodeStateMachine}} and {{TestRunningDatanodeState}} (108 tests) and
{{TestEndPoint}} in server-scm (18 tests) pass, checkstyle is clean. With the
patch the reproduction fails, because Recon receives the FCR again with the
next heartbeat and removes the replica.
Found by TLA+ model checking and code review of the datanode container report
and Recon container tracking paths under HDDS-15926, on commit
916aa1d9949cbb10f2750a40c8704340893131a2. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
