yandrey321 opened a new pull request, #11885:
URL: https://github.com/apache/ozone/pull/11885
## What changes were proposed in this pull request?
**Problem.** `HeartbeatEndpointTask.call` takes the queued container actions
for an SCM out of `StateContext` when it
builds a heartbeat. If SCM does not process that heartbeat, the actions are
lost:
- **The heartbeat fails** with an `IOException`. `putBackIncrementalReports`
restores only command status reports and
incremental container reports.
- **SCM asks the datanode to re-register**, because it doesn't know the
datanode (for example after an SCM restart) or
because the datanode is unhealthy with an incompatible version.
`SCMDatanodeHeartbeatDispatcher.dispatch` then
returns a `ReregisterCommand` without firing any event for the heartbeat,
and since the RPC succeeded, nothing is
put back.
The CLOSE action for a full container is only created again by a later write
(`HddsDispatcher.sendCloseContainerActionIfNeeded`). If no write follows,
the container stays OPEN in SCM until its
pipeline closes or an administrator closes it. While it is OPEN, its delete
transactions are skipped and
`ReplicationManager`, the balancer and the scanner leave it alone. Pipeline
actions are not affected: close pipeline
actions stay queued in `StateContext.PipelineActionMap` until the pipeline
is removed.
**Change.**
- New `StateContext.putBackContainerActions(actions, endpoint)`. It puts the
actions back at the front of that SCM's
queue, in their original order, and skips actions that are queued again
already, because a write may have queued
the same CLOSE action while the heartbeat was in flight. Nothing happens
if the endpoint was removed meanwhile.
- `HeartbeatEndpointTask.call` puts the heartbeat's container actions back
when the heartbeat fails with an
`IOException`, next to `putBackIncrementalReports`, and when the response
contains a `reregisterCommand`.
- The per-SCM container action queue is declared as a `Deque` instead of a
`Queue` to allow adding at the front; it
already was a `LinkedList`.
**Key decisions.**
- **Resending is safe.** The datanode can't tell whether SCM processed a
heartbeat whose response was lost, so it may
send an action twice. SCM's `CloseContainerEventHandler` only finalizes a
container that is still OPEN, and a close
command for a CLOSING container is deduplicated by its lease; datanodes
already send repeated CLOSE actions today,
one for each write past the threshold.
- **Any reregister command triggers the put-back,** not only the
unregistered-datanode case. SCM returns it only from
the two branches that skip the heartbeat.
- **Put back at the front, in the original order,** which is why the queue
becomes a `Deque`. A heartbeat carries
at most `hdds.container.action.max.limit` actions (default 20), so this
only matters when more are pending, for
example when many containers fill up while SCM is down. Then the actions
that were not processed are the oldest,
and they go out first with the next heartbeat. Appending them at the end
of the queue, as the patch attached to the
Jira does, is equally correct (nothing is lost, and everything is sent
eventually), but it puts them behind every
action queued meanwhile, and each failed heartbeat rotates the queue, so
an older CLOSE action can wait a few more
heartbeats (30 s each by default). The cost is changing the declared type
of the queue, which already is a
`LinkedList`.
- **Only container actions.** Incremental container reports dropped on
re-registration are covered by the full
container report sent when registering. The datanode doesn't put back
command status reports after a reregister
either; that is unchanged here.
**Related.** HDDS-17269 (heartbeat tasks queued behind a slow one) is a
separate scheduling problem. With this
change, the container actions drained by such a stale heartbeat are no
longer lost when it gets a reregister reply or
fails; the queued-task backlog itself is not addressed here.
Generated-by: Claude Code (Claude Opus 5.5)
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-17267
## How was this patch tested?
Unit and integration tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
