Devesh Kumar Singh created HDDS-15795:
-----------------------------------------
Summary: Stale recovering container scrubber should use
last-update time instead of container create time
Key: HDDS-15795
URL: https://issues.apache.org/jira/browse/HDDS-15795
Project: Apache Ozone
Issue Type: Task
Components: Ozone Datanode
Reporter: Devesh Kumar Singh
Assignee: Devesh Kumar Singh
## 1. Problem
The stale-recovering-container scrubber marks a `RECOVERING` container
`UNHEALTHY`
once it is older than `ozone.recovering.container.timeout` (default `20m`). The
deadline is computed **once, at container-create time**:
```java
// ContainerSet.addContainer (ContainerSet.java:164-166)
if (container.getContainerData().getState() == RECOVERING) {
recoveringContainerMap.put(clock.millis() + recoveringTimeout, containerId);
}
```
This is an **absolute lifetime cap**, not an activity check. A reconstruction
that is *legitimately still running* (large container, slow disks, or a loaded
cluster) is marked `UNHEALTHY` purely because the clock ran out — the scrubber
cannot tell "slow but healthy" apart from "stuck/abandoned".
Marking an in-flight reconstruction `UNHEALTHY` aborts it, which triggers a
force-delete of the target replica and re-opens the window for the downstream
implicit-OPEN / `CORRUPT_CHUNK` chain (root-caused separately).
## 2. Root cause
The staleness clock is anchored to **create time** and never advances, so an
actively-progressing reconstruction is treated the same as an idle one.
## 3. Proposed solution
Convert the scrubber from an **absolute lifetime cap** into an **idle /
inactivity timeout**: a `RECOVERING` container is marked `UNHEALTHY` only when
it
has received **no writes** for `ozone.recovering.container.timeout`. Every
`WriteChunk` / `PutBlock` to a `RECOVERING` container "touches" it and pushes
its
deadline forward, so an active rebuild is never scrubbed mid-flight; only a
truly
stalled/orphaned `RECOVERING` container ages out.
### Design
1. **Track last-update time on the container.**
Add a `volatile long lastUsedTime` (millis) to `ContainerData`, initialized
at
creation and refreshed on every successful write to the container.
2. **Refresh it on the write path.**
In `KeyValueHandler` write handlers (`handleWriteChunk` / `handlePutBlock`,
or
a shared point after a successful write), call
`containerData.updateLastUsedTime(clock.millis())`.
3. **Make the scrubber decide on idle time, and re-arm active containers.**
In `StaleRecoveringContainerScrubbingService.getTasks()`, when a map entry's
deadline has passed, re-verify against the container's actual `lastUsedTime`:
- `now - lastUsedTime >= recoveringTimeout` → still idle → queue
`markContainerUnhealthy` and remove the entry (current behavior).
- otherwise → the container was written to recently → **remove the stale
entry
and re-insert `lastUsedTime + recoveringTimeout`** as the new deadline
(re-arm), and do **not** mark it.
This preserves the existing sorted-map efficiency (still breaks on the first
not-yet-expired entry) while making the decision activity-aware.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
