[
https://issues.apache.org/jira/browse/HDDS-15795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18094955#comment-18094955
]
Devesh Kumar Singh commented on HDDS-15795:
-------------------------------------------
Thank you [~umamahesh]
> Stale recovering container scrubber should use last-update time instead of
> container create time
> ------------------------------------------------------------------------------------------------
>
> Key: HDDS-15795
> URL: https://issues.apache.org/jira/browse/HDDS-15795
> Project: Apache Ozone
> Issue Type: Task
> Components: Ozone Datanode
> Reporter: Devesh Kumar Singh
> Assignee: Devesh Kumar Singh
> Priority: Major
>
> ## 1. Problem
> The stale-recovering-container scrubber marks a `RECOVERING` container
> `UNHEALTHY`
> once it is older than `ozone.recovering.container.timeout` (default `20m`).
> The
> deadline is computed **once, at container-create time**:
> {code:java}
> // ContainerSet.addContainer (ContainerSet.java:164-166)
> if (container.getContainerData().getState() == RECOVERING) {
> recoveringContainerMap.put(clock.millis() + recoveringTimeout, containerId);
> }
> {code}
> This is an **absolute lifetime cap**, not an activity check. A reconstruction
> that is *legitimately still running* (large container, slow disks, or a loaded
> cluster) is marked `UNHEALTHY` purely because the clock ran out — the scrubber
> cannot tell "slow but healthy" apart from "stuck/abandoned".
> Marking an in-flight reconstruction `UNHEALTHY` aborts it, which triggers a
> force-delete of the target replica and re-opens the window for the downstream
> implicit-OPEN / `CORRUPT_CHUNK` chain (root-caused separately).
> ## 2. Root cause
> The staleness clock is anchored to **create time** and never advances, so an
> actively-progressing reconstruction is treated the same as an idle one.
> ## 3. Proposed solution
> Convert the scrubber from an **absolute lifetime cap** into an **idle /
> inactivity timeout**: a `RECOVERING` container is marked `UNHEALTHY` only
> when it
> has received **no writes** for `ozone.recovering.container.timeout`. Every
> `WriteChunk` / `PutBlock` to a `RECOVERING` container "touches" it and pushes
> its
> deadline forward, so an active rebuild is never scrubbed mid-flight; only a
> truly
> stalled/orphaned `RECOVERING` container ages out.
> ### Design
> 1. **Track last-update time on the container.**
> Add a `volatile long lastUsedTime` (millis) to `ContainerData`,
> initialized at
> creation and refreshed on every successful write to the container.
> 2. **Refresh it on the write path.**
> In `KeyValueHandler` write handlers (`handleWriteChunk` /
> `handlePutBlock`, or
> a shared point after a successful write), call
> `containerData.updateLastUsedTime(clock.millis())`.
> 3. **Make the scrubber decide on idle time, and re-arm active containers.**
> In `StaleRecoveringContainerScrubbingService.getTasks()`, when a map
> entry's
> deadline has passed, re-verify against the container's actual
> `lastUsedTime`:
> - `now - lastUsedTime >= recoveringTimeout` → still idle → queue
> `markContainerUnhealthy` and remove the entry (current behavior).
> - otherwise → the container was written to recently → **remove the stale
> entry
> and re-insert `lastUsedTime + recoveringTimeout`** as the new deadline
> (re-arm), and do **not** mark it.
> This preserves the existing sorted-map efficiency (still breaks on the
> first
> not-yet-expired entry) while making the decision activity-aware.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
