[
https://issues.apache.org/jira/browse/HDDS-15795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devesh Kumar Singh updated HDDS-15795:
--------------------------------------
Description:
Discussed the approach and solution with [~umamahesh]
1. Problem
The stale-recovering-container scrubber marks a `RECOVERING` container
`UNHEALTHY`
once it is older than `ozone.recovering.container.timeout` (default `20m`). The
deadline is computed **once, at container-create time**:
{code:java}
// ContainerSet.addContainer (ContainerSet.java:164-166)
if (container.getContainerData().getState() == RECOVERING) {
recoveringContainerMap.put(clock.millis() + recoveringTimeout, containerId);
}
{code}
This is the gap for lifetime, not an activity check. A reconstruction
that is *legitimately still running* (large container, slow disks, or a loaded
cluster) is marked `UNHEALTHY` purely because the clock ran out — the scrubber
cannot tell "slow but healthy" apart from "stuck/abandoned".
Marking an in-flight reconstruction `UNHEALTHY` aborts it, which triggers a
force-delete of the target replica and re-opens the window for the downstream
implicit-OPEN / `CORRUPT_CHUNK` chain (root-caused separately).
2. Root cause
The staleness clock is anchored to *create time* and never advances, so an
actively-progressing reconstruction is treated the same as an idle one.
3. Proposed solution
Convert the scrubber from an *absolute lifetime cap* into an *idle /
inactivity timeout*: a `RECOVERING` container is marked `UNHEALTHY` only when it
has received *no writes* for `ozone.recovering.container.timeout`. Every
`WriteChunk` / `PutBlock` to a `RECOVERING` container "touches" it and pushes
its
deadline forward, so an active rebuild is never scrubbed mid-flight; only a
truly
stalled/orphaned `RECOVERING` container ages out.
*Design*
1. *Track last-update time on the container*
Add a `volatile long lastUsedTime` (millis) to `ContainerData`, initialized
at
creation and refreshed on every successful write to the container.
2. *Refresh it on the write path*
In `KeyValueHandler` write handlers (`handleWriteChunk` / `handlePutBlock`,
or
a shared point after a successful write), call
`containerData.updateLastUsedTime(clock.millis())`.
3. *Make the scrubber decide on idle time, and re-arm active containers*
In `StaleRecoveringContainerScrubbingService.getTasks()`, when a map entry's
deadline has passed, re-verify against the container's actual `lastUsedTime`:
- `now - lastUsedTime >= recoveringTimeout` → still idle → queue
`markContainerUnhealthy` and remove the entry (current behavior).
- otherwise → the container was written to recently → **remove the stale
entry
and re-insert `lastUsedTime + recoveringTimeout`** as the new deadline
(re-arm), and do *not* mark it.
This preserves the existing sorted-map efficiency (still breaks on the first
not-yet-expired entry) while making the decision activity-aware.
was:
Discussed the approach and solution with [~umamahesh]
1. Problem
The stale-recovering-container scrubber marks a `RECOVERING` container
`UNHEALTHY`
once it is older than `ozone.recovering.container.timeout` (default `20m`). The
deadline is computed **once, at container-create time**:
{code:java}
// ContainerSet.addContainer (ContainerSet.java:164-166)
if (container.getContainerData().getState() == RECOVERING) {
recoveringContainerMap.put(clock.millis() + recoveringTimeout, containerId);
}
{code}
This is the gap for lifetime, not an activity check. A reconstruction
that is *legitimately still running* (large container, slow disks, or a loaded
cluster) is marked `UNHEALTHY` purely because the clock ran out — the scrubber
cannot tell "slow but healthy" apart from "stuck/abandoned".
Marking an in-flight reconstruction `UNHEALTHY` aborts it, which triggers a
force-delete of the target replica and re-opens the window for the downstream
implicit-OPEN / `CORRUPT_CHUNK` chain (root-caused separately).
2. Root cause
The staleness clock is anchored to *create time* and never advances, so an
actively-progressing reconstruction is treated the same as an idle one.
3. Proposed solution
Convert the scrubber from an *absolute lifetime cap* into an *idle /
inactivity timeout*: a `RECOVERING` container is marked `UNHEALTHY` only when it
has received *no writes* for `ozone.recovering.container.timeout`. Every
`WriteChunk` / `PutBlock` to a `RECOVERING` container "touches" it and pushes
its
deadline forward, so an active rebuild is never scrubbed mid-flight; only a
truly
stalled/orphaned `RECOVERING` container ages out.
*Design*
1. *Track last-update time on the container*
Add a `volatile long lastUsedTime` (millis) to `ContainerData`, initialized
at
creation and refreshed on every successful write to the container.
2. *Refresh it on the write path*
In `KeyValueHandler` write handlers (`handleWriteChunk` / `handlePutBlock`,
or
a shared point after a successful write), call
`containerData.updateLastUsedTime(clock.millis())`.
3. *Make the scrubber decide on idle time, and re-arm active containers*
In `StaleRecoveringContainerScrubbingService.getTasks()`, when a map entry's
deadline has passed, re-verify against the container's actual `lastUsedTime`:
- `now - lastUsedTime >= recoveringTimeout` → still idle → queue
`markContainerUnhealthy` and remove the entry (current behavior).
- otherwise → the container was written to recently → **remove the stale
entry
and re-insert `lastUsedTime + recoveringTimeout`** as the new deadline
(re-arm), and do **not** mark it.
This preserves the existing sorted-map efficiency (still breaks on the first
not-yet-expired entry) while making the decision activity-aware.
> Stale recovering container scrubber should use last-update time instead of
> container create time
> ------------------------------------------------------------------------------------------------
>
> Key: HDDS-15795
> URL: https://issues.apache.org/jira/browse/HDDS-15795
> Project: Apache Ozone
> Issue Type: Task
> Components: Ozone Datanode
> Reporter: Devesh Kumar Singh
> Assignee: Devesh Kumar Singh
> Priority: Major
>
> Discussed the approach and solution with [~umamahesh]
> 1. Problem
> The stale-recovering-container scrubber marks a `RECOVERING` container
> `UNHEALTHY`
> once it is older than `ozone.recovering.container.timeout` (default `20m`).
> The
> deadline is computed **once, at container-create time**:
> {code:java}
> // ContainerSet.addContainer (ContainerSet.java:164-166)
> if (container.getContainerData().getState() == RECOVERING) {
> recoveringContainerMap.put(clock.millis() + recoveringTimeout, containerId);
> }
> {code}
> This is the gap for lifetime, not an activity check. A reconstruction
> that is *legitimately still running* (large container, slow disks, or a loaded
> cluster) is marked `UNHEALTHY` purely because the clock ran out — the scrubber
> cannot tell "slow but healthy" apart from "stuck/abandoned".
> Marking an in-flight reconstruction `UNHEALTHY` aborts it, which triggers a
> force-delete of the target replica and re-opens the window for the downstream
> implicit-OPEN / `CORRUPT_CHUNK` chain (root-caused separately).
> 2. Root cause
> The staleness clock is anchored to *create time* and never advances, so an
> actively-progressing reconstruction is treated the same as an idle one.
> 3. Proposed solution
> Convert the scrubber from an *absolute lifetime cap* into an *idle /
> inactivity timeout*: a `RECOVERING` container is marked `UNHEALTHY` only when
> it
> has received *no writes* for `ozone.recovering.container.timeout`. Every
> `WriteChunk` / `PutBlock` to a `RECOVERING` container "touches" it and pushes
> its
> deadline forward, so an active rebuild is never scrubbed mid-flight; only a
> truly
> stalled/orphaned `RECOVERING` container ages out.
> *Design*
> 1. *Track last-update time on the container*
> Add a `volatile long lastUsedTime` (millis) to `ContainerData`,
> initialized at
> creation and refreshed on every successful write to the container.
> 2. *Refresh it on the write path*
> In `KeyValueHandler` write handlers (`handleWriteChunk` /
> `handlePutBlock`, or
> a shared point after a successful write), call
> `containerData.updateLastUsedTime(clock.millis())`.
> 3. *Make the scrubber decide on idle time, and re-arm active containers*
> In `StaleRecoveringContainerScrubbingService.getTasks()`, when a map
> entry's
> deadline has passed, re-verify against the container's actual
> `lastUsedTime`:
> - `now - lastUsedTime >= recoveringTimeout` → still idle → queue
> `markContainerUnhealthy` and remove the entry (current behavior).
> - otherwise → the container was written to recently → **remove the stale
> entry
> and re-insert `lastUsedTime + recoveringTimeout`** as the new deadline
> (re-arm), and do *not* mark it.
> This preserves the existing sorted-map efficiency (still breaks on the
> first
> not-yet-expired entry) while making the decision activity-aware.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
