[
https://issues.apache.org/jira/browse/HDDS-17065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17065:
------------------------------
Attachment: TestBugCR5ContainerSizeCountTaskStopsAfterFailedRun.java
> Recon ContainerSizeCountTask stops for good after one failed run, freezing
> the container size distribution until restart
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17065
> URL: https://issues.apache.org/jira/browse/HDDS-17065
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> CR-5-keep-container-size-count-task-running-and-recount-after-failed-run.patch,
> TestBugCR5ContainerSizeCountTaskStopsAfterFailedRun.java
>
>
> h3. Mechanism
> {{ContainerSizeCountTask.run}} has its {{catch (Throwable)}} outside the
> {{while (canRun())}} loop, and {{ReconScmTask.initializeAndRunTask}} passes
> every exception up. When one run fails, the thread logs "Exception in
> Container Size Distribution task Thread." at DEBUG and ends. The task's
> running flag stays true, so {{isRunning()}} keeps reporting the task as
> running and {{start()}} would not start it again; nothing else does either.
> What fails in practice is the Recon SQL database: the {{truncate}} in
> {{runTask}}, or {{findById}}, {{insert}} or {{update}} in
> {{writeCountsToDB}}, all of which throw a jOOQ {{DataAccessException}} when
> the database rejects the call. Listing the containers is in memory, and
> errors for a single container are already caught per container.
> Letting the loop simply continue is not enough. {{processContainers}} updates
> its in memory {{processedContainers}} before {{writeCountsToDB}}, and that
> write is an insert followed by a separate update. After a failed write the
> next run computes its delta from the already updated map, so the failed
> change is never applied; if the insert went through and the update did not,
> the table also keeps half of the change. Either way the counts stay wrong
> until Recon restarts, which recounts from scratch.
> h3. Trigger
> Any failure of the Recon SQL database during one run of the task (default
> every 60s, {{ozone.recon.task.containercounttask.interval}}), for example an
> outage of an external database configured with
> {{ozone.recon.sql.db.jdbc.url}}, or a full disk under the embedded Derby
> database.
> h3. Impact
> * After the thread has ended, {{GET /api/v1/utilization/containerCount}} and
> the "Container Size Distribution" chart on Recon's Insights page keep showing
> the counts of the last successful run until Recon restarts. If the failed run
> was the first one after startup, they show nothing, or the rows the previous
> Recon process left, since the table persists across restarts. The task logs
> the failure only at DEBUG. In a real database outage
> {{ReconTaskStatusUpdater}} also logs an ERROR "Failed to update table for
> task" without the cause, but nothing reports that the thread has ended.
> {{/api/v1/task/status}} shows the task's last run as failed, with a timestamp
> that no longer advances.
> * Only this statistic is affected; the endpoint is its only reader. SCM,
> datanodes and data are not affected.
> h3. Reproduction
> [^TestBugCR5ContainerSizeCountTaskStopsAfterFailedRun.java] goes in
> {{hadoop-ozone/recon}} (target path and Maven command in the class comment)
> and passes on unmodified source when the bug is present. It runs the real
> {{ContainerSizeCountTask}} on Recon's real SQL schema (embedded Derby) and
> reads the result through the real {{UtilizationEndpoint}}; the container
> manager is a mock and the DAO is a spy of the real DAO whose first write
> throws {{DataAccessException}}.
> * {{oneFailedRunStopsTheTask}} (200ms interval): the first run fails, then
> the cluster gains a second container. After 12 intervals the task had listed
> the containers once, the endpoint returned no rows for two containers,
> {{isRunning()}} returned true, and the task logged nothing at INFO or above
> (the failure is injected at the size count DAO only, so the task status
> writes succeed).
> * {{retryAfterPartialWriteKeepsWrongCounts}} shows why a plain retry is not
> enough: runs are called directly, a container moves from the 2GB bin to the
> 4GB bin, the new 4GB row is inserted and the update of the 2GB row fails.
> After the next successful run the endpoint still returns one container in
> each bin for a single 2.5GB container.
> What was not exercised: a real outage of an external database or a full disk;
> the failure is injected at the DAO.
> h3. Patch
> [^CR-5-keep-container-size-count-task-running-and-recount-after-failed-run.patch],
> against 916aa1d9949cbb10f2750a40c8704340893131a2. It also applies to master
> at 14e0611fdd5.
> {{ContainerSizeCountTask.run}} now catches a failed run inside the loop, logs
> it at ERROR, records the failed run in the task status and waits the interval
> before the next run. {{processContainers}} clears {{processedContainers}}
> when the run fails, so the next run truncates the table and recounts all
> containers, as Recon does at startup. A recount is used instead of rolling
> back the in memory state because the table may already hold part of the
> failed write. The recount costs what Recon already does on its first run
> after startup: a truncate and one row per non empty size bin, on top of the
> pass over the container list that every run makes. A reader may see an empty
> table for that moment, as after a restart.
> Covered by {{TestContainerSizeCountTask.testTaskKeepsRunningAfterFailedRun}}
> (the first write fails, the container must still reach the table) and
> {{TestContainerSizeCountTask.testCountsRecoverAfterFailedWrite}} (after a
> partial write and one more run only the new bin may be counted). Without the
> change the first times out and the second sees both bins. With it the size
> count, endpoint, SCM sync helper and SCM facade tests pass, checkstyle is
> clean, and the attached test no longer shows the bug.
> The same death pattern in {{PipelineSyncTask}} is a separate issue with its
> own patch; the two patches touch different files and apply together.
> Found by code review of the datanode container report and Recon container
> tracking paths, as part of the TLA+ verification effort under HDDS-15926, on
> commit 916aa1d9949cbb10f2750a40c8704340893131a2. Checked against HDDS issues
> and apache/ozone pull requests for duplicates before filing. Related but not
> duplicates: HDDS-12227 (resolved, lowered this log line to DEBUG), HDDS-10293
> (resolved, negative container size, caught per container), HDDS-7632
> (resolved, added the task). The attached patch is a proposal for review.
> Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
