[
https://issues.apache.org/jira/browse/HDDS-17391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17391:
------------------------------
Attachment: MC-2-remove-failed-export-job-from-queue.patch
> Recon export job that fails to create its working directory keeps its queue
> slot, so later exports are rejected as "queue is full"
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17391
> URL: https://issues.apache.org/jira/browse/HDDS-17391
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.2.0, 2.2.1
> Reporter: Siyao Meng
> Priority: Major
> Attachments: MC-2-remove-failed-export-job-from-queue.patch,
> TestBugMC2JobDirFailureQueueSlot.java
>
>
> h3. Mechanism
> {{ExportJobManager.executeExport}} creates the job directory with
> {{Files.createDirectories}} before it removes the job from {{jobQueue}}. When
> that call throws an {{IOException}}, the catch marks the job FAILED and
> deletes its files, but the job is never removed from {{jobQueue}}; the
> {{finally}} only clears {{runningTasks}}. {{ExportJobManager.submitJob}}
> rejects a submission when {{jobQueue.size()}} reaches
> {{ozone.recon.export.max.jobs.total}} (default 4), and its duplicate check
> ignores FAILED jobs, so every such failure, including a retry of the same
> state, keeps one slot.
> The comment in {{ExportJobManager.cancelJob}} ("no-op for COMPLETED/FAILED
> jobs already off the queue") shows that FAILED jobs are not meant to stay in
> the queue.
> h3. Trigger
> # The export directory becomes temporarily unusable, so creating a job
> directory fails (for example wrong permissions or a read only remount of the
> export directory; my test uses a regular file in place of the directory). If
> Recon cannot create the export directory at startup, {{ExportJobManager}}
> only logs the error, and every export then fails at this step while the cause
> persists.
> # Exports are submitted while the fault lasts. Each one ends FAILED and keeps
> its slot.
> # The fault is fixed.
> h3. Impact
> * After 4 such failures (the default limit), every new export gets HTTP 429
> "Export queue is full (max 4 jobs). Please try again later." although no job
> is queued or running. Retrying never helps. Each FAILED row has to be deleted
> in the UI (Completed Exports, Delete), which frees one slot each, or Recon
> restarted.
> * Fewer stale entries still reduce the effective limit, and since
> {{jobQueue}} is insertion ordered, a later queued job reports a queue
> position inflated by the stale entries ahead of it (follows from the code,
> not run).
> * No data is affected; only the export feature.
> h3. Reproduction
> PASS on the unmodified source, deterministic.
> [^TestBugMC2JobDirFailureQueueSlot.java] ({{hadoop-ozone/recon}}) uses only
> public {{ExportJobManager}} methods with the DB layer mocked as in
> {{TestExportJobManager}}, and the default limit. With a regular file in place
> of the export directory, four exports for different states end FAILED and
> report queue positions 1 to 4. After the directory is restored and nothing is
> queued or running, a new export is rejected with "queue is full"; after one
> FAILED job is deleted, an export is accepted and completes. A passing test
> means the defect is present.
> h3. Patch
> [^MC-2-remove-failed-export-job-from-queue.patch], against
> fb2d0310a9c6ea37c72cefb5cc1d5b4db35a7ab1. It also applies to master at
> 9c3e5477ffb.
> The patch moves the removal from {{jobQueue}} ahead of
> {{Files.createDirectories}}, so the job leaves the queue before anything in
> {{executeExport}} can fail. This fixes both the admission slot and the stale
> queue position, and leaves the success path unchanged. Counting only QUEUED
> or RUNNING jobs in {{submitJob}} would fix admission alone and change what
> the limit counts.
> Covered by the new
> {{TestExportJobManager#jobDirectoryCreationFailureFreesQueueSlot}}, which
> also checks that the first job failed on the job directory, and fails without
> the change with "Export queue is full (max 1 jobs)". With the change
> {{TestExportJobManager}}, {{TestExportJob}} and {{TestContainerEndpoint}}
> pass and checkstyle is clean. With the patch the reproduction fails at its
> first queue position assertion, because the FAILED jobs are no longer in the
> queue.
> Found by TLA+ model checking and code review of the Recon unhealthy container
> export job manager under HDDS-15926, on commit
> fb2d0310a9c6ea37c72cefb5cc1d5b4db35a7ab1. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
