Siyao Meng created HDDS-17391:
---------------------------------
Summary: Recon export job that fails to create its working
directory keeps its queue slot, so later exports are rejected as "queue is full"
Key: HDDS-17391
URL: https://issues.apache.org/jira/browse/HDDS-17391
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-2-remove-failed-export-job-from-queue.patch,
TestBugMC2JobDirFailureQueueSlot.java
h3. Mechanism
{{ExportJobManager.executeExport}} creates the job directory with
{{Files.createDirectories}} before it removes the job from {{jobQueue}}. When
that call throws an {{IOException}}, the catch marks the job FAILED and deletes
its files, but the job is never removed from {{jobQueue}}; the {{finally}} only
clears {{runningTasks}}. {{ExportJobManager.submitJob}} rejects a submission
when {{jobQueue.size()}} reaches {{ozone.recon.export.max.jobs.total}} (default
4), and its duplicate check ignores FAILED jobs, so every such failure,
including a retry of the same state, keeps one slot.
The comment in {{ExportJobManager.cancelJob}} ("no-op for COMPLETED/FAILED jobs
already off the queue") shows that FAILED jobs are not meant to stay in the
queue.
h3. Trigger
# The export directory becomes temporarily unusable, so creating a job
directory fails (for example wrong permissions or a read only remount of the
export directory; my test uses a regular file in place of the directory). If
Recon cannot create the export directory at startup, {{ExportJobManager}} only
logs the error, and every export then fails at this step while the cause
persists.
# Exports are submitted while the fault lasts. Each one ends FAILED and keeps
its slot.
# The fault is fixed.
h3. Impact
* After 4 such failures (the default limit), every new export gets HTTP 429
"Export queue is full (max 4 jobs). Please try again later." although no job is
queued or running. Retrying never helps. Each FAILED row has to be deleted in
the UI (Completed Exports, Delete), which frees one slot each, or Recon
restarted.
* Fewer stale entries still reduce the effective limit, and since {{jobQueue}}
is insertion ordered, a later queued job reports a queue position inflated by
the stale entries ahead of it (follows from the code, not run).
* No data is affected; only the export feature.
h3. Reproduction
PASS on the unmodified source, deterministic.
[^TestBugMC2JobDirFailureQueueSlot.java] ({{hadoop-ozone/recon}}) uses only
public {{ExportJobManager}} methods with the DB layer mocked as in
{{TestExportJobManager}}, and the default limit. With a regular file in place
of the export directory, four exports for different states end FAILED and
report queue positions 1 to 4. After the directory is restored and nothing is
queued or running, a new export is rejected with "queue is full"; after one
FAILED job is deleted, an export is accepted and completes. A passing test
means the defect is present.
h3. Patch
[^MC-2-remove-failed-export-job-from-queue.patch], against
fb2d0310a9c6ea37c72cefb5cc1d5b4db35a7ab1. It also applies to master at
9c3e5477ffb.
The patch moves the removal from {{jobQueue}} ahead of
{{Files.createDirectories}}, so the job leaves the queue before anything in
{{executeExport}} can fail. This fixes both the admission slot and the stale
queue position, and leaves the success path unchanged. Counting only QUEUED or
RUNNING jobs in {{submitJob}} would fix admission alone and change what the
limit counts.
Covered by the new
{{TestExportJobManager#jobDirectoryCreationFailureFreesQueueSlot}}, which also
checks that the first job failed on the job directory, and fails without the
change with "Export queue is full (max 1 jobs)". With the change
{{TestExportJobManager}}, {{TestExportJob}} and {{TestContainerEndpoint}} pass
and checkstyle is clean. With the patch the reproduction fails at its first
queue position assertion, because the FAILED jobs are no longer in the queue.
Found by TLA+ model checking and code review of the Recon unhealthy container
export job manager under HDDS-15926, on commit
fb2d0310a9c6ea37c72cefb5cc1d5b4db35a7ab1. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
