shuan1026 opened a new pull request, #11989: URL: https://github.com/apache/ozone/pull/11989
## What changes were proposed in this pull request? `ExportJobManager.executeExport` creates the job directory with `Files.createDirectories` and only then removes the job from `jobQueue`. When directory creation throws an `IOException` (wrong permissions, a read-only remount, or a regular file in place of the export directory, which is also what happens when Recon could not create the directory at startup), the catch marks the job `FAILED` and deletes its files, but the job stays in `jobQueue`; the `finally` only clears `runningTasks`. `submitJob` rejects a submission once `jobQueue.size()` reaches `ozone.recon.export.max.jobs.total` (default 4), and its duplicate check ignores `FAILED` jobs, so every such failure keeps one slot. After four failures every new export gets HTTP 429 `Export queue is full (max 4 jobs)` although nothing is queued or running, and retrying never helps once the fault is fixed. Because `jobQueue` is insertion ordered, the stale entries also inflate `getQueuePosition` of later queued jobs. The only ways out are deleting each `FAILED` row in the UI or restarting Recon. This patch moves the removal from `jobQueue` to the start of `executeExport`, ahead of `Files.createDirectories`, so the job leaves the queue before anything in the worker can fail. The success path is unchanged: the job still becomes `RUNNING` after its directory exists, and `cancelJob`'s "no-op for COMPLETED/FAILED jobs already off the queue" now holds for every `FAILED` job. Counting only `QUEUED`/`RUNNING` jobs in `submitJob` was not chosen because it changes what the limit counts and leaves the queue positions inflated. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-17391 ## How was this patch tested? - New `TestExportJobManager#jobDirectoryCreationFailureFreesQueueSlot`: with a regular file in place of the export directory and `ozone.recon.export.max.jobs.total=1`, the first export ends `FAILED` with the export path in its error message and must have no queue position; after the directory is restored, a second export is accepted and completes. It fails with `expected: 0 but was: 1` before the change (the second submit would otherwise be rejected with `Export queue is full (max 1 jobs)`) and passes after it. - CI: https://github.com/shuan1026/ozone/actions/runs/42613119322 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
