[
https://issues.apache.org/jira/browse/HDDS-17388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17388:
------------------------------
Attachment: TestBugCR7DuplicateExportStatus.java
> Recon unhealthy container export returns 429 Too Many Requests for a
> duplicate export, which retrying never clears
> ------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17388
> URL: https://issues.apache.org/jira/browse/HDDS-17388
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.2.0, 2.2.1
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: CR-7-return-409-for-duplicate-export.patch,
> TestBugCR7DuplicateExportStatus.java
>
>
> h3. Mechanism
> {{ExportJobManager.submitJob}} throws {{IllegalStateException}} in two cases:
> an export for the requested state is already QUEUED, RUNNING or COMPLETED, or
> the export queue is full. {{ContainerEndpoint.startExport}} maps every
> {{IllegalStateException}} to {{429 Too Many Requests}} with {{"error": "Too
> Many Requests"}}. The published {{ozone-default.xml}} ties this 429 to the
> queue limit: {{ozone.recon.export.max.jobs.total}} says "Submissions beyond
> this limit are rejected with HTTP 429." The in-tree {{ReconApi.md}} lists
> only 400 and 429 for {{POST /api/v1/containers/unhealthy/export}} and
> describes 429 as "the export queue is full; retry later". A queue full
> condition clears by waiting, a duplicate does not: it lasts until the
> existing export is deleted, the job fails or Recon restarts (jobs are kept in
> memory only). {{downloadExport}} on the same resource already uses {{409
> Conflict}} for its "not completed yet" conflict. Its 429 for the per job
> download limit ({{ozone.recon.export.max.downloads}}) is a separate,
> documented case and is out of scope here.
> h3. Trigger
> # {{POST /api/v1/containers/unhealthy/export?state=MISSING}}, wait until the
> job is COMPLETED.
> # Repeat the same request. Every repetition gets 429.
> h3. Impact
> A client that follows the documented meaning of 429 retries a request that
> can never succeed. The Recon UI is not affected in practice: it blocks a
> duplicate itself when its job list is current, shows the server's message for
> any error status and does not retry (only its toast duration depends on 429).
> The futile retry case is argued from the documented contract, not run with a
> real client.
> h3. Reproduction
> [^TestBugCR7DuplicateExportStatus.java] ({{hadoop-ozone/recon}}, package
> {{org.apache.hadoop.ozone.recon.api}}) calls the real
> {{ContainerEndpoint.startExport}} over a real {{ExportJobManager}}, with only
> the DB collaborator mocked. It exports MISSING, waits for COMPLETED and
> repeats the request three times; each gets status 429 with error "Too Many
> Requests" and the message "An export for state MISSING already exists. Please
> delete the existing export from the Completed Exports table before starting a
> new one." The test passes on unmodified source, deterministic; a passing test
> means the defect is present.
> h3. Patch
> [^CR-7-return-409-for-duplicate-export.patch], against
> fb2d0310a9c6ea37c72cefb5cc1d5b4db35a7ab1. It also applies to master at
> 9c3e5477ffb.
> The duplicate check now throws {{ExportJobManager.DuplicateExportException}},
> a package private subclass of {{IllegalStateException}} so existing callers
> are unaffected, and {{startExport}} answers it with {{409 Conflict}} and
> {{"error": "Conflict"}}; the message is unchanged. Queue full keeps 429 with
> the same body. The Recon UI needs no change: a 409 shows the same message.
> Covered by
> {{startExportReturnsConflictForDuplicateStateAndTooManyRequestsWhenQueueFull}}
> in the existing {{TestExportJobManager}}, which checks 409 for a duplicate
> and 429 for a full queue through the endpoint. Without the change it fails
> with {{expected: 409 but was: 429}}. With it {{TestExportJobManager}},
> {{TestExportJob}} and {{TestContainerEndpoint}} (54 tests) pass and
> checkstyle is clean. With the patch the reproduction fails at its first
> retry, which gets 409. The in-tree {{ReconApi.md}} and {{recon-api.yaml}}
> still list only 400 and 429 for this call and are not changed, since that
> docs tree is being retired. The {{ozone-default.xml}} text stays correct,
> because 429 now means only that the queue limit was hit.
> Found by code review of the Recon unhealthy container export job manager, as
> part of the TLA+ verification effort under HDDS-15926, on commit
> fb2d0310a9c6ea37c72cefb5cc1d5b4db35a7ab1. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
