Siyao Meng created HDDS-17393:
---------------------------------
Summary: Description of ozone.recon.export.max.jobs.total says it
counts the executing export job, but only waiting jobs are counted
Key: HDDS-17393
URL: https://issues.apache.org/jira/browse/HDDS-17393
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-5-describe-export-job-limit-as-waiting-jobs.patch,
TestBugMC5ExportLimitExcludesRunningJob.java
h3. Mechanism
The description of {{ozone.recon.export.max.jobs.total}} in
{{ozone-default.xml}} and the Javadoc of
{{ReconServerConfigKeys.OZONE_RECON_EXPORT_MAX_JOBS_TOTAL}} say it is the
maximum number of export jobs "(waiting + executing)".
{{ExportJobManager.submitJob}} compares the limit with {{jobQueue.size()}}, and
{{ExportJobManager.executeExport}} removes a job from {{jobQueue}} when it
starts, so the running job is not counted. The limit applies to waiting jobs
only; one running job can exist on top of it. Until HDDS-17391 is fixed, a job
whose directory creation failed also stays in {{jobQueue}} and counts against
the limit, although it is neither waiting nor executing.
{{TestExportJobManager#submitQueueFullThrows}} asserts the code's behaviour:
with the limit at 1 it accepts one running and one waiting job and rejects the
third.
h3. Trigger
# One export is running.
# Further exports for other states are submitted.
h3. Impact
With the default of 4, five jobs are accepted (one RUNNING, four QUEUED) and
only the sixth gets HTTP 429. Operators reading the description expect four.
This is reachable through the REST API; the UI offers five states and allows
one live job per state, so a UI user does not reach the limit at the default.
There is no other effect: a single worker runs the jobs, so the extra job only
waits.
h3. Reproduction
PASS on the unmodified source, deterministic.
[^TestBugMC5ExportLimitExcludesRunningJob.java] ({{hadoop-ozone/recon}}) holds
one job RUNNING on a blocking mocked cursor, submits four more, and checks that
5 jobs are QUEUED or RUNNING and that the sixth is rejected with "Export queue
is full (max 4 jobs)".
h3. Suggested fix
Correct the description rather than the code. Counting the executing job would
lower the effective limit for every deployment (from 5 active jobs to 4 at the
default), change the behaviour of the 2.2 releases and contradict the existing
unit test, for no operational gain. If the description is considered the
contract instead, the alternative is to count QUEUED and RUNNING jobs in
{{submitJob}} and adjust {{submitQueueFullThrows}}.
h3. Patch
[^MC-5-describe-export-job-limit-as-waiting-jobs.patch], against
fb2d0310a9c6ea37c72cefb5cc1d5b4db35a7ab1. It also applies to master at
9c3e5477ffb.
Documentation only: the first sentence of the description becomes "Maximum
number of export jobs waiting in the queue at once, not counting the job that
is executing.", and the Javadoc line matches. No code or test change;
{{submitQueueFullThrows}} already covers the behaviour. The reproduction passes
unchanged with the patch.
Found by TLA+ model checking and code review of the Recon unhealthy container
export job manager under HDDS-15926, on commit
fb2d0310a9c6ea37c72cefb5cc1d5b4db35a7ab1. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
