[
https://issues.apache.org/jira/browse/HDDS-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16440:
------------------------------
Description:
h3. Finding
A missing null check turns a tolerated transient Ratis submit failure
(ServiceException producing a null response) into an uncaught NPE dereferencing
resp.getSuccess() in the DirectoryDeletingService background task. Masked by
BackgroundService.run()'s unconditional catch(Throwable), which logs and
reschedules next interval, and nothing durable is written; absent that guard
the NPE would abort the run and risk stalling the periodic deletion loop
(liveness, no data corruption).
h3. Classification
* Verdict: MASKED
* Severity: Medium
* Source: Specula TLA+ model checking and confirmation debate, finding CR-5
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: om-fso-recursive-deletion
Guidance:
campaigns/ozone-9fbf9ee/targets/005-om-fso-recursive-deletion/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=high --keep-original --max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"om-fso-recursive-deletion|apache/ozone|Java|Use the target-specific
.prompt-extra.md"
{code}
Discovered under HDDS-16429 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
h3. Steps to reproduce
# OM is leader and the DirectoryDeletingService background task runs with
pending deleted directories, so enumeration builds a non-empty purge-path list.
# At batch flush, the OM Ratis submit fails transiently (leader step-down /
not-leader / timeout / interrupt during OM shutdown).
OzoneManagerRatisServer.submitRequest throws ServiceException.
# submitPurgeRequest catches the ServiceException, logs "PurgePaths request
failed. Will retry at next run.", and returns null.
# submitPurgePathsWithBatching then evaluates resp.getSuccess() on that null
response, throwing NullPointerException.
# The NPE is a RuntimeException that none of the enclosing catch blocks (which
catch only IOException / ExecutionException / InterruptedException) handle, so
it propagates out of DirDeletingTask.call() and aborts the whole
directory-deleting run instead of the intended "skip, retry next run".
Reproduced in a unit test that spies the service, stubs submitRequest to throw
ServiceException at its real seam, and drives the package-visible
optimizeDirDeletesAndSubmitRequest with a non-empty purge list. Without the fix
the call throws NullPointerException.
h3. Root cause
submitPurgeRequest (DirectoryDeletingService) returns null on ServiceException
(a normal, expected transient submit failure it explicitly catches and logs).
Its caller submitPurgePathsWithBatching dereferences the response
unconditionally with resp.getSuccess() at both the mid-loop batch flush and the
final batch flush, so a null response causes a NullPointerException. The
intended behavior (return an empty list, retry next run) is defeated: the run
dies instead of retrying.
h3. Suggested fix
Null-check the response at both submitPurgePathsWithBatching flush sites: treat
a null response the same as a failed submit (stop batching and return the empty
list), so a transient submit failure ends the run gracefully and the pending
directories are retried on the next run. See [^HDDS-16440.001.patch]. The patch
adds testPurgeDirectoriesSubmitFailureIsTolerated, which reproduces the NPE
without the fix and passes with it.
This finding is masked in production: the aborted run's un-purged directories
remain in the deleted-directory table and are retried on the next scheduled
run, so the effect is a delayed reclamation of one run rather than permanent
loss. The fix removes the abort so the retry path is taken directly.
Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
test, and review before merge.
was:
h3. Finding
A missing null check turns a tolerated transient Ratis submit failure
(ServiceException producing a null response) into an uncaught NPE dereferencing
resp.getSuccess() in the DirectoryDeletingService background task. Masked by
BackgroundService.run()'s unconditional catch(Throwable), which logs and
reschedules next interval, and nothing durable is written; absent that guard
the NPE would abort the run and risk stalling the periodic deletion loop
(liveness, no data corruption).
h3. Classification
* Verdict: MASKED
* Severity: Medium
* Source: Specula TLA+ model checking and confirmation debate, finding CR-5
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: om-fso-recursive-deletion
Guidance:
campaigns/ozone-9fbf9ee/targets/005-om-fso-recursive-deletion/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=high --keep-original --max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"om-fso-recursive-deletion|apache/ozone|Java|Use the target-specific
.prompt-extra.md"
{code}
Discovered under HDDS-16429 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
> DirectoryDeletingService can NPE on a transient Ratis submit failure (masked
> by BackgroundService catch)
> --------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16440
> URL: https://issues.apache.org/jira/browse/HDDS-16440
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: HDDS-16440.001.patch
>
>
> h3. Finding
> A missing null check turns a tolerated transient Ratis submit failure
> (ServiceException producing a null response) into an uncaught NPE
> dereferencing resp.getSuccess() in the DirectoryDeletingService background
> task. Masked by BackgroundService.run()'s unconditional catch(Throwable),
> which logs and reschedules next interval, and nothing durable is written;
> absent that guard the NPE would abort the run and risk stalling the periodic
> deletion loop (liveness, no data corruption).
> h3. Classification
> * Verdict: MASKED
> * Severity: Medium
> * Source: Specula TLA+ model checking and confirmation debate, finding CR-5
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: om-fso-recursive-deletion
> Guidance:
> campaigns/ozone-9fbf9ee/targets/005-om-fso-recursive-deletion/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=high --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "om-fso-recursive-deletion|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> Discovered under HDDS-16429 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> # OM is leader and the DirectoryDeletingService background task runs with
> pending deleted directories, so enumeration builds a non-empty purge-path
> list.
> # At batch flush, the OM Ratis submit fails transiently (leader step-down /
> not-leader / timeout / interrupt during OM shutdown).
> OzoneManagerRatisServer.submitRequest throws ServiceException.
> # submitPurgeRequest catches the ServiceException, logs "PurgePaths request
> failed. Will retry at next run.", and returns null.
> # submitPurgePathsWithBatching then evaluates resp.getSuccess() on that null
> response, throwing NullPointerException.
> # The NPE is a RuntimeException that none of the enclosing catch blocks
> (which catch only IOException / ExecutionException / InterruptedException)
> handle, so it propagates out of DirDeletingTask.call() and aborts the whole
> directory-deleting run instead of the intended "skip, retry next run".
> Reproduced in a unit test that spies the service, stubs submitRequest to
> throw ServiceException at its real seam, and drives the package-visible
> optimizeDirDeletesAndSubmitRequest with a non-empty purge list. Without the
> fix the call throws NullPointerException.
> h3. Root cause
> submitPurgeRequest (DirectoryDeletingService) returns null on
> ServiceException (a normal, expected transient submit failure it explicitly
> catches and logs). Its caller submitPurgePathsWithBatching dereferences the
> response unconditionally with resp.getSuccess() at both the mid-loop batch
> flush and the final batch flush, so a null response causes a
> NullPointerException. The intended behavior (return an empty list, retry next
> run) is defeated: the run dies instead of retrying.
> h3. Suggested fix
> Null-check the response at both submitPurgePathsWithBatching flush sites:
> treat a null response the same as a failed submit (stop batching and return
> the empty list), so a transient submit failure ends the run gracefully and
> the pending directories are retried on the next run. See
> [^HDDS-16440.001.patch]. The patch adds
> testPurgeDirectoriesSubmitFailureIsTolerated, which reproduces the NPE
> without the fix and passes with it.
> This finding is masked in production: the aborted run's un-purged directories
> remain in the deleted-directory table and are retried on the next scheduled
> run, so the effect is a delayed reclamation of one run rather than permanent
> loss. The fix removes the abort so the retry path is taken directly.
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
