Siyao Meng created HDDS-17732:
---------------------------------
Summary: SnapDiffJobStore.close stops at the first RocksDB error
and leaves the job's temporary column families in the snapshot diff DB until OM
restart
Key: HDDS-17732
URL: https://issues.apache.org/jira/browse/HDDS-17732
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-13-best-effort-snap-diff-job-store-close.patch,
TestBugCR13CloseAfterRocksDbFailure.java
h3. Mechanism
{{SnapDiffJobStore.close}} runs its steps in sequence without try/finally:
{{flushWrites}}, close the write batch and write options, drop every tracked
temporary column family through {{dropAndClose}}, close the column family
options. Any exception ends it:
* A failed {{flushWrites}} leaves the batch as it was, so after a stage flush
failed, the flush in {{close}} repeats the same write, fails again, and no
column family is dropped and no native object is closed.
* {{dropAndClose}} removes the handle from the tracking list before
{{SnapshotUtils.dropColumnFamilyHandle}}, which turns a {{RocksDBException}}
into a {{RuntimeException}}. That handle is then never closed and the remaining
column families are never dropped.
Nothing in a running OM drops them later; only
{{OmSnapshotManager.dropUnknownColumnFamilies}} at OM startup does. A dropped
column family keeps its files on disk while a handle to it is open, so the
unclosed handle also keeps that space until the process exits.
h3. Trigger
A RocksDB write or column family drop fails on the snapshot diff DB while an
optimized snapshot diff job runs, which in practice means a full or failing
disk under {{ozone.om.snapshot.diff.db.dir}}. Only the optimized snapshot diff
path ({{ozone.om.snapshot.diff.optimized.enabled}}, default false) uses
{{SnapDiffJobStore}}, and the class is not in any release yet. The job store is
closed by the try with resources in
{{SnapshotDiffManager.generateSnapshotDiffReport}}. Each job creates nine
temporary column families (seven for OBS buckets), plus up to four more when
its in memory sets spill, all sized by the diff, so the job that fills the
volume is the one whose data stays.
If the error happens before the report is complete, the job is marked FAILED as
usual. The store is closed after {{updateJobStatusToDone}}, and some families
are still open then (on an OBS bucket the four classified families and the
dependency node family). If one of those drops fails, the job stays DONE, which
is correct because its report is complete; the existing caller then calls
{{updateJobStatusToFailed}}, which throws an {{IllegalStateException}} for the
DONE job before the close error is logged. That caller behaviour is not changed
here.
h3. Impact
* The temporary column families of the failed job, with their data, stay in the
snapshot diff DB until the OM restarts. When the snapshot diff DB is on its own
volume, the OM keeps running and the space is not released, so later diff jobs
can keep failing for lack of space.
* One {{ColumnFamilyHandle}} (drop path), or the write batch, write options,
column family options and all handles (flush path) are never closed (native
memory).
* Job state is not corrupted: a FAILED job can be resubmitted under a new job
ID, and a DONE job keeps its complete report.
The legacy path, {{SnapshotDiffManager.generateLegacySnapshotDiffReport}},
drops its three temporary column families with the same sequential pattern,
which is in every release from 1.4.0 to 2.2.2; it is handled separately.
h3. Reproduction
[^TestBugCR13CloseAfterRocksDbFailure.java] ({{hadoop-ozone/ozone-manager}}): a
real RocksDB and the store's public calls; the RocksDB error is injected with a
Mockito spy on the {{RocksDB}} object (write fails; or the first drop is
applied and then fails, as a full disk does when RocksDB cannot persist its
OPTIONS file). On d3b75c75bf9ad475551202477c3a21a3f4b968b7 both tests fail:
after the failed flush all nine temporary column families of the job remain,
after the failed drop the other eight remain and the handle of the failed drop
is still open. With the patch both pass. A real out of space error was
reproduced only by the original investigation (on a 16 MB disk image), not with
the attached test.
h3. Patch
[^CR-13-best-effort-snap-diff-job-store-close.patch], against
d3b75c75bf9ad475551202477c3a21a3f4b968b7. It also applies to master at
bbc8505e00d.
{{close}} now attempts every step even if an earlier one fails: the flush
failure no longer skips the rest, every tracked column family is attempted (the
first failure is thrown with later ones added as suppressed), and the write
batch, write options and column family options are closed in finally blocks.
{{dropAndClose}} closes the handle in a finally block. A store that closes
without error behaves as before.
Covered by two tests in the existing {{TestMergeJoinSnapDiffWriter}}, which
already drives {{SnapDiffJobStore}} on a real RocksDB: a failed flush and a
failed drop, each asserting that {{close}} reports the error and no column
family of the job is left. Without the change both fail on the column family
check: the flush test finds all nine families of the job, the drop test the
other eight. With it {{TestMergeJoinSnapDiffWriter}},
{{TestDagDiffSequentialReader}}, {{TestFullDiffSequentialReader}},
{{TestSnapDiffPathResolver}}, {{TestSnapDiffDependencyGraph}} and
{{TestSnapshotDiffManager}} (139 tests) pass and checkstyle is clean.
Found by code review of the OM optimized snapshot diff paths, as part of the
TLA+ verification effort under HDDS-15926, on commit
d3b75c75bf9ad475551202477c3a21a3f4b968b7. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
