[
https://issues.apache.org/jira/browse/HDDS-17011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andy Huang reassigned HDDS-17011:
---------------------------------
Assignee: Andy Huang
> Datanode automatic small SST file compaction leaks a native
> CompactRangeOptions on every range compaction
> ---------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17011
> URL: https://issues.apache.org/jira/browse/HDDS-17011
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Andy Huang
> Priority: Minor
> Attachments:
> CR-5-close-compact-range-options-in-auto-compaction.patch,
> TestBugCR5CompactRangeOptionsLeak.java
>
>
> h3. Mechanism
> {{DatanodeStoreSchemaThreeImpl.compactionIfNeeded}} creates a
> {{ManagedCompactRangeOptions}} for each column family level that has more
> small SST files than the threshold and passes it to
> {{RocksDatabase.compactRange}}. Neither closes it. RocksJava has no finalizer
> or cleaner for {{CompactRangeOptions}} (checked in rocksdbjni 10.10.1.1), so
> its native memory is never freed. The datanode leak detector reports it as
> "ManagedCompactRangeOptions is not closed properly" once the Java object is
> collected.
> h3. Trigger
> {{hdds.datanode.rocksdb.auto-compaction-small-sst-file}} is enabled (the
> default) and a column family level of a per-disk DB has more than
> {{hdds.datanode.rocksdb.auto-compaction-small-sst-file-num-threshold}}
> (default 512) SST files below the size threshold (default 1MB). Each
> scheduled {{HddsVolume.compactDb}} run that compacts such a level leaks one
> object per level.
> h3. Impact
> A small native leak for the life of the datanode process, plus leak detector
> warnings. A {{CompactRangeOptions}} is a small fixed size native struct; at
> most one leaks per column family level over the threshold per scheduled run,
> and the run happens every 120 minutes per volume by default
> ({{hdds.datanode.rocksdb.auto-compaction-small-sst-file.interval.minutes}}),
> so growth is slow. The byte count was not measured. No functional impact.
> h3. Reproduction
> PASS, unmodified source. [^TestBugCR5CompactRangeOptionsLeak.java]
> ({{hadoop-hdds/container-service}}) initializes a real schema V3
> {{HddsVolume}} with the file count threshold set to 1, writes and flushes
> blocks in three rounds, calls {{HddsVolume.compactDb}} after each, and then
> runs GC until the leak detector has reported one leaked
> {{ManagedCompactRangeOptions}} per compaction (4 of 4). Each report's
> allocation stack points at
> {{DatanodeStoreSchemaThreeImpl.compactionIfNeeded}} called from
> {{HddsVolume.compactDb}}. A passing test means the defect is present.
> h3. Patch
> [^CR-5-close-compact-range-options-in-auto-compaction.patch], against
> cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at
> 2536fe9cbc3.
> The options are allocated in try with resources around the existing
> compaction call, so they are closed after each compaction and also when it
> throws. Covered by {{testAutoCompactionClosesCompactRangeOptions}} in the
> existing {{TestKeyValueContainer}}, which does not depend on GC: it captures
> the options passed to {{compactRange}} through spies on a real store and
> asserts that none still owns its native handle after {{compactionIfNeeded}}.
> Without the change it fails; with it {{TestKeyValueContainer}},
> {{TestHddsVolume}} and {{TestDatanodeStoreCache}} (233 tests) pass and
> checkstyle is clean. With the patch the reproduction fails because no leak is
> reported.
> The same unclosed allocation exists in {{RocksDBManualCompaction}} ({{ozone
> repair ldb compact}}), which runs in a short lived CLI process; it could be
> fixed in a follow up.
> Found by code review of the datanode volume and per-disk RocksDB lifecycle
> paths, as part of the TLA+ verification effort under HDDS-15926, on commit
> cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
