bbejeck opened a new pull request, #22378:
URL: https://github.com/apache/kafka/pull/22378
Fixes two close-path native memory leaks in `RocksDBStore` that caused ~15
GB RSS / OOM in EOSv2 soak after 2.5 days. Native heap profiles attribute >60%
of allocations to
`Java_org_rocksdb_ColumnFamilyOptions_newColumnFamilyOptions` →
`BlockBasedTableFactory::InitializeOptions` →
`LRUCacheOptions::MakeSharedCache`.
### Issue 1 — leaked `ColumnFamilyOptions` (KAFKA-20456 follow-up)
`createOffsetsCFOptions()` returned a `new ColumnFamilyOptions()` that was
passed to a `ColumnFamilyDescriptor` and then dropped — never stored on a
field, never closed. Each construction auto-allocates a
default `BlockBasedTableFactory` + 8 MB `LRUCache` on the JNI side;
segmented/windowed stores amplify the leak per segment per task.
**Fix:** store the options on a `RocksDBStore.offsetsCfOptions` field and
close it in `close()` / `closeNativeResources()`. Also `setNoBlockCache(true)`
since the offsets CF doesn't need a block cache.
### Issue 2 — CF handles leak on close-path exceptions (KIP-1035)
`AbstractColumnFamilyAccessor.close()` writes a `closedState` marker to
the offsets CF. If that write throws (EOSv2 fencing, unclean shutdown — already
noted in the existing code comment), the subsequent
`offsetColumnFamilyHandle.close()` was skipped.
`SingleColumnFamilyAccessor.close()` and `DualColumnFamilyAccessor.close()` had
the same non-`finally` ordering, so the data / old / new CF handles leaked
too. `RocksDBStore.close()` swallows the resulting `RocksDBException`,
making the leak silent.
**Fix:** `try/finally` in all three `close()` methods so every
`ColumnFamilyHandle.close()` runs even when an earlier write or super-close
throws
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
