[
https://issues.apache.org/jira/browse/KAFKA-20688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lucy Liu updated KAFKA-20688:
-----------------------------
Description:
`RocksDBTimestampedStoreWithHeaders.openRocksDB(...)` allocates a `new
ColumnFamilyOptions()` solely to construct an `Options` for a
`RocksDB.listColumnFamilies(...)` probe call. The temporary
`ColumnFamilyOptions` instance is not referenced anywhere outside the
try-with-resources block, and `Options.close()` does not cascade-close the
inner `ColumnFamilyOptions` in the RocksDB JNI contract. The instance is
unreachable from any close path, leaking one native `LRUCache` block cache per
`openRocksDB` invocation. Each new windowed-store segment open compounds the
leak.
*Affected code*
[https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeaders.java#L75-L80]
{code:java}
final List<byte[]> existingCFs;
try (final Options options = new Options(dbOptions, new ColumnFamilyOptions()))
{
existingCFs = RocksDB.listColumnFamilies(options, dbDir.getAbsolutePath());
} catch (final RocksDBException e) {
throw new ProcessorStateException("Error listing column families for store "
+ name, e);
} {code}
This path is only reached when header-aware state stores are enabled
(KIP-1271). Workloads that do not enable that feature are unaffected.
was:
`RocksDBTimestampedStoreWithHeaders.openRocksDB(...)` allocates a `new
ColumnFamilyOptions()` solely to construct an `Options` for a
`RocksDB.listColumnFamilies(...)` probe call. The temporary
`ColumnFamilyOptions` instance is not referenced anywhere outside the
try-with-resources block, and `Options.close()` does not cascade-close the
inner `ColumnFamilyOptions` in the RocksDB JNI contract. The instance is
unreachable from any close path, leaking one native `LRUCache` block cache per
`openRocksDB` invocation. Each new windowed-store segment open compounds the
leak.
## Affected code
https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeaders.java#L75-L80
```
final List<byte[]> existingCFs;
try (final Options options = new Options(dbOptions, new
ColumnFamilyOptions())) {
existingCFs = RocksDB.listColumnFamilies(options,
dbDir.getAbsolutePath());
} catch (final RocksDBException e) {
throw new ProcessorStateException("Error listing column families for
store " + name, e);
}
```
This path is only reached when header-aware state stores are enabled
(KIP-1271). Workloads that do not enable that feature are unaffected.
> Memory leak in RocksDBTimestampedStoreWithHeaders.openRocksDB when probing
> column families
> ------------------------------------------------------------------------------------------
>
> Key: KAFKA-20688
> URL: https://issues.apache.org/jira/browse/KAFKA-20688
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 4.3.0
> Reporter: Lucy Liu
> Priority: Blocker
> Fix For: 4.3.1
>
>
> `RocksDBTimestampedStoreWithHeaders.openRocksDB(...)` allocates a `new
> ColumnFamilyOptions()` solely to construct an `Options` for a
> `RocksDB.listColumnFamilies(...)` probe call. The temporary
> `ColumnFamilyOptions` instance is not referenced anywhere outside the
> try-with-resources block, and `Options.close()` does not cascade-close the
> inner `ColumnFamilyOptions` in the RocksDB JNI contract. The instance is
> unreachable from any close path, leaking one native `LRUCache` block cache
> per `openRocksDB` invocation. Each new windowed-store segment open compounds
> the leak.
>
> *Affected code*
> [https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeaders.java#L75-L80]
> {code:java}
> final List<byte[]> existingCFs;
> try (final Options options = new Options(dbOptions, new
> ColumnFamilyOptions())) {
> existingCFs = RocksDB.listColumnFamilies(options, dbDir.getAbsolutePath());
> } catch (final RocksDBException e) {
> throw new ProcessorStateException("Error listing column families for store
> " + name, e);
> } {code}
> This path is only reached when header-aware state stores are enabled
> (KIP-1271). Workloads that do not enable that feature are unaffected.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
