[
https://issues.apache.org/jira/browse/KAFKA-14432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lucas Brutschy updated KAFKA-14432:
-----------------------------------
Description:
Relying on finalizers in RocksDB has been deprecated for a long time, and
starting with rocksdb 7, finalizers are removed completely (see
[https://github.com/facebook/rocksdb/pull/9523]).
Kafka Streams currently relies on finalizers in parts to not leak memory. This
needs to be resolved before we can upgrade to RocksDB 7.
See [https://github.com/apache/kafka/pull/12809] .
This is a native heap profile after running Kafka Streams without finalizers
for a few hours:
{code:java}
Total: 13547.5 MB
12936.3 95.5% 95.5% 12936.3 95.5% rocksdb::port::cacheline_aligned_alloc
438.5 3.2% 98.7% 438.5 3.2% rocksdb::BlockFetcher::ReadBlockContents
84.0 0.6% 99.3% 84.2 0.6% rocksdb::Arena::AllocateNewBlock
45.9 0.3% 99.7% 45.9 0.3% prof_backtrace_impl
8.1 0.1% 99.7% 14.6 0.1% rocksdb::BlockBasedTable::PutDataBlockToCache
6.4 0.0% 99.8% 12941.4 95.5% Java_org_rocksdb_Statistics_newStatistics___3BJ
6.1 0.0% 99.8% 6.9 0.1% rocksdb::LRUCacheShard::Insert@2d8b20
5.1 0.0% 99.9% 6.5 0.0% rocksdb::VersionSet::ProcessManifestWrites
3.9 0.0% 99.9% 3.9 0.0% rocksdb::WritableFileWriter::WritableFileWriter
3.2 0.0% 99.9% 3.2 0.0% std::string::_Rep::_S_create{code}
was:
Relying on finalizers in RocksDB has been deprecated for a long time, and
starting with rocksdb 7, finalizers are removed completely (see
[https://github.com/facebook/rocksdb/pull/9523]).
Kafka Streams currently relies on finalizers in parts to not leak memory. This
needs to be resolved before we can upgrade to RocksDB 7.
See [https://github.com/apache/kafka/pull/12809] .
This is a native heap profile after running Kafka Streams without finalizers
for a few hours:
Total: 13547.5 MB
12936.3 95.5% 95.5% 12936.3 95.5% rocksdb::port::cacheline_aligned_alloc
438.5 3.2% 98.7% 438.5 3.2% rocksdb::BlockFetcher::ReadBlockContents
84.0 0.6% 99.3% 84.2 0.6% rocksdb::Arena::AllocateNewBlock
45.9 0.3% 99.7% 45.9 0.3% prof_backtrace_impl
8.1 0.1% 99.7% 14.6 0.1%
rocksdb::BlockBasedTable::PutDataBlockToCache
6.4 0.0% 99.8% 12941.4 95.5%
Java_org_rocksdb_Statistics_newStatistics___3BJ
6.1 0.0% 99.8% 6.9 0.1% rocksdb::LRUCacheShard::Insert@2d8b20
5.1 0.0% 99.9% 6.5 0.0%
rocksdb::VersionSet::ProcessManifestWrites
3.9 0.0% 99.9% 3.9 0.0%
rocksdb::WritableFileWriter::WritableFileWriter
3.2 0.0% 99.9% 3.2 0.0% std::string::_Rep::_S_create
> RocksDBStore relies on finalizers to not leak memory
> ----------------------------------------------------
>
> Key: KAFKA-14432
> URL: https://issues.apache.org/jira/browse/KAFKA-14432
> Project: Kafka
> Issue Type: Bug
> Reporter: Lucas Brutschy
> Assignee: Lucas Brutschy
> Priority: Major
>
> Relying on finalizers in RocksDB has been deprecated for a long time, and
> starting with rocksdb 7, finalizers are removed completely (see
> [https://github.com/facebook/rocksdb/pull/9523]).
> Kafka Streams currently relies on finalizers in parts to not leak memory.
> This needs to be resolved before we can upgrade to RocksDB 7.
> See [https://github.com/apache/kafka/pull/12809] .
> This is a native heap profile after running Kafka Streams without finalizers
> for a few hours:
> {code:java}
> Total: 13547.5 MB
> 12936.3 95.5% 95.5% 12936.3 95.5% rocksdb::port::cacheline_aligned_alloc
> 438.5 3.2% 98.7% 438.5 3.2% rocksdb::BlockFetcher::ReadBlockContents
> 84.0 0.6% 99.3% 84.2 0.6% rocksdb::Arena::AllocateNewBlock
> 45.9 0.3% 99.7% 45.9 0.3% prof_backtrace_impl
> 8.1 0.1% 99.7% 14.6 0.1% rocksdb::BlockBasedTable::PutDataBlockToCache
> 6.4 0.0% 99.8% 12941.4 95.5% Java_org_rocksdb_Statistics_newStatistics___3BJ
> 6.1 0.0% 99.8% 6.9 0.1% rocksdb::LRUCacheShard::Insert@2d8b20
> 5.1 0.0% 99.9% 6.5 0.0% rocksdb::VersionSet::ProcessManifestWrites
> 3.9 0.0% 99.9% 3.9 0.0% rocksdb::WritableFileWriter::WritableFileWriter
> 3.2 0.0% 99.9% 3.2 0.0% std::string::_Rep::_S_create{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
