[
https://issues.apache.org/jira/browse/KAFKA-19390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18023820#comment-18023820
]
Chia-Ping Tsai commented on KAFKA-19390:
----------------------------------------
trunk:
https://github.com/apache/kafka/commit/ea7b145860e17f9d90dd06cd983597f1c564b435
https://github.com/apache/kafka/commit/daece61a5054191cd652e502d06756e258cc2e94
4.1:
https://github.com/apache/kafka/commit/807866ca6d563892579bb93efd3e64061824c4fe
https://github.com/apache/kafka/commit/7f8c2a5fc948700b67879788c118d234aa5a66d8
4.0:
3.9:
https://github.com/apache/kafka/commit/e37f31e8b2dcf64df63a86411f1b3e4be4d93171
> AbstractIndex#resize() does not release old mmap on Linux
> ---------------------------------------------------------
>
> Key: KAFKA-19390
> URL: https://issues.apache.org/jira/browse/KAFKA-19390
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 3.8.1
> Reporter: Masahiro Mori
> Assignee: Masahiro Mori
> Priority: Major
> Labels: Linux
> Fix For: 3.9.2, 4.2.0
>
> Attachments: orig_cpu.png, orig_disk_io.png,
> orig_network_processor.png, orig_request_handler.png, patched_cpu.png,
> patched_disk_io.png, patched_network_processor.png,
> patched_request_handler.png
>
>
> Our kafka broker crashed with the following error:
> {code:java}
> [2025-03-29 09:37:03,218] ERROR Error while appending records to
> <topic>-<partition> in dir /kafka-logs/data ...
> java.io.IOException: Map failed
> at java.base/sun.nio.ch.FileChannelImpl.mapInternal(FileChannelImpl.java:1127)
> at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1032)
> at
> org.apache.kafka.storage.internals.log.AbstractIndex.createMappedBuffer(AbstractIndex.java:467)
> at
> org.apache.kafka.storage.internals.log.AbstractIndex.createAndAssignMmap(AbstractIndex.java:105)
> at
> org.apache.kafka.storage.internals.log.AbstractIndex.<init>(AbstractIndex.java:83)
> at org.apache.kafka.storage.internals.log.TimeIndex.<init>(TimeIndex.java:65)
> at
> org.apache.kafka.storage.internals.log.LazyIndex.loadIndex(LazyIndex.java:242)
> at org.apache.kafka.storage.internals.log.LazyIndex.get(LazyIndex.java:179)
> at
> org.apache.kafka.storage.internals.log.LogSegment.timeIndex(LogSegment.java:146)
> at
> org.apache.kafka.storage.internals.log.LogSegment.readMaxTimestampAndOffsetSoFar(LogSegment.java:201)
> at
> org.apache.kafka.storage.internals.log.LogSegment.maxTimestampSoFar(LogSegment.java:211)
> at
> org.apache.kafka.storage.internals.log.LogSegment.append(LogSegment.java:262)
> at kafka.log.LocalLog.append(LocalLog.scala:417)
> ...
> Caused by: java.lang.OutOfMemoryError: Map failed
> at java.base/sun.nio.ch.FileChannelImpl.map0(Native Method)
> at java.base/sun.nio.ch.FileChannelImpl.mapInternal(FileChannelImpl.java:1124)
> ... 33 more{code}
> We found that kafka process hit the vm.max_map_count limit (which was set to
> 262144) and most of the mapped entries correspond to deleted index files.
> {code:java}
> > sudo cat /proc/${KAFKA_PID}/maps | grep deleted
> 7d8c5cc00000-7d8c5d600000 rw-s 00000000 08:11 202854769
> /kafka-logs/data/topic1-22/00000000332910579773.timeindex.deleted (deleted)
> 7d8c5d800000-7d8c5e200000 rw-s 00000000 08:11 202854768
> /kafka-logs/data/topic1-22/00000000332910579773.index.deleted (deleted)
> 7d8c67400000-7d8c67e00000 rw-s 00000000 08:11 202562514
> /kafka-logs/data/topic2-116/00000000165968090794.timeindex.deleted (deleted)
> 7d8c68000000-7d8c68a00000 rw-s 00000000 08:11 202562513
> /kafka-logs/data/topic2-116/00000000165968090794.index.deleted (deleted)
> 7d8c6d400000-7d8c6de00000 rw-s 00000000 08:11 202596518
> /kafka-logs/data/topic2-356/00000000168702579081.timeindex.deleted (deleted)
> 7d8c6e000000-7d8c6ea00000 rw-s 00000000 08:11 202596517
> /kafka-logs/data/topic2-356/00000000168702579081.index.deleted (deleted)
> 7d8c71c00000-7d8c72600000 rw-s 00000000 08:11 202798981
> /kafka-logs/data/topic3-433/00000000116740630582.timeindex.deleted (deleted)
> 7d8c72800000-7d8c73200000 rw-s 00000000 08:11 202798980
> /kafka-logs/data/topic3-433/00000000116740630582.index.deleted (deleted)
> 7d8c77c00000-7d8c78600000 rw-s 00000000 08:11 202754947
> /kafka-logs/data/topic3-74/00000000118067749684.timeindex.deleted (deleted)
> 7d8c78800000-7d8c79200000 rw-s 00000000 08:11 202754946
> /kafka-logs/data/topic3-74/00000000118067749684.index.deleted (deleted)
> 7d8c79400000-7d8c79e00000 rw-s 00000000 08:11 202813710
> /kafka-logs/data/topic2-82/00000000162756700035.timeindex.deleted (deleted)
> 7d8c7a000000-7d8c7aa00000 rw-s 00000000 08:11 202813709
> /kafka-logs/data/topic2-82/00000000162756700035.index.deleted (deleted)
> 7d8c7ac00000-7d8c7b600000 rw-s 00000000 08:11 202596526
> /kafka-logs/data/topic2-355/00000000169939763750.timeindex.deleted (deleted)
> 7d8c7b800000-7d8c7c200000 rw-s 00000000 08:11 202596525
> /kafka-logs/data/topic2-355/00000000169939763750.index.deleted (deleted)
> 7d8c7c400000-7d8c7ce00000 rw-s 00000000 08:11 202562498
> /kafka-logs/data/topic2-295/00000000168913981903.timeindex.deleted (deleted)
> 7d8c7d000000-7d8c7da00000 rw-s 00000000 08:11 202562497
> /kafka-logs/data/topic2-295/00000000168913981903.index.deleted (deleted)
> 7d8c80c00000-7d8c81600000 rw-s 00000000 08:11 202754939
> /kafka-logs/data/topic3-13/00000000115588098896.timeindex.deleted (deleted)
> 7d8c81800000-7d8c82200000 rw-s 00000000 08:11 202754938
> /kafka-logs/data/topic3-13/00000000115588098896.index.deleted (deleted)
> 7d8c83c00000-7d8c84600000 rw-s 00000000 08:11 202798989
> /kafka-logs/data/topic3-314/00000000118254254601.timeindex.deleted (deleted)
> 7d8c84800000-7d8c85200000 rw-s 00000000 08:11 202798988
> /kafka-logs/data/topic3-314/00000000118254254601.index.deleted (deleted)
> ...{code}
> In AbstractIndex.resize(), the old memory mapping is explicitly unmapped on
> windows or z/OS using safeForceUnmap(), but on Linux the unmapping step is
> skipped.
> The same issue was originally reported in KAFKA-7442, but the corresponding
> pull request was never merged.
> We propose that resize() should call safeForceUnmap() on all platforms to
> prevent stale mappings from lingering.
> ref: [https://speakerdeck.com/lycorptech_jp/20250609b]
> h2. Test
> We tested our patch on the following system environment and found no
> measurable performance regression.
> *Environment*
> * Kafka: 3.8.1
> * Java: OpenJDK 17
> * OS: Rocky Linux 9
> * Broker: 10 brokers for original, 10 brokers for patched
> ** CPU: Intel Xeon Silver 4310 (2 sockets)
> ** Memory: 250 GB
> ** Disk: HDD
> *Workloads*
> * Peak throughput per broker:
> ** 6K req/s
> ** 85.2 MB/s
> h3. Idle percentage of request handler and network processor threads
> original:
> !orig_request_handler.png|width=1000,height=222!
> !orig_network_processor.png|width=999,height=163!
> patched:
> !patched_request_handler.png|width=997,height=228!
> !patched_network_processor.png|width=1004,height=160!
> h3. Resource load
> original:
> !orig_cpu.png|width=1000,height=227!
> !orig_disk_io.png|width=997,height=168!
> patched:
> !patched_cpu.png|width=999,height=227!
> !patched_disk_io.png|width=1002,height=160!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
