Hi Claude, Thanks for writing this KIP. This issue seems particularly thorny, and I appreciate everyone's effort to address this.
I want to share my concern with the KIP's proposal of the use of memory mapped files – mmap is Java's achilles heel, Kafka should make less use of it, not more. The JVM often needs to stop all application threads (aka mutator threads) before some operations, such as GC, optimizations, redefinitions, internal cleanups and various other internal reasons. This is known as Safepointing. Because the JVM cannot forcefully stop threads, it must instead wait for each thread to observe the Safepointing request, mark themselves as safe and stop. A single delayed thread can leave the whole JVM hanging, waiting. Reads and writes to memory mapped files can trigger system interrupts, which can block on IO for prolonged amounts of time. One particualrly bad example is hitting the page cage dirty ratio, and having to flush all of the page cage, in a potentially large (high RAM) system into a potentially slow filesystem. I have seen pauses as extreme as 1 minute, and others have reported There are other public reports on this. [1][2] Safepointing in the JVM is designed with mechanisms to prevent having to wait for a single busy thread: Threads mark themselves as safe before waiting on locks, before system calls, before doing JNI, etc, and upon returning they check if a Safepoint is ongoing. So if a read or write syscall takes a bit longer that's fine, the JVM won't halt for Safepointing, it will proceed knowing that any thread stuck on a syscall will stop if necessary when it returns. But there's no protection against long system interrups. >From the JVM's perspective the use of mmap is just a simple memory access, so there's no Safepointing protection around that. The kernel does know nor care for Java's Safepointing, and does not treat halting a single unsuspecting thread for a longer period of time with the severity that it may imply during a JVM Safepoint. So for this reason, I urge you to consider alternatives to the use of memory mapped files. Best, -- Igor https://groups.google.com/g/mechanical-sympathy/c/LFrJPhyVOJ4 https://groups.google.com/g/mechanical-sympathy/c/tepoA7PRFRU/m/7HbSINaFBgAJ
