nandini12396 opened a new pull request, #21102: URL: https://github.com/apache/kafka/pull/21102
This change addresses high GC pressure by allocating tiered storage fetch buffers in direct (off-heap) memory instead of the JVM heap. When direct memory is exhausted, the system gracefully falls back to heap allocation with a warning. Problem: During tiered storage reads, heap-allocated buffers bypass young generation and go directly to old generation (humongous allocations). Under high read load, these accumulate rapidly and trigger frequent, expensive G1 Old Generation collections causing significant GC pause times. Solution: - Introduced DirectBufferPool that pools direct buffers using WeakReferences, allowing GC to reclaim buffers under memory pressure - Modified RemoteLogInputStream to use pooled direct buffers instead of per-request heap allocation - Graceful fallback to heap allocation when direct memory is exhausted -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
