DanielLeens commented on issue #10953: URL: https://github.com/apache/seatunnel/issues/10953#issuecomment-4550521147
Thanks for the heap-dump-level report. After checking the current Kafka source path, this does look like a real bug in the default buffering behavior rather than just an aggressive workload. In `KafkaSource#createReader(...)`, `reader_cache_queue_size` is used as the capacity of a `LinkedBlockingQueue<RecordsWithSplitIds<ConsumerRecord<byte[], byte[]>>>`. And in `KafkaPartitionSplitReader.fetch()`, each queue element is a full `consumer.poll(...)` result, not a single message. So the current bound is effectively “number of cached poll batches”, not “number of cached records” or “bytes”. That means with large messages and downstream backpressure, the default value `1024` can translate into a very large in-memory buffer, which matches the heap analysis you shared. As a temporary workaround, please reduce `reader_cache_queue_size` substantially for large-message workloads, and also consider reducing the Kafka consumer batch size so each queued poll result is smaller. Since you marked that you are willing to submit a PR, I’m assigning this issue to you. From the maintainer side, the key fix direction is to make the buffering contract safer by default, not only to document the current behavior. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
