nzw921rx opened a new pull request, #10954: URL: https://github.com/apache/seatunnel/pull/10954
Closes https://github.com/apache/seatunnel/issues/10953 ## Purpose The Kafka source connector's internal element queue (`LinkedBlockingQueue`) between the fetcher thread and the reader thread uses a default capacity of 1024. Each element in the queue is one complete `consumer.poll()` batch (up to `max.poll.records` = 500 messages), not a single message. Under back-pressure, the queue fills up and can consume gigabytes of heap, causing Full GC and OOM. ## Changes - **`KafkaSourceOptions.java`**: Change default value of `reader_cache_queue_size` from 1024 to 2. Update description to clarify the semantics. - **`docs/en/connectors/source/Kafka.md`**: Add dedicated `reader_cache_queue_size` section explaining the batch-based queue semantics and memory implications. - **`docs/zh/connectors/source/Kafka.md`**: Same as above in Chinese. ## Why 2? - Each queue element holds one full `consumer.poll()` result, 2 elements are sufficient to keep the fetcher-reader pipeline saturated without idle gaps. - With `max.poll.records = 500` and 12 KB messages, worst-case memory drops from ~6 GB (1024) to ~12 MB (2). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
