Carl-Zhou-CN commented on PR #6408: URL: https://github.com/apache/seatunnel/pull/6408#issuecomment-2308044857
> Please share more information about why we need refactor kafka? And what we can get after refactor? > Please share more information about why we need refactor kafka? And what we can get after refactor? Firstly, I believe there is room for improvement in the previous approach where a new consumer thread is created for each split, and each consumer thread polls in a loop. Enhancements could involve having one consumer thread per parallelism level, ensuring clearer offset management, and promptly removing consumed partitions to reduce overhead. Secondly, a more seamless integration of streaming and batch processing can be achieved, eliminating the need for consumer threads to differentiate between streaming and batch behaviors. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
