DanielLeens commented on issue #11471: URL: https://github.com/apache/seatunnel/issues/11471#issuecomment-5009051498
Thanks for narrowing this down to the `EXACTLY_ONCE` topic case. I checked the current Kafka batch source enumerator path, and this does look like a genuine connector bug. In batch mode the enumerator currently uses the topic's latest offsets from Kafka admin metadata as the split end offsets. For transactional topics, those raw log end offsets can include transaction control records, while the consumer only emits the visible user records. That matches your reproduction very closely: `100` user records are consumed and committed, but the batch source can still wait for an end offset that is effectively counting an extra control record. So this is worth keeping open as a valid Kafka source bug around transactional-topic completion in batch mode. There is not a connector-side config workaround that fully fixes the completion condition right now. The practical workaround is to avoid using the current batch Kafka source path against transactional topics created with `EXACTLY_ONCE` until this is corrected. We've labeled this as `help wanted` — contributions are very welcome. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
