skonto edited a comment on issue #24613: [SPARK-27549][SS] Add support for committing kafka offsets per batch for supporting external tooling URL: https://github.com/apache/spark/pull/24613#issuecomment-493974211 > Why not use one group and listConsumerGroupOffsets? @gaborgsomogyi yes `listConsumerGroupOffsets` could be used but is it implemented for most clients? Check [here](https://github.com/edenhill/librdkafka/issues/2173). How faster is it actually, I wanted to check how it is implemented. On the other hand when I say filtering, I dont mean filter the whole topic, it would mean pick up from the latest and as you see new records coming in that topic, process them or not based on the filter. Of course that could be also slow. I dont have a clear view of the performance at the moment but I dont mind switching to that special call. @seglo thoughts? When you say one group what do you mean? If I create the groupId per source per query then partial data may be assigned.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
