gaborgsomogyi commented on issue #25135: [SPARK-28367][SS] Use new KafkaConsumer.poll API in Kafka connector URL: https://github.com/apache/spark/pull/25135#issuecomment-520830078 To make a small extract about the discussion in the Kafka dev channel the proposal where tends to be agreement is the following: * Call `AdminClient.listTopics` to get possible topic for subscribe pattern (in case of assign and subscribe such step not needed). Here there are 2 possibilities. 1. Do the filtering in Spark code 2. Kafka adds regex to the mentioned API Worth to mention such filtering is happening on the client side in `Consumer` at the moment so this would not cause any unwanted extra memory consumption. * [KIP-396](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97551484) would add `AdminClient.listOffsets` API from where offsets can be obtained without data polling.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
