skonto edited a comment on issue #24613: [SPARK-27549][SS] Add support for committing kafka offsets per batch for supporting external tooling URL: https://github.com/apache/spark/pull/24613#issuecomment-494313373 @gaborgsomogyi > Something like monitoringGroupId Using the source options affects all the queries right? While what we want is a specific id per query and you can only pass a **full predictable groupId** at the point when the query object is created (start). @HeartSaVioR > EDIT: This case might be still an issue even if query ID is being used as consumer ID - This may also occur when queries are started/restarted in quick succession. Could you elaborate a bit more on this? I think since the streaming object is kept around on a quick restart you could just start again from the beginning creating the ids. Do you mean that there will be issues because of re-using the same id in a short period of time? In that case I think the brokers are notified since the consumer is closed in a graceful shutdown of the query.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
