skonto edited a comment on issue #24613: [SPARK-27549][SS] Add support for committing kafka offsets per batch for supporting external tooling URL: https://github.com/apache/spark/pull/24613#issuecomment-494324725 I understand the danger it is the same danger as with `kafka.group.id` but I was thinking if they dont set `commitOffsetsOnCheckpoints` nothing happens. So if they set that they should know what they are doing right? Current approach in structured streaming is we take care of the details so you dont need to reason about streaming details, I agree with that for most users. What if an admin knows what he is doing though and wants to monitor a specific query in production? There is a trade-off here make it easy for monitoring vs hiding the details.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
