HeartSaVioR commented on PR #38517: URL: https://github.com/apache/spark/pull/38517#issuecomment-1354530532
I'm not in favor of introducing change against normal microbatch execution. As I commented, it seems to me as high risk one. Instead of smooth transition for all cases, I'd say we should just support transition from async to sync only when the checkpoint interval is set to 0, say, when the query has contiguous offsets. If they want to switch the mode for the query which uses checkpoint interval, rerun the query with async mode with setting up checkpoint interval to 0, and see the query running for a couple of batches, and terminate. This will also simply work for the rollback case of Spark version. Transition from sync to async is always feasible, as long as the query does not touch the limitations. What do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
