Hi, I have been experimenting with the Continuous mode and the Micro batch mode in Spark Structured Streaming. When enabling checkpoint to S3 instead of the local File System we see that Continuous mode has no change in latency (expected due to async checkpointing) however the Micro-batch mode experiences degradation likely due to sync checkpointing.
Is there any way to get async checkpointing in the micro-batching mode as well to improve latency. Could that be done with custom checkpointing logic ? Any pointers / experiences in that direction would be helpful.
