Hi Team,
I am trying with KafkaIO.Read => FileIO.Write in *Batch* mode with
SparkRunner.
With this use case, offsets should be committed at the end of the
pipeline once the files are written to the target location.
I have couple of queries around it:
1. I tried with commitOffsetsInFinalize(), but the offsets are not
committed even though the pipeline succeeded.
Is *commitOffsetsInFinalize() *applicable for only when KafkaIO.Read
uses stream mode?
2. Is there any way we can get offsets back to the driver to commit
once the pipeline finishes?
Thank You,
Shrikant Bang.
