Hello Shrikant, There is a known issue about using KafkaIO as Bounded source [1] Please, take a look on the preceding discussion on this. [2] I’m not sure if there is a workaround for this problem.
[1] https://issues.apache.org/jira/browse/BEAM-6466 [2] https://lists.apache.org/thread.html/bcec8a1fb166029a4adf3f3491c407d49843406020b20f203ec3c2d2@%3Cuser.beam.apache.org%3E > On 17 Dec 2020, at 10:35, shrikant bang <[email protected]> wrote: > > Hi Team, > > I am trying with KafkaIO.Read => FileIO.Write in Batch mode with > SparkRunner. > > With this use case, offsets should be committed at the end of the > pipeline once the files are written to the target location. > > I have couple of queries around it: > I tried with commitOffsetsInFinalize(), but the offsets are not committed > even though the pipeline succeeded. > Is commitOffsetsInFinalize() applicable for only when KafkaIO.Read uses > stream mode? > > Is there any way we can get offsets back to the driver to commit once the > pipeline finishes? > > Thank You, > Shrikant Bang.
