Re: KafkaIO reading from latest offset when pipeline fails on FlinkRunner

2018-02-11 Thread Sushil Ks
On Thu, Feb 1, 2018 at 1:29 AM, Sushil Ks <sushil...@gmail.com> wrote: > >> Hi, >>Apologies for delay in my reply, >> >> @Raghu Angadi >> This checkpoints 20 mins, as you mentioned before any >> checkpoint is created and if the p

Re: KafkaIO reading from latest offset when pipeline fails on FlinkRunner

2018-02-01 Thread Sushil Ks
eckpoint, the pipeline could restart without any state, >>> in which case KafkaIO would read from latest offset. There is probably some >>> way to verify if pipeline is restarting from a checkpoint. >>> >>> On Sun, Jan 7, 2018 at 10:57 PM, Sushil Ks <sushil...@gmail.co

Re: KafkaIO reading from latest offset when pipeline fails on FlinkRunner

2018-01-08 Thread Sushil Ks
PM, "Reuven Lax" <re...@google.com> wrote: > Do you set ENABLE_AUTO_COMMIT_CONFIG? > > On Sun, Jan 7, 2018 at 10:57 PM, Sushil Ks <sushil...@gmail.com> wrote: > >> HI Aljoscha, >>The issue is let's say I consumed 100 elements in 5 &g

Re: KafkaIO reading from latest offset when pipeline fails on FlinkRunner

2018-01-07 Thread Sushil Ks
the rest 30 elements. Wanted to know if this is expected? In case if you still having doubt let me know will share a code snippet. Regards, Sushil Ks

Re: KafkaIO reading from latest offset when pipeline fails on FlinkRunner

2018-01-04 Thread Sushil Ks
*bump* On Dec 15, 2017 11:22 PM, "Lukasz Cwik" <lc...@google.com> wrote: > +dev@beam.apache.org > > On Thu, Dec 14, 2017 at 11:27 PM, Sushil Ks <sushil...@gmail.com> wrote: > >> Hi Likasz, >>I am not sure whether I can reprod