Re: Checkpointing is not performing well

having any side effects. Use that value as task manager count and >>>>> then start adding your state backend. First you can try with Rocks DB. >>>>> With >>>>> reduced task manager count you might get good results. >>>>> >>>>> Reg

Re: Checkpointing is not performing well

ep 8, 2019 at 10:15 AM Rohan Thimmappa < >>>> rohan.thimma...@gmail.com> wrote: >>>> >>>>> Ravi, have you looked at the io operation(iops) rate of the disk? You >>>>> can monitoring the iops performance and tune it accordingly with your wor

Re: Checkpointing is not performing well

ppa < >>> rohan.thimma...@gmail.com> wrote: >>> >>>> Ravi, have you looked at the io operation(iops) rate of the disk? You >>>> can monitoring the iops performance and tune it accordingly with your work >>>> load. This helped us in our proje

Re: Checkpointing is not performing well

ch all the parameters. >>> >>> Rohan >>> >>> >>> -- >>> *From:* Ravi Bhushan Ratnakar >>> *Sent:* Saturday, September 7, 2019 5:38 PM >>> *To:* Rafi Aroch >>> *Cc:* user >>&g

Re: Checkpointing is not performing well

t; Rohan >> >> >> -- >> *From:* Ravi Bhushan Ratnakar >> *Sent:* Saturday, September 7, 2019 5:38 PM >> *To:* Rafi Aroch >> *Cc:* user >> *Subject:* Re: Checkpointing is not performing well >> >> Hi Rafi, >>

Re: Checkpointing is not performing well

s. > > Rohan > > > -- > *From:* Ravi Bhushan Ratnakar > *Sent:* Saturday, September 7, 2019 5:38 PM > *To:* Rafi Aroch > *Cc:* user > *Subject:* Re: Checkpointing is not performing well > > Hi Rafi, > > Thank you for your quick

Re: Checkpointing is not performing well

: Ravi Bhushan Ratnakar Sent: Saturday, September 7, 2019 5:38 PM To: Rafi Aroch Cc: user Subject: Re: Checkpointing is not performing well Hi Rafi, Thank you for your quick response. I have tested with rocksdb state backend. Rocksdb required significantly more taskmanager to perform as compare

Re: Checkpointing is not performing well

Hi Rafi, Thank you for your quick response. I have tested with rocksdb state backend. Rocksdb required significantly more taskmanager to perform as compare to filesystem state backend. The problem here is that checkpoint process is not fast enough to complete. Our requirement is to do checkout

Re: Checkpointing is not performing well

Hi Ravi, Consider moving to RocksDB state backend, where you can enable incremental checkpointing. This will make you checkpoints size stay pretty much constant even when your state becomes larger.

Checkpointing is not performing well

Hi All, I am writing a streaming application using Flink 1.9. This application consumes data from kinesis stream which is basically avro payload. Application is using KeyedProcessFunction to execute business logic on the basis of correlation id using event time characteristics with below