Re: Sync and Async checkpoint time

2018-01-31 Thread Stefan Richter
נואר 2018 21:10 > To: Sofer, Tovi [ICG-IT] <ts72...@imceu.eu.ssmb.com> > Cc: user@flink.apache.org > Subject: Re: Sync and Async checkpoint time > > Hi, > > this looks like the timer service is the culprit for this problem. Timers are > currently not stored in the state b

RE: Sync and Async checkpoint time

2018-01-31 Thread Sofer, Tovi
To: Sofer, Tovi [ICG-IT] <ts72...@imceu.eu.ssmb.com> Cc: user@flink.apache.org Subject: Re: Sync and Async checkpoint time Hi, this looks like the timer service is the culprit for this problem. Timers are currently not stored in the state backend, but in a separate on-heap data str

Re: Sync and Async checkpoint time

2018-01-30 Thread Stefan Richter
Hi, this looks like the timer service is the culprit for this problem. Timers are currently not stored in the state backend, but in a separate on-heap data structure that does not support copy-on-write or async snapshots in general. Therefore, writing the timers for a snapshot is always

Sync and Async checkpoint time

2018-01-30 Thread Sofer, Tovi
Hi group, In our project we are using asynchronous FSStateBackend, and we are trying to move to distributed storage - currently S3. When using this storage we are experiencing issues of high backpressure and high latency, in comparison of local storage. We are trying to understand the reason,