Re: 答复: Slow flink checkpoint

- > *发件人:* Fabian Hueske <fhue...@gmail.com> > *发送时间:* 2018年4月16日 16:21 > *收件人:* makeyang > *抄送:* user; Aljoscha Krettek > *主题:* Re: Slow flink checkpoint > > Hi everybody, > > Thanks so much for looking into this issue and posting the detailed > de

答复: Slow flink checkpoint

构<http://tig.jd.com/> tig.jd.com TIG官网 发件人: Fabian Hueske <fhue...@gmail.com> 发送时间: 2018年4月16日 16:21 收件人: makeyang 抄送: user; Aljoscha Krettek 主题: Re: Slow flink checkpoint Hi everybody, Thanks so much for looking into this issue and posting the detailed de

Re: Slow flink checkpoint

since flink forward SF has done. can you guys give some minutes to take a look at this issue and give some thoughts on it? help to review/comments on my desgin? or give us a design so that I can help to implement it. thanks a lot. -- Sent from:

Re: Slow flink checkpoint

Hi Stefan , Fabian , Keyang is engineer in our team, he has do a lot of efforts on the timers' snapshot async. What do you think of his idea？ Best, Deqiang TIG.JD.COM > 在 2018年4月1日，下午7:21，makeyang 写道： > > I have put a lot of

Re: Slow flink checkpoint

the test is very promising. the time sync part takes from couple of seconds to couple of mill-seconds. 1000x time reduce(overall time not save since it is just move from sync to async) are u guys interested in this change? -- Sent from:

Re: Slow flink checkpoint

I have put a lot of efforts on this issue and try to resolve it: 1. let me describe current timers' snapshot path first: a) for each keygroup, invoke InternalTimeServiceManager.snapshotStateForKeyGroup b) InternalTimeServiceManager create a InternalTimerServiceSerializationProxy to write

Re: Slow flink checkpoint

Hi, Yes, you cannot start a separate thread to cleanup the state. State is managed by Flink and can only be accessed at certain points in time when the user code is called. If you are using event time, another trick you could play is to only register all timers on (currentWatermark + 1). That

Re: Slow flink checkpoint

Hi Fabian , Reduce the number of timers is a good idea. But in my application the timer is different from the key registered follow the keyBy . May be it can't work with an upper and lower bound. I try modify the flink resource and start a thread to clean the expired keyed sate, but it

Re: Slow flink checkpoint

Hi, yes, that is correct, the timer service is currently only available in main-memory and only with synchronous snapshots. this topic is on our TODO list for after the Flink 1.5 release. Best, Stefan > Am 16.03.2018 um 09:03 schrieb Fabian Hueske : > > Hi, > > AFAIK,

Re: Slow flink checkpoint

Hi, AFAIK, that's not possible. The only "solution" is to reduce the number of timers. Whether that's possible or not, depends on the application. For example, if you use timers to clean up state, you can work with an upper and lower bound and only register one timer for each (upper - lower)

Slow flink checkpoint

Hi, I'm run a job on Flink streaming. I found with the increase in the number of 'InternalTimer' object the checkpoint more and more slowly. Is there any way to solve this problem ? such as make the "timeServiceManager" snapshot async. Thanks