Re: [Discuss] Proposing FLIP-25 - Support User State TTL Natively in Flink

Hi, I like that this design combines the few previously discussed approaches together to address a much broader scope of TTL use cases. Each previously discussed approach alone focuses on a narrower scenario than this one, and has its own limitations. I'm glad to see the community is willing to s

Re: [Discuss] Proposing FLIP-25 - Support User State TTL Natively in Flink

Hi, I like the separation of visibility and clean-up. So far the design only addressed the clean-up aspect and aimed to have state accessible as long as possible, i.e., until it was cleared. We did not consider the use case of compliance. Support for strict visibility is a good idea, IMO. Moreove

Re: [Discuss] Proposing FLIP-25 - Support User State TTL Natively in Flink

Hi andrey, Thanks for this doc! TBH, personally I prefer the approach you outlined in the doc over the previous one that purly based on timers. I think this approach looks very similar to the approach I outlined in this thread before, so it still face the challenges that @Bowen outlined, but I

Re: [Discuss] Proposing FLIP-25 - Support User State TTL Natively in Flink

Hi everybody, We have been recently brainstorming ideas around state TTL in Flink and compiled our thoughts in the following design doc: https://docs.google.com/document/d/1SI_WoXAfOd4_NKpGyk4yh3mf59g12pSGNXRtNFi-tgM

Re: [Discuss] Proposing FLIP-25 - Support User State TTL Natively in Flink

Hi Bowen, Thanks for your clarification, I agree that we should wait for the timer on RocksDB to be finished, after that we could even do some micro-benchmark before start implementing. Best, Sihua On 05/27/2018 15:07，Bowen Li wrote： Thank you Fabian and Sihua. I explained more in the d

Re: [Discuss] Proposing FLIP-25 - Support User State TTL Natively in Flink

Thank you Fabian and Sihua. I explained more in the doc itself and its comments. I believe the bottom line of v1 are 1) it shouldn't be worse than how users implement TTL themselves today, 2) we should develop a generic architecture for TTL for all (currently two) state backends (impl can vary), 3)

Re: [Discuss] Proposing FLIP-25 - Support User State TTL Natively in Flink

Hi, thanks for your reply Fabian, about the overhead of storing the key bytes twice, I think maybe the situation is even a bit worse, in general, it means that the total amount of data to be stored has doubled(for each key, we need to store two records, one for timer, one for state). This ma

Re: [Discuss] Proposing FLIP-25 - Support User State TTL Natively in Flink

Hi, Yes. IMO it makes sense to put the logic into the abstract base classes to share the implementation across different state backends and state primitives. The overhead of storing the key twice is a valid concern, but I'm not sure about the approach to add a timestamp to each value. How would we

Re: [Discuss] Proposing FLIP-25 - Support User State TTL Natively in Flink

Hi Fabian, thanks you very much for the reply, just a alternative. Can we implement the TTL logical in `AbstractStateBackend` and `AbstractState`? A simplest way is to append the `ts` to the state's value? and we use the backend's `current time`(its also can be event time and process time) to ju

Re: [Discuss] Proposing FLIP-25 - Support User State TTL Natively in Flink

Hi Sihua, I think it makes sense to couple state TTL to the timer service. We'll need some kind of timers to expire state, so I think we should reuse components that we have instead of implementing another timer service. Moreover, using the same timer service and using the public state APIs helps

Re: [Discuss] Proposing FLIP-25 - Support User State TTL Natively in Flink

Hi Bowen, thanks for your doc! I left some comments on the doc, the main concerning is that it makes me feel like a coupling that the TTL need to depend on `timer`. Because I think the TTL is a property of the state, so it should be backed by the state backend. If we implement the TTL base on th

Re: [Discuss] Proposing FLIP-25 - Support User State TTL Natively in Flink

Thank you, Fabian! I've created the FLIP-25 page . To continue our discussion points: 1. I see what you mean now. I totally agree. Since we don't completely know it now, shall we experiment or prototype

Re: [Discuss] Proposing FLIP-25 - Support User State TTL Natively in Flink

Hi Bowen, 1. The motivation to keep the TTL logic outside of the state backend was mainly to avoid state backend custom implementations. If we have a generic approach that would work for all state backends, we could try to put the logic into a base class like AbstractStateBackend. After all, state

Re: [Discuss] Proposing FLIP-25 - Support User State TTL Natively in Flink

Thanks Fabian! Here're my comments inline, and let me know your thoughts. 1. Where should the TTL code reside? In the state backend or in the operator? I believe TTL code should not reside in state backend, because a critical design is that TTL is independent of and transparent to state backends.

Re: [Discuss] Proposing FLIP-25 - Support User State TTL Natively in Flink

Hi Bowen, Thanks for updating the proposal. This looks pretty good (as I said before). There are a few areas, that are not yet fully fleshed out: 1. Where should the TTL code reside? In the state backend or in the operator? 2. How to get notified about state accesses? I guess this depends on 1. 3

Re: [Discuss] Proposing FLIP-25 - Support User State TTL Natively in Flink

Hello community, We've come up with a completely new design for Flink state TTL, documented here, and have run it by a few Flink PMC/committers. What do you think? We'd love to hear feedbacks from you Thanks, Bowen > On Wed, Feb 7, 2018 at 12:53 PM, Fabian Hueske wrote: > Hi Bowen, > > Than

Re: [Discuss] Proposing FLIP-25 - Support User State TTL Natively in Flink

Hi Bowen, Thanks for the proposal! I think state TTL would be a great feature! Actually, we have implemented this for SQL / Table API [1]. I've added a couple of comments to the design doc. In principle, I'm not sure if this functionality should be added to the state backends. We could also use t