Thanks for initiating the discussion and wrap-up the conclusion Andrey, and thanks all for participating.
Just to confirm, that for the out-of-order case, the conclusion is to update the data and timestamp with the currently-being-processed record w/o checking whether it's an old data, right? In this way we could save the comparison-on-each-record cost while may delete data earlier than its time-to-live, seems a fair trade-off for me but not sure whether this could satisfy all real-world demand. Anyway I think it's fine to keep-it-as-is and discuss/improve if any user requirement emerges later. Will start/continue the development/implementation following the conclusion if no objections. Thanks. Best Regards, Yu On Mon, 15 Apr 2019 at 21:58, Andrey Zagrebin <and...@ververica.com> wrote: > Hi everybody, > > Thanks a lot for your detailed feedback on this topic. > It looks like we can already do some preliminary wrap-up for this > discussion. > > As far as I see we have the following trends: > > *Last access timestamp: **Event timestamp of currently being processed > record* > > *Current timestamp to check expiration, *two options: > - *Last emitted watermark* > *- **Current processing time* > > From the implementation perspective, it does not seem to be a big deal to > make it configurable as we already have processing time provider. Although, > it looks like our TtlTimeProvider would need two methods from now on: > getAccessTimestamp and getCurrentTimestamp. > > The biggest concern is out-of-orderness problem. In general, from Flink > side it does not look that we can do a lot about it except putting again a > caveat into the user docs about it. It depends on the use case whether the > out-of-orderness can be tolerated or not and whether an additional stream > ordering operator needs to be put before TTL state access. > > I would still consider TTL event time feature to be implemented as we have > some user requests for it. Any further feedback is appreciated. > > Best, > Andrey > > On Tue, Apr 9, 2019 at 5:26 PM aitozi <gjying1...@gmail.com> wrote: > >> Hi, Andrey >> >> I think ttl state has another scenario to simulate the slide window with >> the >> process function. User can define a state to store the data with the >> latest >> 1 day. And trigger calculate on the state every 5min. It is a operator >> similar to slidewindow. But i think it is more efficient than the >> slidewindow because it dont have to store the redundant data and the >> expire >> data can be delete automatic. >> >> However with the support of ttl state based on processing time we can just >> implement the processing slide window. If we can support ttl based on >> event >> time I think we can expand this capacity. >> >> So in this scenario, the event-time-accesstime/watermark-expiration-check >> will be the proper combination. >> >> I think if can add the interface to allow user to custom will be >> flexible. >> >> Thanks, >> Aitozi >> >> >> >> -- >> Sent from: >> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ >> >
