Hi Vas, The FLIP and the examples look great. I really like the separation between the updating and the event-time worlds. I think with this we have a clear, easy-to-use deduplication feature that willl help people easily perform one of the most common tasks we see.
Regarding one of the open points, I think we can drop the `reset_ttl_on_duplicate` parameter for DEDUPLICATE_WINDOW. Since this is window-based event-time processing, most of the state is taken care of by windows closing. The additional `state_ttl` is just a last-resort cleanup mechanism for windows that never close, so it should be enough on its own without an extra parameter to configure it. I'd be interested to hear how useful others find it for DEDUPLICATE. Kind regards, Gustavo On Tue, 10 Nov 2026 at 16:14, Vas Shabu <[email protected]> wrote: > Hi all, > > I'd like to propose FLIP-622: Add built-in DEDUPLICATE and > DEDUPLICATE_WINDOW PTFs [1] for discussion. It supersedes FLIP-614 [2]. > While working on DEDUPLICATE_KEEP_LAST, we found a cleaner design: one > function per output mode, with first or last as an argument. > DEDUPLICATE_KEEP_FIRST is unreleased, so we want to raise this before > anything ships. > > Today deduplication is written as a ROW_NUMBER() over-window. That pattern > is planned as deduplication only when ordered by a time attribute and > becomes a general Top-N when ordered by a version column. This FLIP adds > two built-in PTFs, each with keep => 'first' | 'last'. > > DEDUPLICATE keeps one row per key. Its output is updating, and it accepts > insert-only and updating input: > > SELECT * FROM DEDUPLICATE( > input => TABLE(clicks) PARTITION BY event_id, > keep => 'first' > ); > > DEDUPLICATE_WINDOW emits one final row per key and event-time window, and > its output is insert-only: > > SELECT * FROM DEDUPLICATE_WINDOW( > input => TABLE(clicks) PARTITION BY event_id, > keep => 'first', > duration => INTERVAL '5' MINUTES, > on_time => DESCRIPTOR(ts) > ); > > With this proposal, we aim to improve user experience by introducing a > user-friendly feature to perform one of the most common tasks in Flink. > Looking forward to your feedback and thoughts. > > Kind regards, > Vas Shabu > > > [1] > https://cwiki.apache.org/confluence/spaces/FLINK/pages/462816464/FLIP-622+Add+built-in+DEDUPLICATE+and+DEDUPLICATE_WINDOW+PTFs > [2] > https://cwiki.apache.org/confluence/spaces/FLINK/pages/451975182/FLIP-614+Add+built-in+DEDUPLICATE_KEEP_FIRST+PTF >
