Hi Vas,
thanks for proposing this FLIP. We had a lot of offline discussion for
this design document, but I'm quite happy about the outcome. DEDUPLICATE
is one of the most required transformations in stream processing and
having a dedicated and highly optimized PTF for this will make many use
cases easier.
Big +1 for this FLIP.
Reg Gustavo's feedback: I agree we can drop reset_ttl_on_duplicate
parameter and make it the default for DEDUPLICATE and skip if for
DEDUPLICATE_WINDOW. Also regarding the other open questions: nullability
sorting is defined by Flink's generated comparators already. And reg the
changelog mode, where ever we can declare append we should.
Thanks,
Timo
On 10.11.26 16:40, Gustavo de Morais wrote:
Hi Vas,
The FLIP and the examples look great. I really like the separation between
the updating and the event-time worlds. I think with this we have a clear,
easy-to-use deduplication feature that willl help people easily perform one
of the most common tasks we see.
Regarding one of the open points, I think we can drop the
`reset_ttl_on_duplicate` parameter for DEDUPLICATE_WINDOW. Since this is
window-based event-time processing, most of the state is taken care of by
windows closing. The additional `state_ttl` is just a last-resort cleanup
mechanism for windows that never close, so it should be enough on its own
without an extra parameter to configure it. I'd be interested to hear how
useful others find it for DEDUPLICATE.
Kind regards,
Gustavo
On Tue, 10 Nov 2026 at 16:14, Vas Shabu <[email protected]> wrote:
Hi all,
I'd like to propose FLIP-622: Add built-in DEDUPLICATE and
DEDUPLICATE_WINDOW PTFs [1] for discussion. It supersedes FLIP-614 [2].
While working on DEDUPLICATE_KEEP_LAST, we found a cleaner design: one
function per output mode, with first or last as an argument.
DEDUPLICATE_KEEP_FIRST is unreleased, so we want to raise this before
anything ships.
Today deduplication is written as a ROW_NUMBER() over-window. That pattern
is planned as deduplication only when ordered by a time attribute and
becomes a general Top-N when ordered by a version column. This FLIP adds
two built-in PTFs, each with keep => 'first' | 'last'.
DEDUPLICATE keeps one row per key. Its output is updating, and it accepts
insert-only and updating input:
SELECT * FROM DEDUPLICATE(
input => TABLE(clicks) PARTITION BY event_id,
keep => 'first'
);
DEDUPLICATE_WINDOW emits one final row per key and event-time window, and
its output is insert-only:
SELECT * FROM DEDUPLICATE_WINDOW(
input => TABLE(clicks) PARTITION BY event_id,
keep => 'first',
duration => INTERVAL '5' MINUTES,
on_time => DESCRIPTOR(ts)
);
With this proposal, we aim to improve user experience by introducing a
user-friendly feature to perform one of the most common tasks in Flink.
Looking forward to your feedback and thoughts.
Kind regards,
Vas Shabu
[1]
https://cwiki.apache.org/confluence/spaces/FLINK/pages/462816464/FLIP-622+Add+built-in+DEDUPLICATE+and+DEDUPLICATE_WINDOW+PTFs
[2]
https://cwiki.apache.org/confluence/spaces/FLINK/pages/451975182/FLIP-614+Add+built-in+DEDUPLICATE_KEEP_FIRST+PTF
