Gustavo de Morais created FLINK-41562:
-----------------------------------------
Summary: Declare insert-only output for DEDUPLICATE when it can
only emit inserts, and keep the rowtime attribute
Key: FLINK-41562
URL: https://issues.apache.org/jira/browse/FLINK-41562
Project: Flink
Issue Type: Sub-task
Components: Table SQL / Planner
Reporter: Gustavo de Morais
{{DEDUPLICATE}} declares updating output, but some calls can only ever emit +I.
The planner should declare insert-only output for them, so append-only sinks
and event-time operators can consume the result without {{TO_CHANGELOG}}.
*When the output is insert-only:*
* {{keep => 'first'}} without {{sequence_by}}: in arrival order, the first
record is never replaced.
* PARTITION BY covers every input column (exact duplicates): later records are
identical, and an identical record emits nothing. Holds for both {{keep}}
values.
{{state_ttl}} expiry is fine: an expired key emits +I again.
*Example:* exact duplicates left in a sink after reprocessing. Today this needs
{{ROW_NUMBER()}} plus {{TO_CHANGELOG}}.
{code:sql}
INSERT INTO trades_dedup -- append-only sink
SELECT * FROM DEDUPLICATE(
input => TABLE(trades_with_dups) PARTITION BY (trade_id, ticker, quantity,
price)
)
{code}
Observation regarding rowtime/watermarks: the output is a subset of the input,
in the same order with unchanged timestamps, so the input's watermarks stay
valid. Options:
# The planner keeps the input's rowtime attribute on the output column and
forwards watermarks. No new argument.
# Allow {{on_time}} only in these cases. Simpler in the PTF framework, but more
things for the user to understand.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)