Gustavo de Morais created FLINK-41549:
-----------------------------------------
Summary: FLIP-621: Add built-in TO_APPEND PTF
Key: FLINK-41549
URL: https://issues.apache.org/jira/browse/FLINK-41549
Project: Flink
Issue Type: New Feature
Components: Table SQL / API, Table SQL / Planner
Reporter: Gustavo de Morais
Assignee: Gustavo de Morais
Many queries produce an updating result (upsert sources, deduplication,
aggregations, joins), but the place they write to only accepts appends: an
append-only Kafka topic, an append-only sink, or an operator that requires
insert-only input. Since {{TO_CHANGELOG}} (FLIP-564) was released, one use case
keeps coming up: write inserts and updates to an append-only log and drop
deletes. {{TO_CHANGELOG}} can do this, but it requires an {{op_mapping}} that
drops deletes plus a projection that removes the op column.
This FLIP adds a built-in {{TO_APPEND}} process table function that converts an
updating table into an append-only table in one call. INSERT and UPDATE_AFTER
rows are emitted as INSERT rows, UPDATE_BEFORE and DELETE rows are dropped, and
the output keeps the input schema. The function is stateless and has row
semantics. With FLINK-41371 and FLINK-41516, an upsert input with an upsert key
is consumed as emitted, without a {{ChangelogNormalize}}.
{code:sql}
-- Input: upsert table 'customers' with primary key 'id'
-- +U[id:'c1', city:'Berlin']
-- +U[id:'c1', city:'Munich']
-- -D[id:'c1']
SELECT * FROM TO_APPEND(input => TABLE customers)
-- Output (append-only):
-- +I[id:'c1', city:'Berlin']
-- +I[id:'c1', city:'Munich']
{code}
The Table API exposes the same function as {{Table.toAppend()}}.
FLIP:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-621%3A+Add+built-in+TO_APPEND+PTF
Discussion thread:
https://lists.apache.org/thread/g5gytjmlfhglokl7zpcfqd6xh70snjb8
Vote thread: https://lists.apache.org/thread/5lof8ms2s2ycn090ozjzvz1jqfcs7z7m
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)