Hi everyone,
I'd like to propose FLIP-621: Add built-in TO_APPEND PTF [1] for
discussion. Since we added TO_CHANGELOG, we've seen the same pattern come
up frequently: users have an updating table and want to write its inserts
and updates to an append-only sink, dropping deletes. TO_CHANGELOG can do
this, but it needs an op_mapping that drops deletes plus a projection that
removes the op column.
TO_APPEND gives everyone else a very easy way to stay in the append world,
without having to understand the changelog behind it. Full signature:
SELECT * FROM TO_APPEND(input => TABLE my_table);
Inserts and updates are emitted as inserts, update-befores and deletes are
dropped, and the input schema stays unchanged. Following the PTF guidelines
[2], we want to provide functions that solve one problem at a time.
TO_APPEND fits that well: it is a stateless, row-semantics PTF with an easy
to reason implementation. It also covers the use case discussed in the
mailing list for adding one more parameter to TO_CHANGELOG [3]. Looking
forward to your feedback and thoughts.
Kind regards,
Gustavo
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-621%3A+Add+built-in+TO_APPEND+PTF
[2]
https://cwiki.apache.org/confluence/spaces/FLINK/pages/462816208/Built-in+Process+Table+Function+Guidelines
[3] https://lists.apache.org/thread/pfd6ktpyh95j126jm0qolngkrm2bpyhg
