Hi Gustavo, The config and fail as default sounds good!
Thanks, Hao On Thu, Feb 26, 2026 at 3:13 AM Gustavo de Morais <[email protected]> wrote: > Thanks for the feedback, Leonard and Hao! > > That's a valid point regarding invalid op codes, Hao. We could add an > invalid_op_handling parameter to the PTFs with three options: 'FAIL', > 'LOG', or 'SKIP'. It would be an additional parameter and look like this: > > ... > invalid_op_handling => 'FAIL' | 'LOG' | 'SKIP' > ... > > DLQ would be a way higher LOE to implement so I'd say we skip that option > for now. I'd actually lean toward making FAIL as the default since it's a > serious issue and should be addressed explicitly. Users can then change it > if necessary. > > Kind regards > Gustavo > > On Thu, 26 Feb 2026 at 07:23, Hao Li via dev <[email protected]> wrote: > > > Thanks Gustavo! +1 to the proposal. > > > > Only question I have is error handling: looks it's proposed to ignore > > invalid ops in `ops_mapping` and log warning. I'm wondering if we can > make > > it configurable that it can also fail or log to DLQ? Ignore silently make > > it hard to reason what's going on and if all records have invalid ops, it > > would be lots of warnings. > > > > Thanks, > > Hao > > > > On Wed, Feb 25, 2026 at 7:18 AM Leonard Xu <[email protected]> wrote: > > > > > +1 for the proposal, It will help Flink CDC users a lot especially for > > SQL > > > user cases. > > > > > > The naming FROM_CHANGELOG/TO_CHANGELOG are aligned with DataStream API > > > style which makes sense to me. > > > > > > Best, > > > Leonard > > > > > > On Thu, Feb 19, 2026 at 12:52 AM Gustavo de Morais < > > [email protected] > > > > > > > wrote: > > > > > > > Hi Timo and Dawid, > > > > > > > > Thanks for taking a look and the feedback! To address your valid > point, > > > > I've included a new section called "4.3.6 Invalid op_mapping > > > Combinations", > > > > Dawid. Apart from that, I've also included one functionality to both > > > > functions called "passthrough". It allows users to carry top-level > > fields > > > > (e.g., ts_ms) through the PTF when before/after are used, instead of > > > > dropping them. It seems to be a common use and a useful additional > > > > functionality. Documentation and use cases were added. > > > > > > > > If there are no further comments, I'd like to start the vote in the > > > coming > > > > days. > > > > > > > > Kind regards, > > > > Gustavo > > > > > > > > On Fri, 13 Feb 2026 at 12:13, Dawid Wysakowicz < > [email protected] > > > > > > > wrote: > > > > > > > > > Hi Gustavo, > > > > > Thank you for this very well written FLIP ! I really liked the > > > examples, > > > > it > > > > > helps understand the purpose well. > > > > > > > > > > +1 for this proposal. > > > > > > > > > > One comment from my side. I understand there are only certain > > > > combinations > > > > > that are allowed in `op_mapping`, especially in TO_CHANGELOG PTF. > How > > > do > > > > > you plan to handle invalid cases? For example, what would happen > if I > > > > > specify: ["INSERT, DELETE", "ID"]. I suspect you'd fail during > > > planning, > > > > > am I right? Could we make it explicit in the FLIP? > > > > > > > > > > Best, > > > > > Dawid > > > > > > > > > > On Fri, 13 Feb 2026 at 12:04, Timo Walther <[email protected]> > > wrote: > > > > > > > > > > > Hi Gustavo, > > > > > > > > > > > > thank you for this excellent design doc and coming up with all > > these > > > > > > different combinations of use cases. These two PTFs will > > > significantly > > > > > > improve the Flink CDC story and help polishing the stream/table > > > duality > > > > > > story. > > > > > > > > > > > > +1 for this proposal. > > > > > > > > > > > > Naming-wise I could also imagine calling it TO_CDC/FROM_CDC. But > > the > > > > > > current naming fits well to other API endpoints and Flink > > > terminology, > > > > > > which is why I support FROM_CHANGELOG/TO_CHANGELOG. > > > > > > > > > > > > Cheers, > > > > > > Timo > > > > > > > > > > > > On 06.02.26 18:52, Gustavo de Morais wrote: > > > > > > > Hi everyone, > > > > > > > > > > > > > > I'd like to propose FLIP-564: Support *FROM_CHANGELOG* and > > > > > *TO_CHANGELOG* > > > > > > > built-in PTFs [1] for discussion. > > > > > > > > > > > > > > Flink's DataStream API offers flexible methods like > > > > toChangelogStream() > > > > > > and > > > > > > > fromChangelogStream() to work with changelog streams, but SQL > > users > > > > > > > currently lack this capability. This FLIP introduces two > built-in > > > > > Process > > > > > > > Table Functions (PTFs) to bring *similar* functionality with > > > > additional > > > > > > > features to Flink SQL: > > > > > > > > > > > > > > > > > > > > > - *FROM_CHANGELOG*: Converts an append-only stream of CDC > > > records > > > > > > into a > > > > > > > dynamic table, enabling, for example, custom CDC connector > > > > > > implementations > > > > > > > directly in SQL. > > > > > > > - *TO_CHANGELOG*: Converts a dynamic table back into an > > > > append-only > > > > > > > changelog stream - the first operator that makes it > possible > > to > > > > > > convert a > > > > > > > retract/upsert stream back to append in SQL. > > > > > > > > > > > > > > > > > > > > > Both PTFs support flexible operation mapping (e.g., > > Debezium-style > > > > 'c', > > > > > > > 'u', 'd' codes), before/after image handling, configurable > state > > > TTL, > > > > > and > > > > > > > watermark-based ordering. They are designed to work > > symmetrically, > > > so > > > > > > > FROM_CHANGELOG(TO_CHANGELOG(table)) round-trips correctly. > > > > > > > > > > > > > > The naming follows the existing DataStream API convention. > > > > Alternative > > > > > > > names were considered (e.g., ENCODE_CHANGELOG, DEMATERIALIZE) - > > > they > > > > > are > > > > > > > under the rejected alternatives. If you have any better > > > suggestions, > > > > > feel > > > > > > > free to share them here. > > > > > > > > > > > > > > Looking forward to your feedback and thoughts. > > > > > > > > > > > > > > Kind regards, > > > > > > > Gustavo de Morais > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-564%3A+Support+FROM_CHANGELOG+and+TO_CHANGELOG+built-in+PTFs > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
