Hi everyone, Bumping this one. I'd be happy if we have other opinions here.
Kind regards, Gustavo On Thu, 1 Oct 2026 at 15:28, Gustavo de Morais <[email protected]> wrote: > Hi Ferenc, > > Thanks for the thread. I'm genuinely interested in the community's opinion > on an additional include_op_column. I see three scenarios: > > 1. The one you mentioned: > A retract table produces +I, +U, -U, -D events and users want to convert > it to append only. After turning it into an append stream, the user might > be interested only in +I and +U and will use the added "op" column to > filter out -U and -D. This generally requires the "op" column. In your > case, the deletion information is already represented in the data schema > for historical reasons, so you don't need the "op" column. I wonder how > common that is and if we should add a new parameter for this specific case, > or if we are fine with the EXCLUDE idiomatic approach, since this doesn't > sound like a common case. > > 2. The user is certain that their updating table will never contain either > -U or -D due to the nature of the data. They want to keep the schema, and > the op column is not necessary for their use case. > > 3.A retract table produces +I, +U, -U, -D events and the user only needs > +I and +U but doesn't mind the duplicated records that will appear due to > -D and -U. This is less efficient and will create unnecessary duplicate > records. However, the user doesn't mind. > > Looking forward to hearing what the community thinks is the most common > case here, and whether it's worth a dedicated parameter. > > Kind regards, > Gustavo > > > > > On Thu, 1 Oct 2026 at 15:51, Ferenc Csaky via dev <[email protected]> > wrote: > >> Hello devs, >> >> I would like to discuss a question around TO_CHANGELOG. The PTF turns a >> dynamic >> table into an insert-only table and prepends an explicit operation-code >> column. >> PR #29102 proposes an optional argument to omit that column and preserve >> the >> input schema [1]. It would default to the current behavior, so existing >> queries >> and compiled plans remain compatible. >> >> The motivating use case is a generalized Flink SQL job used with tables of >> different schemas. It transforms a retract stream into a special upsert >> stream >> that retains only +I and +U; deletion information is already represented >> in >> the data schema for historical reasons. Since the schemas are not known in >> advance, `SELECT *` is the practical projection. Retaining the schema >> avoids a >> custom UDF solely for removing the extra operation column. >> >> There are two possible designs: >> >> 1. Add an optional TO_CHANGELOG parameter, for example >> `include_op_column => FALSE`, to omit the operation column. >> 2. Keep the PTF API as it is and use the SQL projection that will become >> available with the Calcite 1.42 upgrade: >> >> SELECT * EXCLUDE(op) >> FROM TO_CHANGELOG(input => TABLE source_table, ...); >> >> Omitting the operation column does not make a retract stream append-only >> in >> the usual sense. Callers that receive updates or deletes need an >> appropriate >> representation or filtering strategy; otherwise different change events >> can >> look identical after the operation code is removed. The motivating >> pipeline >> handles this separately. >> >> While `SELECT * EXCLUDE(op)` would solve this specific use case, I would >> prefer >> changing the PTF. Requiring TO_CHANGELOG to change the output schema >> feels like >> a sometimes unnecessary restriction. An option with a reasonable default >> adds >> little complexity, and most users do not even need to know about it, >> while it >> leaves more room for flexibility to retain the input schema with `SELECT >> *` >> when needed. >> >> WDYT? >> >> Best, >> Ferenc >> >> [1] https://github.com/apache/flink/pull/29102 >> >
