Hi everyone, Thank you for initiating this discussion. I support adding the optional argument here.
1. TO_CHANGELOG already decides what appears in the output with op_mapping and produces_full_deletes. Whether the code column is emitted at all is the same kind of decision 2. EXCLUDE is SQL-only, so with this argument we could keep the same idiom between SQL and Table API. 3. The cost is minimal. The PR is additive, defaults to the current behavior, and most users will never see the argument. Overall, +1 as long as we correctly document in the include_op_column section that -U/-D rows will look like inserts and will show up as duplicate records, and recommend pairing it with an op_mapping that only keeps INSERT and UPDATE_AFTER. Best regards, Mate On Thu, Oct 15, 2026 at 3:23 PM Gustavo de Morais <[email protected]> wrote: > Hi everyone, > > Bumping this one. I'd be happy if we have other opinions here. > > Kind regards, > Gustavo > > On Thu, 1 Oct 2026 at 15:28, Gustavo de Morais <[email protected]> > wrote: > > > Hi Ferenc, > > > > Thanks for the thread. I'm genuinely interested in the community's > opinion > > on an additional include_op_column. I see three scenarios: > > > > 1. The one you mentioned: > > A retract table produces +I, +U, -U, -D events and users want to convert > > it to append only. After turning it into an append stream, the user might > > be interested only in +I and +U and will use the added "op" column to > > filter out -U and -D. This generally requires the "op" column. In your > > case, the deletion information is already represented in the data schema > > for historical reasons, so you don't need the "op" column. I wonder how > > common that is and if we should add a new parameter for this specific > case, > > or if we are fine with the EXCLUDE idiomatic approach, since this doesn't > > sound like a common case. > > > > 2. The user is certain that their updating table will never contain > either > > -U or -D due to the nature of the data. They want to keep the schema, and > > the op column is not necessary for their use case. > > > > 3.A retract table produces +I, +U, -U, -D events and the user only needs > > +I and +U but doesn't mind the duplicated records that will appear due to > > -D and -U. This is less efficient and will create unnecessary duplicate > > records. However, the user doesn't mind. > > > > Looking forward to hearing what the community thinks is the most common > > case here, and whether it's worth a dedicated parameter. > > > > Kind regards, > > Gustavo > > > > > > > > > > On Thu, 1 Oct 2026 at 15:51, Ferenc Csaky via dev <[email protected]> > > wrote: > > > >> Hello devs, > >> > >> I would like to discuss a question around TO_CHANGELOG. The PTF turns a > >> dynamic > >> table into an insert-only table and prepends an explicit operation-code > >> column. > >> PR #29102 proposes an optional argument to omit that column and preserve > >> the > >> input schema [1]. It would default to the current behavior, so existing > >> queries > >> and compiled plans remain compatible. > >> > >> The motivating use case is a generalized Flink SQL job used with tables > of > >> different schemas. It transforms a retract stream into a special upsert > >> stream > >> that retains only +I and +U; deletion information is already represented > >> in > >> the data schema for historical reasons. Since the schemas are not known > in > >> advance, `SELECT *` is the practical projection. Retaining the schema > >> avoids a > >> custom UDF solely for removing the extra operation column. > >> > >> There are two possible designs: > >> > >> 1. Add an optional TO_CHANGELOG parameter, for example > >> `include_op_column => FALSE`, to omit the operation column. > >> 2. Keep the PTF API as it is and use the SQL projection that will become > >> available with the Calcite 1.42 upgrade: > >> > >> SELECT * EXCLUDE(op) > >> FROM TO_CHANGELOG(input => TABLE source_table, ...); > >> > >> Omitting the operation column does not make a retract stream append-only > >> in > >> the usual sense. Callers that receive updates or deletes need an > >> appropriate > >> representation or filtering strategy; otherwise different change events > >> can > >> look identical after the operation code is removed. The motivating > >> pipeline > >> handles this separately. > >> > >> While `SELECT * EXCLUDE(op)` would solve this specific use case, I would > >> prefer > >> changing the PTF. Requiring TO_CHANGELOG to change the output schema > >> feels like > >> a sometimes unnecessary restriction. An option with a reasonable default > >> adds > >> little complexity, and most users do not even need to know about it, > >> while it > >> leaves more room for flexibility to retain the input schema with `SELECT > >> *` > >> when needed. > >> > >> WDYT? > >> > >> Best, > >> Ferenc > >> > >> [1] https://github.com/apache/flink/pull/29102 > >> > > >
