Hi Ferenc, Thanks for the thread. I'm genuinely interested in the community's opinion on an additional include_op_column. I see three scenarios:
1. The one you mentioned: A retract table produces +I, +U, -U, -D events and users want to convert it to append only. After turning it into an append stream, the user might be interested only in +I and +U and will use the added "op" column to filter out -U and -D. This generally requires the "op" column. In your case, the deletion information is already represented in the data schema for historical reasons, so you don't need the "op" column. I wonder how common that is and if we should add a new parameter for this specific case, or if we are fine with the EXCLUDE idiomatic approach, since this doesn't sound like a common case. 2. The user is certain that their updating table will never contain either -U or -D due to the nature of the data. They want to keep the schema, and the op column is not necessary for their use case. 3.A retract table produces +I, +U, -U, -D events and the user only needs +I and +U but doesn't mind the duplicated records that will appear due to -D and -U. This is less efficient and will create unnecessary duplicate records. However, the user doesn't mind. Looking forward to hearing what the community thinks is the most common case here, and whether it's worth a dedicated parameter. Kind regards, Gustavo On Thu, 1 Oct 2026 at 15:51, Ferenc Csaky via dev <[email protected]> wrote: > Hello devs, > > I would like to discuss a question around TO_CHANGELOG. The PTF turns a > dynamic > table into an insert-only table and prepends an explicit operation-code > column. > PR #29102 proposes an optional argument to omit that column and preserve > the > input schema [1]. It would default to the current behavior, so existing > queries > and compiled plans remain compatible. > > The motivating use case is a generalized Flink SQL job used with tables of > different schemas. It transforms a retract stream into a special upsert > stream > that retains only +I and +U; deletion information is already represented in > the data schema for historical reasons. Since the schemas are not known in > advance, `SELECT *` is the practical projection. Retaining the schema > avoids a > custom UDF solely for removing the extra operation column. > > There are two possible designs: > > 1. Add an optional TO_CHANGELOG parameter, for example > `include_op_column => FALSE`, to omit the operation column. > 2. Keep the PTF API as it is and use the SQL projection that will become > available with the Calcite 1.42 upgrade: > > SELECT * EXCLUDE(op) > FROM TO_CHANGELOG(input => TABLE source_table, ...); > > Omitting the operation column does not make a retract stream append-only in > the usual sense. Callers that receive updates or deletes need an > appropriate > representation or filtering strategy; otherwise different change events can > look identical after the operation code is removed. The motivating pipeline > handles this separately. > > While `SELECT * EXCLUDE(op)` would solve this specific use case, I would > prefer > changing the PTF. Requiring TO_CHANGELOG to change the output schema feels > like > a sometimes unnecessary restriction. An option with a reasonable default > adds > little complexity, and most users do not even need to know about it, while > it > leaves more room for flexibility to retain the input schema with `SELECT *` > when needed. > > WDYT? > > Best, > Ferenc > > [1] https://github.com/apache/flink/pull/29102 >
