Hi Mate,
Thanks for contributing to the discussion.
I agree with 1 and 3. Regarding 2, the Table API already has an equivalent:
table.toChangelog(...).dropColumns($("op")), or withoutColumns($("op")) in
a select.
Kind regards,
Gustavo
On Thu, 15 Oct 2026 at 16:48, Mate Czagany <[email protected]> wrote:
> Hi everyone,
>
> Thank you for initiating this discussion. I support adding the optional
> argument here.
>
> 1. TO_CHANGELOG already decides what appears in the output with op_mapping
> and produces_full_deletes. Whether the code column is emitted at all is the
> same kind of decision
>
> 2. EXCLUDE is SQL-only, so with this argument we could keep the same idiom
> between SQL and Table API.
>
> 3. The cost is minimal. The PR is additive, defaults to the current
> behavior, and most users will never see the argument.
>
> Overall, +1 as long as we correctly document in the include_op_column
> section that -U/-D rows will look like inserts and will show up as
> duplicate records, and recommend pairing it with an op_mapping that only
> keeps INSERT and UPDATE_AFTER.
>
> Best regards,
> Mate
>
> On Thu, Oct 15, 2026 at 3:23 PM Gustavo de Morais <[email protected]>
> wrote:
>
> > Hi everyone,
> >
> > Bumping this one. I'd be happy if we have other opinions here.
> >
> > Kind regards,
> > Gustavo
> >
> > On Thu, 1 Oct 2026 at 15:28, Gustavo de Morais <[email protected]>
> > wrote:
> >
> > > Hi Ferenc,
> > >
> > > Thanks for the thread. I'm genuinely interested in the community's
> > opinion
> > > on an additional include_op_column. I see three scenarios:
> > >
> > > 1. The one you mentioned:
> > > A retract table produces +I, +U, -U, -D events and users want to
> convert
> > > it to append only. After turning it into an append stream, the user
> might
> > > be interested only in +I and +U and will use the added "op" column to
> > > filter out -U and -D. This generally requires the "op" column. In your
> > > case, the deletion information is already represented in the data
> schema
> > > for historical reasons, so you don't need the "op" column. I wonder how
> > > common that is and if we should add a new parameter for this specific
> > case,
> > > or if we are fine with the EXCLUDE idiomatic approach, since this
> doesn't
> > > sound like a common case.
> > >
> > > 2. The user is certain that their updating table will never contain
> > either
> > > -U or -D due to the nature of the data. They want to keep the schema,
> and
> > > the op column is not necessary for their use case.
> > >
> > > 3.A retract table produces +I, +U, -U, -D events and the user only
> needs
> > > +I and +U but doesn't mind the duplicated records that will appear due
> to
> > > -D and -U. This is less efficient and will create unnecessary duplicate
> > > records. However, the user doesn't mind.
> > >
> > > Looking forward to hearing what the community thinks is the most common
> > > case here, and whether it's worth a dedicated parameter.
> > >
> > > Kind regards,
> > > Gustavo
> > >
> > >
> > >
> > >
> > > On Thu, 1 Oct 2026 at 15:51, Ferenc Csaky via dev <
> [email protected]>
> > > wrote:
> > >
> > >> Hello devs,
> > >>
> > >> I would like to discuss a question around TO_CHANGELOG. The PTF turns
> a
> > >> dynamic
> > >> table into an insert-only table and prepends an explicit
> operation-code
> > >> column.
> > >> PR #29102 proposes an optional argument to omit that column and
> preserve
> > >> the
> > >> input schema [1]. It would default to the current behavior, so
> existing
> > >> queries
> > >> and compiled plans remain compatible.
> > >>
> > >> The motivating use case is a generalized Flink SQL job used with
> tables
> > of
> > >> different schemas. It transforms a retract stream into a special
> upsert
> > >> stream
> > >> that retains only +I and +U; deletion information is already
> represented
> > >> in
> > >> the data schema for historical reasons. Since the schemas are not
> known
> > in
> > >> advance, `SELECT *` is the practical projection. Retaining the schema
> > >> avoids a
> > >> custom UDF solely for removing the extra operation column.
> > >>
> > >> There are two possible designs:
> > >>
> > >> 1. Add an optional TO_CHANGELOG parameter, for example
> > >> `include_op_column => FALSE`, to omit the operation column.
> > >> 2. Keep the PTF API as it is and use the SQL projection that will
> become
> > >> available with the Calcite 1.42 upgrade:
> > >>
> > >> SELECT * EXCLUDE(op)
> > >> FROM TO_CHANGELOG(input => TABLE source_table, ...);
> > >>
> > >> Omitting the operation column does not make a retract stream
> append-only
> > >> in
> > >> the usual sense. Callers that receive updates or deletes need an
> > >> appropriate
> > >> representation or filtering strategy; otherwise different change
> events
> > >> can
> > >> look identical after the operation code is removed. The motivating
> > >> pipeline
> > >> handles this separately.
> > >>
> > >> While `SELECT * EXCLUDE(op)` would solve this specific use case, I
> would
> > >> prefer
> > >> changing the PTF. Requiring TO_CHANGELOG to change the output schema
> > >> feels like
> > >> a sometimes unnecessary restriction. An option with a reasonable
> default
> > >> adds
> > >> little complexity, and most users do not even need to know about it,
> > >> while it
> > >> leaves more room for flexibility to retain the input schema with
> `SELECT
> > >> *`
> > >> when needed.
> > >>
> > >> WDYT?
> > >>
> > >> Best,
> > >> Ferenc
> > >>
> > >> [1] https://github.com/apache/flink/pull/29102
> > >>
> > >
> >
>
