Hi everyone,
thanks for starting the discussion Gustavo. On one side, I understand
the need for having this convenience feature, but on the other side I
see the following downsides of it:
1. The function is called TO_CHANGELOG, a changelog needs an op field.
Omitting it leaves a false impression that a changelog suddenly becomes
an insert-only pipeline.
2. For all functions, we should avoid overloading functions too heavily.
The number and shape of parameters should be well-defined to avoid
confusion and reduce documentation and special cases in implementation.
Keep the feature matrix dense.
3. A PTF should not handle cases that can be easily done in SQL.
So overall I agree with Gustavo, downstream SQL can do the projection.
In the future, many users might also want to remove the partition key or
rowtime column that a PTF adds. We should rather put effort in support
SELECT * EXCLUDE and have better projection means going forward.
Cheers,
Timo
On 21.10.26 12:05, Gustavo de Morais wrote:
Hi Mate,
Thanks for contributing to the discussion.
I agree with 1 and 3. Regarding 2, the Table API already has an equivalent:
table.toChangelog(...).dropColumns($("op")), or withoutColumns($("op")) in
a select.
Kind regards,
Gustavo
On Thu, 15 Oct 2026 at 16:48, Mate Czagany <[email protected]> wrote:
Hi everyone,
Thank you for initiating this discussion. I support adding the optional
argument here.
1. TO_CHANGELOG already decides what appears in the output with op_mapping
and produces_full_deletes. Whether the code column is emitted at all is the
same kind of decision
2. EXCLUDE is SQL-only, so with this argument we could keep the same idiom
between SQL and Table API.
3. The cost is minimal. The PR is additive, defaults to the current
behavior, and most users will never see the argument.
Overall, +1 as long as we correctly document in the include_op_column
section that -U/-D rows will look like inserts and will show up as
duplicate records, and recommend pairing it with an op_mapping that only
keeps INSERT and UPDATE_AFTER.
Best regards,
Mate
On Thu, Oct 15, 2026 at 3:23 PM Gustavo de Morais <[email protected]>
wrote:
Hi everyone,
Bumping this one. I'd be happy if we have other opinions here.
Kind regards,
Gustavo
On Thu, 1 Oct 2026 at 15:28, Gustavo de Morais <[email protected]>
wrote:
Hi Ferenc,
Thanks for the thread. I'm genuinely interested in the community's
opinion
on an additional include_op_column. I see three scenarios:
1. The one you mentioned:
A retract table produces +I, +U, -U, -D events and users want to
convert
it to append only. After turning it into an append stream, the user
might
be interested only in +I and +U and will use the added "op" column to
filter out -U and -D. This generally requires the "op" column. In your
case, the deletion information is already represented in the data
schema
for historical reasons, so you don't need the "op" column. I wonder how
common that is and if we should add a new parameter for this specific
case,
or if we are fine with the EXCLUDE idiomatic approach, since this
doesn't
sound like a common case.
2. The user is certain that their updating table will never contain
either
-U or -D due to the nature of the data. They want to keep the schema,
and
the op column is not necessary for their use case.
3.A retract table produces +I, +U, -U, -D events and the user only
needs
+I and +U but doesn't mind the duplicated records that will appear due
to
-D and -U. This is less efficient and will create unnecessary duplicate
records. However, the user doesn't mind.
Looking forward to hearing what the community thinks is the most common
case here, and whether it's worth a dedicated parameter.
Kind regards,
Gustavo
On Thu, 1 Oct 2026 at 15:51, Ferenc Csaky via dev <
[email protected]>
wrote:
Hello devs,
I would like to discuss a question around TO_CHANGELOG. The PTF turns
a
dynamic
table into an insert-only table and prepends an explicit
operation-code
column.
PR #29102 proposes an optional argument to omit that column and
preserve
the
input schema [1]. It would default to the current behavior, so
existing
queries
and compiled plans remain compatible.
The motivating use case is a generalized Flink SQL job used with
tables
of
different schemas. It transforms a retract stream into a special
upsert
stream
that retains only +I and +U; deletion information is already
represented
in
the data schema for historical reasons. Since the schemas are not
known
in
advance, `SELECT *` is the practical projection. Retaining the schema
avoids a
custom UDF solely for removing the extra operation column.
There are two possible designs:
1. Add an optional TO_CHANGELOG parameter, for example
`include_op_column => FALSE`, to omit the operation column.
2. Keep the PTF API as it is and use the SQL projection that will
become
available with the Calcite 1.42 upgrade:
SELECT * EXCLUDE(op)
FROM TO_CHANGELOG(input => TABLE source_table, ...);
Omitting the operation column does not make a retract stream
append-only
in
the usual sense. Callers that receive updates or deletes need an
appropriate
representation or filtering strategy; otherwise different change
events
can
look identical after the operation code is removed. The motivating
pipeline
handles this separately.
While `SELECT * EXCLUDE(op)` would solve this specific use case, I
would
prefer
changing the PTF. Requiring TO_CHANGELOG to change the output schema
feels like
a sometimes unnecessary restriction. An option with a reasonable
default
adds
little complexity, and most users do not even need to know about it,
while it
leaves more room for flexibility to retain the input schema with
`SELECT
*`
when needed.
WDYT?
Best,
Ferenc
[1] https://github.com/apache/flink/pull/29102
