Hello devs, I would like to discuss a question around TO_CHANGELOG. The PTF turns a dynamic table into an insert-only table and prepends an explicit operation-code column. PR #29102 proposes an optional argument to omit that column and preserve the input schema [1]. It would default to the current behavior, so existing queries and compiled plans remain compatible.
The motivating use case is a generalized Flink SQL job used with tables of different schemas. It transforms a retract stream into a special upsert stream that retains only +I and +U; deletion information is already represented in the data schema for historical reasons. Since the schemas are not known in advance, `SELECT *` is the practical projection. Retaining the schema avoids a custom UDF solely for removing the extra operation column. There are two possible designs: 1. Add an optional TO_CHANGELOG parameter, for example `include_op_column => FALSE`, to omit the operation column. 2. Keep the PTF API as it is and use the SQL projection that will become available with the Calcite 1.42 upgrade: SELECT * EXCLUDE(op) FROM TO_CHANGELOG(input => TABLE source_table, ...); Omitting the operation column does not make a retract stream append-only in the usual sense. Callers that receive updates or deletes need an appropriate representation or filtering strategy; otherwise different change events can look identical after the operation code is removed. The motivating pipeline handles this separately. While `SELECT * EXCLUDE(op)` would solve this specific use case, I would prefer changing the PTF. Requiring TO_CHANGELOG to change the output schema feels like a sometimes unnecessary restriction. An option with a reasonable default adds little complexity, and most users do not even need to know about it, while it leaves more room for flexibility to retain the input schema with `SELECT *` when needed. WDYT? Best, Ferenc [1] https://github.com/apache/flink/pull/29102
