Arvid Heise created FLINK-40267:
-----------------------------------
Summary: CREATE MATERIALIZED TABLE skips persisted-metadata-column
validation applied by CREATE OR ALTER
Key: FLINK-40267
URL: https://issues.apache.org/jira/browse/FLINK-40267
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Affects Versions: 2.3.0, 2.2.0, 2.4.0
Reporter: Arvid Heise
Assignee: Arvid Heise
Fix For: 2.4.0
SqlCreateOrAlterMaterializedTableConverter#getMergedSchema selects one of two
validators for the same invariant — every persisted column in the schema part
must be produced by the query part:
- CREATE OR ALTER → MaterializedTableUtils#validatePersistedColumnsUsedByQuery:
checks physical columns and non-virtual metadata columns, unwraps
SqlTableColumnPosition.
- plain CREATE → converter-private validatePhysicalColumnsUsedByQuery:
continues on anything that is not a SqlRegularColumn.
A CREATE MATERIALIZED TABLE declaring a persisted metadata column absent from
the AS SELECT is therefore accepted. The materialized table is created, and the
refresh pipeline fails at compile time in DynamicSinkUtils with "Column types
of query result and sink do not match / Different number of columns", because
persisted metadata columns are part of the sink's consumed row type.
{noformat}
CREATE TABLE src (order_id STRING NOT NULL, price DOUBLE);
CREATE MATERIALIZED TABLE mt (
`order_id` STRING NOT NULL,
`price` DOUBLE,
`kafka_ts` TIMESTAMP_LTZ(3) METADATA FROM 'timestamp'
) AS SELECT order_id, price FROM src;
{noformat}
The identical statement written as CREATE OR ALTER on a non-existing table is
rejected synchronously with a precise message, which contradicts the FLIP-546
equivalence of the two forms.
The split originates in FLINK-38674 (physical-only check, added inline for
schema definition on CREATE) and FLINK-38766 (persisted check, added for ALTER
MATERIALIZED TABLE ADD). FLINK-39700 extracted the former and routed only
CREATE OR ALTER to the latter.
Fix: call validatePersistedColumnsUsedByQuery on both branches and drop
validatePhysicalColumnsUsedByQuery (strict superset). Parameterize the
statement kind in PERSISTED_COLUMN_NOT_USED_IN_QUERY, whose prefix is hardcoded
to "Failed to execute ALTER MATERIALIZED TABLE statement." and already misfires
on the CREATE OR ALTER path. Update the message assertions in
SqlMaterializedTableNodeToOperationConverterTest and add CREATE-side coverage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
