[
https://issues.apache.org/jira/browse/FLINK-40541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-40541:
-----------------------------------
Labels: pull-request-available (was: )
> LATERAL SNAPSHOT join rejects a valid build-side watermark when it is
> declared on a column hidden by the column-expansion strategy
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-40541
> URL: https://issues.apache.org/jira/browse/FLINK-40541
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
> Reporter: Fabian Hueske
> Assignee: Fabian Hueske
> Priority: Major
> Labels: pull-request-available
>
> *Description*
> The {{LATERAL SNAPSHOT}} join requires the build-side input to declare
> exactly one watermark (row-time attribute), which drives the operator's LOAD
> phase. The planner detects this watermark by counting row-time indicator
> fields in the row type of the projected {{TABLE}} argument.
> When the watermark is declared on a column that the configured
> column-expansion strategy hides from {{SELECT *}} — e.g. a {{VIRTUAL}}
> metadata column combined with
> {{TableConfigOptions.TABLE_COLUMN_EXPANSION_STRATEGY =
> EXCLUDE_DEFAULT_VIRTUAL_METADATA_COLUMNS}} — the {{TABLE t}} argument expands
> to a projection that drops the row-time attribute. The row-time count is then
> {{0}}, and the query is wrongly rejected with:
> {code}
> org.apache.flink.table.api.ValidationException: LATERAL SNAPSHOT requires a
> watermark on the build-side input.
> {code}
> even though the table does declare a watermark.
> *How to reproduce*
> {code:sql}
> SET 'table.column-expansion-strategy' =
> 'EXCLUDE_DEFAULT_VIRTUAL_METADATA_COLUMNS';
> CREATE TABLE build_side (
> bk STRING,
> bv INT,
> rt TIMESTAMP_LTZ(3) METADATA VIRTUAL,
> WATERMARK FOR rt AS rt
> ) WITH (...);
> SELECT * FROM probe
> JOIN LATERAL SNAPSHOT(
> input => TABLE build_side,
> load_completed_condition => 'user_time',
> load_completed_time => CAST(TIMESTAMP '2026-07-01 00:00:00' AS
> TIMESTAMP_LTZ(3))
> ) AS s ON probe.pk = s.bk;
> {code}
> Explicitly re-projecting the hidden column (e.g. {{SELECT *, rt FROM
> build_side}} in a preceding CTE/view) works around the problem, confirming
> that the watermark itself is present and only the visibility of the row-time
> column is at fault.
> *Proposed fix*
> Detect the build-side row-time attribute from the watermark below any
> projection that hides it, rather than from a visible time-attribute column,
> and re-append the row-time attribute from below the hiding projection so it
> reaches the operator. Run the watermark validation on the rewritten build
> side instead of the raw {{TABLE}} argument. The existing top-level {{Calc}}
> wrapper projects the re-appended column away, so the user-visible output is
> unchanged.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
