[
https://issues.apache.org/jira/browse/FLINK-40002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Hughes reassigned FLINK-40002:
----------------------------------
Assignee: Jim Hughes
> `PushFilterInCalcIntoTableSourceScanRule` misroutes metadata predicates,
> crashing compiled-plan restore
> -------------------------------------------------------------------------------------------------------
>
> Key: FLINK-40002
> URL: https://issues.apache.org/jira/browse/FLINK-40002
> Project: Flink
> Issue Type: Bug
> Reporter: Jim Hughes
> Assignee: Jim Hughes
> Priority: Major
>
> When a `Filter` sits above a `Calc` above a table source scan,
> `PushFilterInCalcIntoTableSourceScanRule` handles metadata-column predicates
> incorrectly in two ways:
> **1. Crash on compiled-plan restore**
> For sources that implement both `SupportsFilterPushDown` and
> `SupportsReadingMetadata`, the rule routes all predicates — including
> metadata-column predicates — through the physical
> `resolveFiltersAndCreateTableSourceTable()` path, producing a
> `FilterPushDownSpec` that stores raw `RexInputRef` indices. When
> `ProjectPushDownSpec` later narrows the scan's row type, those indices no
> longer correspond to the right columns. Restoring the compiled plan crashes.
> **2. Missed push-down for metadata-only sources**
> The rule's `matches()` guard only calls `canPushdownFilter()` (physical
> filter check). For sources that implement `SupportsReadingMetadata` but not
> `SupportsFilterPushDown`, the rule never fires at all, silently skipping
> metadata filter push-down.
> **Root cause**
> The metadata-filter-aware classification logic introduced in
> `PushFilterIntoTableSourceScanRule` (the sibling rule for `Filter` directly
> above a scan) was never applied to `PushFilterInCalcIntoTableSourceScanRule`.
> The two rules are structurally parallel but diverged.
> There is also a latent correctness issue in the predicate classification
> shared by both rules: the old heuristic identifies metadata columns by
> `inputRef.getIndex() >= physicalColumnCount`, which breaks when projection
> has already narrowed the scan row type and indices have shifted. A mixed `OR`
> predicate like `id > 0 OR m0 > 0` (referencing both physical and metadata
> columns) that cannot be split by AND-decomposition must stay as a runtime
> Calc filter — it must not enter either the physical or metadata push-down
> path. The old code would route it into the physical path, which crashes on
> restore for the same index-shift reason.
> **Impact**
> Any query involving a `Calc` above a scan against a source implementing
> `SupportsReadingMetadata` that has a metadata-column predicate will either
> crash on compiled-plan restore or silently skip push-down. Sources
> implementing only `SupportsReadingMetadata` (no physical push-down) are
> entirely unoptimized through the Calc path.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
