Jim Hughes created FLINK-40002:
----------------------------------
Summary: `PushFilterInCalcIntoTableSourceScanRule` misroutes
metadata predicates, crashing compiled-plan restore
Key: FLINK-40002
URL: https://issues.apache.org/jira/browse/FLINK-40002
Project: Flink
Issue Type: Bug
Reporter: Jim Hughes
When a `Filter` sits above a `Calc` above a table source scan,
`PushFilterInCalcIntoTableSourceScanRule` handles metadata-column predicates
incorrectly in two ways:
**1. Crash on compiled-plan restore**
For sources that implement both `SupportsFilterPushDown` and
`SupportsReadingMetadata`, the rule routes all predicates — including
metadata-column predicates — through the physical
`resolveFiltersAndCreateTableSourceTable()` path, producing a
`FilterPushDownSpec` that stores raw `RexInputRef` indices. When
`ProjectPushDownSpec` later narrows the scan's row type, those indices no
longer correspond to the right columns. Restoring the compiled plan crashes.
**2. Missed push-down for metadata-only sources**
The rule's `matches()` guard only calls `canPushdownFilter()` (physical filter
check). For sources that implement `SupportsReadingMetadata` but not
`SupportsFilterPushDown`, the rule never fires at all, silently skipping
metadata filter push-down.
**Root cause**
The metadata-filter-aware classification logic introduced in
`PushFilterIntoTableSourceScanRule` (the sibling rule for `Filter` directly
above a scan) was never applied to `PushFilterInCalcIntoTableSourceScanRule`.
The two rules are structurally parallel but diverged.
There is also a latent correctness issue in the predicate classification shared
by both rules: the old heuristic identifies metadata columns by
`inputRef.getIndex() >= physicalColumnCount`, which breaks when projection has
already narrowed the scan row type and indices have shifted. A mixed `OR`
predicate like `id > 0 OR m0 > 0` (referencing both physical and metadata
columns) that cannot be split by AND-decomposition must stay as a runtime Calc
filter — it must not enter either the physical or metadata push-down path. The
old code would route it into the physical path, which crashes on restore for
the same index-shift reason.
**Impact**
Any query involving a `Calc` above a scan against a source implementing
`SupportsReadingMetadata` that has a metadata-column predicate will either
crash on compiled-plan restore or silently skip push-down. Sources implementing
only `SupportsReadingMetadata` (no physical push-down) are entirely unoptimized
through the Calc path.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
