Jim Hughes created FLINK-39421:
----------------------------------
Summary: Metadata filter push-down for table sources
Key: FLINK-39421
URL: https://issues.apache.org/jira/browse/FLINK-39421
Project: Flink
Issue Type: Improvement
Components: Table SQL / Planner
Reporter: Jim Hughes
Predicates on metadata columns (e.g., Kafka offset, timestamp, partition) are
excluded from the existing SupportsFilterPushDown mechanism. FilterPushDownSpec
serializes RexInputRef indices relative to the scan's produced row type, but
metadata columns are appended after physical columns. When ProjectPushDownSpec
narrows the row type during compiled plan restoration, metadata indices become
out-of-bounds, causing ArrayIndexOutOfBoundsException.
This ticket introduces a dedicated metadata filter push-down path:
- New supportsMetadataFilterPushDown() and applyMetadataFilters() default
methods on SupportsReadingMetadata (opt-in; existing sources unaffected)
- New MetadataFilterPushDownSpec that snapshots the row type at optimization
time, solving the index mismatch
- Extension to PushFilterIntoTableSourceScanRule to classify predicates as
physical, metadata, or mixed
See FLIP-574
(https://cwiki.apache.org/confluence/display/FLINK/FLIP-574+Metadata+Filter+Push-Down+for+Table+Sources)
for full design.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
