Hi Jim,
thanks for spotting this and coming up with a solution. I agree that the
combination of different ability interfaces was not well tested. The
interface looks clean to me and fits into the existing ones.
+1 for this proposal.
Thanks,
Timo
On 10.04.26 21:40, Jim Hughes via dev wrote:
Hi all,
I'd like to open a discussion on FLIP-574: Metadata Filter Push-Down for
Table Sources. See
https://cwiki.apache.org/confluence/display/FLINK/FLIP-574+Metadata+Filter+Push-Down+for+Table+Sources
Predicates on metadata columns (e.g., Kafka offset, timestamp, partition)
cannot be pushed through the existing SupportsFilterPushDown path because
FilterPushDownSpec's serialized indices break during compiled plan
restoration when ProjectPushDownSpec narrows the row type.
This FLIP adds a dedicated metadata filter push-down path via new default
methods on SupportsReadingMetadata, with a separate
MetadataFilterPushDownSpec that snapshots the row type at optimization
time. Existing sources are unaffected (opt-in design).
A PR is available at https://github.com/apache/flink/pull/27913 to show how
this would work.
Looking forward to your feedback.
Thanks in advance,
Jim
