AHeise opened a new pull request, #29385:
URL: https://github.com/apache/flink/pull/29385
## What is the purpose of the change
Predicates on an element of an ARRAY, MAP or VARIANT column never reach
`SupportsFilterPushDown#applyFilters`:
```sql
SELECT * FROM src WHERE a[1] = 3;
SELECT * FROM src WHERE m['k'] = 3;
SELECT * FROM src WHERE CAST(v['k'] AS INT) = 3;
```
Element access is Calcite's `ITEM`. `RexNodeToExpressionConverter#visitCall`
resolves operators it does not map explicitly by name, finds no built-in
function called `ITEM` (the counterpart is `BuiltInFunctionDefinitions.AT`,
named `at`), and classifies the conjunct as unconvertible, so it stays in the
Calc above the scan. The reverse mapping `AT -> ITEM` already exists in
`DirectConvertRule`.
## Brief change log
- `RexNodeToExpressionConverter` converts `ITEM` to `AT` when the first
operand is ARRAY, MAP or VARIANT, the types `AT` accepts. `ITEM` on a ROW or
structured type (`r['f']`) stays unconvertible.
- The values test source (`FilterUtils`) can push down and evaluate `AT` on
maps, arrays and variants, plus a `CAST`/`TRY_CAST` directly around `AT`. Its
value getters return the raw field instead of casting to `Comparable`, so
containers can be navigated.
## Verifying this change
This change added tests and can be verified as follows:
- `RexNodeExtractorTest#testExtractItem`: ARRAY, MAP and VARIANT element
predicates convert to `at(...)`; a ROW string-key predicate and the 0-based
`arr[OFFSET(0)]` (`SqlLibraryOperators.OFFSET`, also `SqlKind.ITEM`) stay
unconverted.
- `PushFilterIntoTableSourceScanRuleTest`: `testFilterOnArrayElement`,
`testFilterOnMapValue`, `testFilterOnVariantField` and
`testFilterOnNestedArrayElement` (`nested.arr[1]`) show the predicate in the
scan's `filter=[...]`.
- `TableSourceScanRestoreTest`: new program
`FILTER_PUSHDOWN_ON_COLLECTION_ELEMENTS` pushes all three predicates and
restores the compiled plan from a savepoint, which re-converts the stored
`ITEM` predicates. It is skipped for node version 1 because a version-1 plan
cannot be generated any more (same as `TableSinkRestoreTest`). Without the
converter change the restore fails with `>(ITEM($1, 1), 1) can not be converted
to Expression`.
- Existing table source ITs with element filters on filterable nested fields
(`TableSourceITCase#testNestedFilterOnMap`, `#testNestedFilterOnArray`) still
pass.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
Sources implementing `SupportsFilterPushDown` (and `SupportsDeletePushDown`
for `DELETE ... WHERE arr[1] = 3`) start receiving `AT` calls. A source that
does not recognize a call keeps it as remaining, so results do not change, but
plans of sources that accept such filters do.
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code (Claude Opus 5.5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
