pan3793 opened a new pull request, #58580:
URL: https://github.com/apache/spark/pull/58580
### What changes were proposed in this pull request?
Type coercion of the join keys may wrap the pruning key of dynamic partition
pruning (DPP) in a
cast, e.g. `cast(f.store_id as bigint) IN dynamicpruning#1` when an INT
partition column joins a
BIGINT key. `DataSourceV2Strategy.translateRuntimeFilterV2` only translated
a bare (nested)
column, so such a filter was never pushed to the source and the scan read
all partitions.
This PR translates every `InSubqueryExec` runtime filter by building
`InSet(child, values)` from
the subquery result and unwrapping the cast with
`UnwrapCastInBinaryComparison.unwrapCast`, the
same code the optimizer applies to `InSet` with literal values. The result
is then translated:
- `InSet(col, values)` becomes `IN(col, values)` with the values converted
to the column type.
Values that do not round-trip (out of range, rounded) are dropped and
nulls are kept, exactly
as the rule does, so the supported casts are the rule's (numeric and
boolean upcasts, excluding
the lossy INT to FLOAT and LONG to FLOAT/DOUBLE).
- `if(isnull(col), null, false)`, returned by the rule when no value is
representable in the
column type, becomes `AlwaysFalse`, as no row can match.
- Anything else stays untranslated, as before.
`UnwrapCastInBinaryComparison.unwrapCast` becomes `private[sql]` for this.
The rule itself is
unchanged.
### Why are the changes needed?
A DSv2 scan showed `dynamicpruningexpression(cast(store_id as bigint) IN
...)` in the plan, yet
the runtime filter never reached the source, so all partitions were scanned.
### Does this PR introduce _any_ user-facing change?
Yes. DSv2 sources now receive runtime IN filters whose pruning key is
wrapped in a lossless
numeric cast, so the scan prunes partitions instead of reading all of them.
Query results do not
change.
### How was this patch tested?
- New unit tests for `translateRuntimeFilterV2` in
`DataSourceV2StrategySuite`: bare and nested
columns, cast unwrapping with out-of-range, rounded and null values, all
values dropped, and
lossy casts that must not be unwrapped.
- New end-to-end test in `DynamicPartitionPruningV2Suite`, which runs
against the in-memory
tables taking runtime filters as V1 filters, V2 predicates and Catalyst
expressions, with AQE
on and off.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5.1
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01U871Pf4hhRfc3197gQMBc1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
