pan3793 opened a new pull request, #58524: URL: https://github.com/apache/spark/pull/58524
### What changes were proposed in this pull request? In `DataSourceV2Strategy.translateRuntimeFilterV2`, match an `InSubqueryExec` whose subquery result is empty before the pushable-column check, and translate it to `AlwaysFalse`. `expr IN ()` is false for every row regardless of `expr`, so the translation is valid even when `expr` itself is untranslatable, e.g. a join key wrapped in a cast inserted by type coercion (`cast(col as bigint) IN dynamicpruning#x`). The test connector `InMemoryTableWithV2Filter` is updated to honor `AlwaysFalse` by pruning all partitions. ### Why are the changes needed? When the build side of a DPP join produces no rows, the runtime filter degenerates to `key IN ()`. Today this is either emitted as a zero-value `IN` predicate that connectors cannot interpret meaningfully (Spark's own `InMemoryTableWithV2Filter` ignores it: its `filter()` requires `children().length > 1`), or, when the key is wrapped in a cast, dropped entirely with "Can't translate ... unsupported expression". Either way the scan reads every partition to produce zero output rows. Translating to `AlwaysFalse` prunes everything and makes the connector contract explicit: connectors receive a well-defined predicate instead of a degenerate IN. ### Does this PR introduce _any_ user-facing change? No correctness change. DSv2 scans receiving an empty runtime IN filter can now prune all partitions instead of reading everything. ### How was this patch tested? New tests in `DynamicPartitionPruningV2FilterSuite` covering a bare pruning key and a cast-wrapped pruning key; both assert `filteredPartitions` is empty. Previously all 25 partitions were read. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Fable 5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
