szehon-ho opened a new pull request, #58356: URL: https://github.com/apache/spark/pull/58356
### What changes were proposed in this pull request? This backports #58296 to `branch-4.3`. The patch improves test coverage for `SupportsRuntimeCatalystFiltering` in dynamic partition pruning and storage-partitioned joins, and rejects nested runtime-filter attributes that violate the connector contract. The cherry-pick had one conflict in `KeyGroupedPartitioningSuite.scala`, resolved to preserve the 4.3 branch's existing test organization and tagging while adding the new runtime-filter suites. ### Why are the changes needed? Spark 4.3 contains `SupportsRuntimeCatalystFiltering`, but lacks the coverage and contract validation added by #58296. Backporting the change prevents regressions in Catalyst-expression runtime filtering and rejects invalid nested filter attributes consistently. ### Does this PR introduce _any_ user-facing change? No new user-facing change beyond #58296. For connector authors, a scan that declares a nested runtime-filter attribute is rejected with an internal error instead of silently widening it to the enclosing top-level column. ### How was this patch tested? Run locally with `SPARK_LOCAL_IP=127.0.0.1`: ```text build/sbt 'sql/testOnly org.apache.spark.sql.connector.DataSourceV2CatalystRuntimeFilterSuite org.apache.spark.sql.DynamicPartitionPruningV2FilterSuiteAEOff org.apache.spark.sql.DynamicPartitionPruningV2FilterSuiteAEOn org.apache.spark.sql.DynamicPartitionPruningV2CatalystFilterSuiteAEOff org.apache.spark.sql.DynamicPartitionPruningV2CatalystFilterSuiteAEOn org.apache.spark.sql.connector.KeyGroupedPartitioningSuite org.apache.spark.sql.connector.KeyGroupedPartitioningRuntimeFilterSuite org.apache.spark.sql.connector.KeyGroupedPartitioningCatalystRuntimeFilterSuite' ``` 277 tests succeeded, 0 failed, and 2 were ignored. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor with Claude Opus 5 for the original patch. Generated-by: OpenAI Codex with GPT-5 for the backport and conflict resolution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
