szehon-ho opened a new pull request, #58296: URL: https://github.com/apache/spark/pull/58296
### What changes were proposed in this pull request? `SupportsRuntimeCatalystFiltering` was added by SPARK-58523, which covered row-level operations (`RowLevelOperationCatalystRuntimeFilterSuiteBase` and its group- and delta-based subclasses) and a core set of pushdown tests. It was not exercised in the two other places the predicate-based runtime filtering interfaces are: dynamic partition pruning and storage-partitioned joins. This PR closes that gap by reusing the existing suites rather than cloning them. **Test fixtures** - `InMemoryBaseTable`: `CatalystRuntimeFilteringScan` now prunes on nested partition keys. It previously looked up a partition attribute by joining `fieldNames` into a single top-level name, so a `GetStructField` chain such as `s.part` never matched and no pruning happened. A `partitionKeyPath` helper now reads the dotted path out of the expression, and partition attributes are keyed by that path. - `InMemoryCatalystRuntimeFilterTable`: takes `numRowsPerSplit`, so a partition key can have several splits (needed for the partially-clustered SPJ test), and derives `filterAttributes()` / `fullyPushedFilterAttributes()` from a shared helper. - `InMemoryTableCatalystRuntimeFilterCatalog`: adds `InMemoryCatalystRuntimeFilterCatalog`, the `InMemoryCatalog` counterpart, so the Catalyst fixture can be used where functions and procedures are needed. - `InMemoryTableWithV2Filter`: threads `numRowsPerSplit` through to `InMemoryBaseTable`. **Reused suites** - `DynamicPartitionPruningSuite`: adds `DynamicPartitionPruningV2CatalystFilterSuiteAEOff` / `AEOn`, mirroring the existing `DynamicPartitionPruningV2FilterSuite` pair. - `KeyGroupedPartitioningSuite`: the shared SPJ fixtures move to a new `KeyGroupedPartitioningSuiteBase`, and the three runtime-filtering tests move to a `KeyGroupedPartitioningRuntimeFilterTests` trait. Two small suites then run those three tests once per interface: `KeyGroupedPartitioningRuntimeFilterSuite` (predicate-based, the default `InMemoryCatalog`) and `KeyGroupedPartitioningCatalystRuntimeFilterSuite`. - `DistributionAndOrderingSuiteBase`: `catalogClassName` becomes overridable so a subclass can vary the catalog behind `testcat`. Note for reviewers: `SPARK-42038: partially clustered: with dynamic partition filtering` and `SPARK-45652: SPJ should handle empty partition after dynamic filtering` are unchanged, but now report under `KeyGroupedPartitioningRuntimeFilterSuite` instead of `KeyGroupedPartitioningSuite`. Most of the diff in that file is this movement. **New tests in `DataSourceV2CatalystRuntimeFilterSuite`** Covering behavior specific to pushing Catalyst expressions: a DPP filter on a nested partition field arriving with the nested access intact, multiple predicates pushed in a single `filter()` call, a filter with no V2 translation being pushed instead of dropped, a scan implementing both runtime filtering interfaces being rejected, and the `filterAttributes()` contract when a reported attribute is not a top-level scan attribute. The last of these pins current behavior rather than asserting an error: a nested reference over a struct resolves and widens to the enclosing struct attribute, so declaring `s.tz` makes filters over every field of `s` eligible. This matches the two predicate-based interfaces, which resolve the same way. One test calls `PushDownUtils.replanWithRuntimeFilters` directly, to reach the SPJ partitioning-preservation checks that a well-behaved source cannot trigger: dropping `HasPartitionKey`, reporting a partition key that was not in the original partitioning, or growing a key's split count. ### Why are the changes needed? `SupportsRuntimeCatalystFiltering` is the path a scan takes when runtime filters are pushed as Catalyst expressions instead of connector predicates. Dynamic partition pruning and storage-partitioned joins are the two features that produce those filters, and neither was tested against this interface, so regressions in the Catalyst path would not have been caught by the suites that cover the equivalent predicate-based path. ### Does this PR introduce _any_ user-facing change? No. This is test-only; every changed file is under `src/test/`. ### How was this patch tested? Test-only change. Locally, on the rebased branch: - `DataSourceV2CatalystRuntimeFilterSuite`, `KeyGroupedPartitioningRuntimeFilterSuite`, `KeyGroupedPartitioningCatalystRuntimeFilterSuite`, `DynamicPartitionPruningV2CatalystFilterSuiteAEOff`, `DynamicPartitionPruningV2CatalystFilterSuiteAEOn`: 98 tests passed. - `KeyGroupedPartitioningSuite`: 96 tests passed. It had 99 before this change; the difference is the three runtime-filtering tests that moved to `KeyGroupedPartitioningRuntimeFilterSuite`, which passes 3. - `DynamicPartitionPruningV2FilterSuiteAEOff` and `WriteDistributionAndOrderingSuite`, as regression checks on the shared fixture and base-class changes: 215 tests passed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor with Claude Opus 5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
