hussein-awala opened a new pull request, #59340: URL: https://github.com/apache/spark/pull/59340
### What changes were proposed in this pull request? Use `PLAN_EXPRESSION` to prune the subquery-expression scan in `DeduplicateRelations` at both the plan and expression levels. The relation-renewal traversal, attribute remapping, processing order and subquery recursion are unchanged. The transform retains the default `UnknownRuleId`: the closure depends on mutable traversal state, so ineffective-rule caching would be unsafe. Add five focused Catalyst regression tests and a small, generic rule microbenchmark. JIRA: https://issues.apache.org/jira/browse/SPARK-60135 ### Why are the changes needed? The existing scan calls `transformExpressions` even though its only case matches `SubqueryExpression`. Plans and expression branches without subqueries cannot match that case. The existing tree-pattern metadata lets the rule avoid this work. This is a narrow optimization, not conditional skipping of the complete deduplication rule. ### Does this PR introduce _any_ user-facing change? No. Query semantics and public APIs are unchanged. ### How was this patch tested? Local JDK 21.0.8 validation: ```sh HEAP_SIZE=4g SERIAL_SBT_TESTS=1 build/sbt -batch -no-colors \ -J-Xmx4g -J-XX:ActiveProcessorCount=2 \ 'set LocalProject("catalyst") / Test / javaOptions += "-XX:ActiveProcessorCount=2"' \ 'catalyst/testOnly *DeduplicateRelationsSuite *AnalysisSuite *ResolveSubquerySuite' \ 'catalyst/scalaStyleOnCompile' 'catalyst/scalaStyleOnTest' ``` - 509 tests passed across 11 suites, including all five new tests. - Both Catalyst scalastyle checks passed. - An upstream-source negative control failed the two pruning assertions, while semantic tests passed. - A temporary concrete-rule-ID mutation failed the shared-subquery test, checking that stateful renewal must not use ineffective-rule memoization. That mutation was removed before final validation. The synthetic benchmark constructs projections containing arithmetic expressions, with either no subquery or one scalar subquery. Setup is outside timing; repeated calls reuse the plan and warm its pattern bits. Run it with: ```sh HEAP_SIZE=4g build/sbt \ 'catalyst/Test/runMain org.apache.spark.sql.catalyst.analysis.DeduplicateRelationsBenchmark' ``` Local best nanoseconds per rule invocation: | Columns | Shape | Original | Candidate | |---:|---|---:|---:| | 100 | No subqueries | 38252.5 | 14195.0 | | 100 | One scalar subquery | 41162.9 | 17480.4 | | 1000 | No subqueries | 431890.8 | 180462.5 | | 1000 | One scalar subquery | 435289.2 | 195271.3 | These are local synthetic microbenchmark observations, not end-to-end query speedups. The full Spark test matrix and standardized CI benchmark workflow have not been run. This PR is intentionally a draft. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Pi Coding Agent 1.1.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
