sunchao opened a new pull request, #59146: URL: https://github.com/apache/spark/pull/59146
### What changes were proposed in this pull request? Mark `SparkPlan.executeColumnarRDD` transient, matching `executeRDD`, so serializing a plan retained by whole-stage code generation does not capture its cached columnar RDD. Add a regression in `SparkPlanSuite` that uses real sort-merge join codegen references, primes the columnar cache with an RDD containing a nonserializable sentinel, and checks driver-side cache reuse, evaluator serialization/deserialization, and deserialized-plan cleanup. Draft pending JIRA linkage and full source-build validation. ### Why are the changes needed? `SortMergeJoinExec` adds its plan to codegen references for executor cleanup. If a descendant columnar plan has initialized its execution cache, serializing the evaluator follows that cached RDD even though the evaluator receives executable inputs separately. This captures unnecessary driver-side objects and can cause `NotSerializableException`. The regression fails before the change through `WholeStageCodegenEvaluatorFactory.references -> SortMergeJoinExec -> ColumnarToRowExec -> executeColumnarRDD -> cached RDD -> sentinel`. Excluding the cache fixes that path while preserving driver-side reuse. ### Does this PR introduce _any_ user-facing change? Yes. Whole-stage evaluator serialization no longer captures a referenced plan's cached columnar execution RDD. Query results and public APIs are unchanged. This fixes the demonstrated capture path; it does not claim to eliminate all sources of large task closures. ### How was this patch tested? - Compiled the exact modified `SparkPlan.scala` and `SparkPlanSuite.scala` from master at `a026a0be4c88e7ace59d1da7c18ab73b0d81edde` against public Spark 4.2.0 dependencies with Scala 2.13.18 and ScalaTest 3.2.16. The new regression passed. - A control removing only `@transient` failed with the expected `NotSerializableException` through `executeColumnarRDD`. - The complete `SparkPlanSuite` in this class overlay passed 12 of 13 tests. The existing SPARK-58120 `BatchScanExec` canonicalization test failed identically with unmodified `SparkPlan`; the cached 4.2.0 `BatchScanExec` predates that master fix. This is not a full current-master build. - Scoped Scalastyle with the repository configuration passed for both modified files: zero errors and warnings. `git diff --check` passed. - Attempted `build/sbt 'sql/testOnly org.apache.spark.sql.execution.SparkPlanSuite'`. Project loading was blocked by missing BOM dependencies and a read-only local Maven cache. Full source-build validation remains outstanding. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-6). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
