L. C. Hsieh created SPARK-61666:
-----------------------------------
Summary: Avoid serializing the child plan into every task from
SortExec and join operators codegen
Key: SPARK-61666
URL: https://issues.apache.org/jira/browse/SPARK-61666
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
Follow-up of SPARK-61661, which stopped HashAggregateExec whole-stage codegen
from referencing the plan node itself, so its child subtree is no longer
serialized into every task binary.
The same ctx.addReferenceObj("plan", this) pattern remains in:
- SortExec (createSorter)
- SortMergeJoinExec (cleanupResources)
- SortMergeAsOfJoinExec (cleanupResources)
- ShuffledHashJoinExec (buildHashedRelation)
Each of these serializes the operator and its whole child subtree into every
task of the stage. This is not a mechanical port of SPARK-61661:
SortExec.createSorter writes the mutable rowSorter field on the node, and
SortMergeJoinExec/SortMergeAsOfJoinExec.cleanupResources reach that same field
through their children. That works today only because both "plan" references
deserialize to the same object within one task binary, so the shared sorter
state has to move together with the helpers.
ShuffledHashJoinExec.buildHashedRelation needs its own audit.
Once these are fixed, add a test-time guard that rejects SparkPlan objects in
what any WholeStageCodegenExec stage ships to tasks (its closure / evaluator
factory and the generated code references), so the pattern does not come back.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]