L. C. Hsieh created SPARK-61666:
-----------------------------------

             Summary: Avoid serializing the child plan into every task from 
SortExec and join operators codegen
                 Key: SPARK-61666
                 URL: https://issues.apache.org/jira/browse/SPARK-61666
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: L. C. Hsieh


Follow-up of SPARK-61661, which stopped HashAggregateExec whole-stage codegen 
from referencing the plan node itself, so its child subtree is no longer 
serialized into every task binary.

The same ctx.addReferenceObj("plan", this) pattern remains in:
- SortExec (createSorter)
- SortMergeJoinExec (cleanupResources)
- SortMergeAsOfJoinExec (cleanupResources)
- ShuffledHashJoinExec (buildHashedRelation)

Each of these serializes the operator and its whole child subtree into every 
task of the stage. This is not a mechanical port of SPARK-61661: 
SortExec.createSorter writes the mutable rowSorter field on the node, and 
SortMergeJoinExec/SortMergeAsOfJoinExec.cleanupResources reach that same field 
through their children. That works today only because both "plan" references 
deserialize to the same object within one task binary, so the shared sorter 
state has to move together with the helpers. 
ShuffledHashJoinExec.buildHashedRelation needs its own audit.

Once these are fixed, add a test-time guard that rejects SparkPlan objects in 
what any WholeStageCodegenExec stage ships to tasks (its closure / evaluator 
factory and the generated code references), so the pattern does not come back.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to