[
https://issues.apache.org/jira/browse/SPARK-61668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh updated SPARK-61668:
--------------------------------
Affects Version/s: 4.4.0
(was: 5.0.0)
> Avoid serializing the child plans into every task from ShuffledHashJoinExec
> codegen
> -----------------------------------------------------------------------------------
>
> Key: SPARK-61668
> URL: https://issues.apache.org/jira/browse/SPARK-61668
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: L. C. Hsieh
> Priority: Major
>
> Follow-up of SPARK-61661, which stopped HashAggregateExec whole-stage codegen
> from referencing the plan node itself, so its child subtree is no longer
> serialized into every task binary.
> ShuffledHashJoinExec codegen still calls ctx.addReferenceObj("plan", this) to
> call buildHashedRelation from the generated code, so the operator and both of
> its child subtrees are serialized into every task of the stage.
> buildHashedRelation needs an audit of what it reads (build keys, build plan
> output and metrics) so that the generated code can reference a small helper
> instead.
> The SortExec / SortMergeJoinExec / SortMergeAsOfJoinExec instances of the
> same pattern are tracked in SPARK-61666, which also covers adding a test-time
> guard against SparkPlan objects in what codegen stages ship to tasks once
> both are fixed.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]