[
https://issues.apache.org/jira/browse/SPARK-28520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takeshi Yamamuro resolved SPARK-28520.
--------------------------------------
Resolution: Fixed
Fix Version/s: 3.0.0
Target Version/s: (was: 3.0.0)
Resolved by
[https://github.com/apache/spark/pull/25260|https://github.com/apache/spark/pull/25260#issuecomment-515752501]
> WholeStageCodegen does not work property for LocalTableScanExec
> ---------------------------------------------------------------
>
> Key: SPARK-28520
> URL: https://issues.apache.org/jira/browse/SPARK-28520
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Kousuke Saruta
> Assignee: Kousuke Saruta
> Priority: Minor
> Fix For: 3.0.0
>
>
> Code is not generated for LocalTableScanExec although proper situations.
> If a LocalTableScanExec plan has the direct parent plan which supports
> WholeStageCodegen,
> the LocalTableScanExec plan also should be within a WholeStageCodegen domain.
> But code is not generated for LocalTableScanExec and InputAdapter is inserted
> for now.
> {code}
> val df1 = spark.createDataset(1 to 10).toDF
> val df2 = spark.createDataset(1 to 10).toDF
> val df3 = df1.join(df2, df1("value") === df2("value"))
> df3.explain(true)
> ...
> == Physical Plan ==
> *(1) BroadcastHashJoin [value#1], [value#6], Inner, BuildRight
> :- LocalTableScan [value#1] //
> LocalTableScanExec is not within a WholeStageCodegen domain
> +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int,
> false] as bigint)))
> +- LocalTableScan [value#6]
> {code}
> {code}
> scala> df3.queryExecution.executedPlan.children.head.children.head.getClass
> res4: Class[_ <: org.apache.spark.sql.execution.SparkPlan] = class
> org.apache.spark.sql.execution.InputAdapter
> {code}
> For the current implementation of LocalTableScanExec, codegen is enabled in
> case `parent` is not null
> but `parent` is set in `consume`, which is called after `insertInputAdapter`
> so it doesn't work as intended.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
