revans2 commented on a change in pull request #25008: [SPARK-28213][SQL]
Replace ColumnarBatchScan with equivilant from Columnar
URL: https://github.com/apache/spark/pull/25008#discussion_r307775259
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/LogicalPlanTagInSparkPlanSuite.scala
##########
@@ -44,9 +44,14 @@ class LogicalPlanTagInSparkPlanSuite extends
TPCDSQuerySuite {
}
// A scan plan tree is a plan tree that has a leaf node under zero or more
Project/Filter nodes.
- private def isScanPlanTree(plan: SparkPlan): Boolean = plan match {
- case p: ProjectExec => isScanPlanTree(p.child)
- case f: FilterExec => isScanPlanTree(f.child)
+ // Because of how codegen and columnar to row transitions work, we may have
InputAdaptors
+ // and ColumnarToRow transformations in the middle of it, but they will not
have the tag
+ // we want, so skip them if they are the first thing we see
+ private def isScanPlanTree(plan: SparkPlan, first: Boolean): Boolean = plan
match {
+ case i: InputAdapter if !first => isScanPlanTree(i.child, false)
Review comment:
Prior to this PR, a subclass of ColumanarBatchScan would be in a plan that
looked like.
```
INPUT (subclass of ColumnarBatchScan) -> (code gen supported nodes) ->
WholeStageCodegenExec -> ...
```
After this change, it now looks like
```
INPUT (not a subclass of ColumnarBatchScan) -> InputAdapter ->
ColumnarToRowExec -> (code gen supported nodes) -> WholeStageCodegenExec -> ...
```
Because the INPUT class no longer supports code generation the code
generation rule will insert an InputAdapter after it and before the
ColumnarToRowExec that does support code generation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
