revans2 commented on a change in pull request #25008: [SPARK-28213][SQL]
Replace ColumnarBatchScan with equivilant from Columnar
URL: https://github.com/apache/spark/pull/25008#discussion_r301815320
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
##########
@@ -1293,7 +1293,8 @@ class SubquerySuite extends QueryTest with
SharedSQLContext {
checkAnswer(df, Seq(Row(0, 0), Row(2, 0)))
// need to execute the query before we can examine fs.inputRDDs()
assert(df.queryExecution.executedPlan match {
- case WholeStageCodegenExec(fs @ FileSourceScanExec(_, _, _,
partitionFilters, _, _, _)) =>
+ case WholeStageCodegenExec(ColumnarToRowExec(InputAdapter(
Review comment:
WholeStageCodeGen marks the end of a code generation stage. InputAdapter
marks the beginning of a code generation stage. So what we had before was a
WholeStageCodeGen that had it's first entry a FileSourceScanExec because before
this change FileSourceScanExec supported code generation to convert
ColumnarBatchs into rows. The InputAdapter would logically have been a child
of FileSourceScanExec, but it has no children so it is not there.
After this change ColumnarToRowExec is the only thing in the code generation
stage, so it is flanked by the WholeStageCodegenExec and the InputAdaptor.
FileSourceScanExec is returning batches and is not doing code gen because it is
not needed any longer.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
