revans2 commented on a change in pull request #25008: [SPARK-28213][SQL]
Replace ColumnarBatchScan with equivilant from Columnar
URL: https://github.com/apache/spark/pull/25008#discussion_r302127555
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##########
@@ -141,11 +148,11 @@ case class FileSourceScanExec(
optionalBucketSet: Option[BitSet],
dataFilters: Seq[Expression],
override val tableIdentifier: Option[TableIdentifier])
- extends DataSourceScanExec with ColumnarBatchScan {
+ extends DataSourceScanExec {
Review comment:
Correct, but there were only 2 things that the code gen was doing. Either
convert ColumnarBatch into UnsafeRows or to convert whatever other rows were
being returned by the DataSourceScanExec into UnsafeRows. The ColumnarBatch
conversion is now covered by ColumnarToRowExec. The row to row conversion is
covered by UnsafeProjections that are either inserted as a part of this patch
or were already in the code, so we ended up doing a double conversion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
