xuechendi commented on a change in pull request #34396:
URL: https://github.com/apache/spark/pull/34396#discussion_r747235627
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala
##########
@@ -458,6 +462,34 @@ case class RowToColumnarExec(child: SparkPlan) extends
RowToColumnarTransition {
// This avoids calling `schema` in the RDD closure, so that we don't need
to include the entire
// plan (this) in the closure.
val localSchema = this.schema
+ if (enableArrowColumnVector) {
+ val maxRecordsPerBatch = SQLConf.get.arrowMaxRecordsPerBatch
+ val timeZoneId = SQLConf.get.sessionLocalTimeZone
+ return child.execute().mapPartitionsInternal { rowIterator =>
+ val context = TaskContext.get()
+ val allocator = ArrowUtils.getDefaultAllocator
+ val bytesIterator = ArrowConverters
+ .toBatchIterator(rowIterator, localSchema, maxRecordsPerBatch,
timeZoneId, context)
Review comment:
@HyukjinKwon , yes, That is what we do, we have some implemented
operators and plug into spark using Spark Extension configuration +
columnarRules to replace original row physical plans, that is why we need
RowToColumnarExec and ColumnarToRowExec to do InternalRow and
ArrowColumnarBatch conversion. I think one difference with mapInArrow(scala
version) is we want to leverage the built-in attribute supportColumnar to avoid
code changes to spark application.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
