HyukjinKwon commented on a change in pull request #34396:
URL: https://github.com/apache/spark/pull/34396#discussion_r747972652
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala
##########
@@ -458,6 +462,34 @@ case class RowToColumnarExec(child: SparkPlan) extends
RowToColumnarTransition {
// This avoids calling `schema` in the RDD closure, so that we don't need
to include the entire
// plan (this) in the closure.
val localSchema = this.schema
+ if (enableArrowColumnVector) {
+ val maxRecordsPerBatch = SQLConf.get.arrowMaxRecordsPerBatch
+ val timeZoneId = SQLConf.get.sessionLocalTimeZone
+ return child.execute().mapPartitionsInternal { rowIterator =>
+ val context = TaskContext.get()
+ val allocator = ArrowUtils.getDefaultAllocator
+ val bytesIterator = ArrowConverters
+ .toBatchIterator(rowIterator, localSchema, maxRecordsPerBatch,
timeZoneId, context)
Review comment:
Okay, so there are:
1. `ColumnarBatch`
2. `ColumnarBatch` backed by Arrow
3. `ArrowRecordBatch`
and what this PR does creates 2. from rows (instead of 1.). More
specifically: `Rows -> 2` transition.
I was thinking something like:
`RowsToArrowExec` converts rows to 3. and `ArrowToColumnarExec` converts 2.
More specifically `Rows -> 3 -> 2` transition.
I haven't taken a deeper look but I was thinking about that we should better
have such physical plans and separations. I have been looking through related
Arrow feature requests .. and my guts say it would be very useful to have
these. At least I would like to reuse it.
Does it make sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
