xuechendi commented on a change in pull request #34396:
URL: https://github.com/apache/spark/pull/34396#discussion_r748941892
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala
##########
@@ -458,6 +462,34 @@ case class RowToColumnarExec(child: SparkPlan) extends
RowToColumnarTransition {
// This avoids calling `schema` in the RDD closure, so that we don't need
to include the entire
// plan (this) in the closure.
val localSchema = this.schema
+ if (enableArrowColumnVector) {
+ val maxRecordsPerBatch = SQLConf.get.arrowMaxRecordsPerBatch
+ val timeZoneId = SQLConf.get.sessionLocalTimeZone
+ return child.execute().mapPartitionsInternal { rowIterator =>
+ val context = TaskContext.get()
+ val allocator = ArrowUtils.getDefaultAllocator
+ val bytesIterator = ArrowConverters
+ .toBatchIterator(rowIterator, localSchema, maxRecordsPerBatch,
timeZoneId, context)
Review comment:
@BryanCutler, are you saying converting between `ArrowRecordBatch` <->
`ColumnarBatch` is expensive? As far as I understand the codes, using Arrow
`VectorSchemaRoot` to load `ArrowRecordBatch` just doing `ArrowBuf`
reference_count increment then construct as a Arrow List[ValueVector], and this
is the same method `ArrowConverters.fromBatchIterator` uses.
It will be great if you can help me to understand your expected way, thanks!
I would like to use any existing method in ArrowConverters or add some to
it, the thing is the method you previously mentioned I replied before
[`fromBatchIterator`](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowConverters.scala#L132-L136)
is converting to RDD[InternalRow] not RDD[ColumnarBatch](This is why I didn't
use it), but I can extract the shared part of the codes as a new function, and
called by both `fromBatchIterator`(RDD[InternalRow]) and a new function, let's
name it as `toColumnarBatchIterator`(RDD[ColumnarBatch])
Do you think it makes sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
