xuechendi commented on a change in pull request #34396:
URL: https://github.com/apache/spark/pull/34396#discussion_r745365973
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala
##########
@@ -458,6 +462,34 @@ case class RowToColumnarExec(child: SparkPlan) extends
RowToColumnarTransition {
// This avoids calling `schema` in the RDD closure, so that we don't need
to include the entire
// plan (this) in the closure.
val localSchema = this.schema
+ if (enableArrowColumnVector) {
+ val maxRecordsPerBatch = SQLConf.get.arrowMaxRecordsPerBatch
+ val timeZoneId = SQLConf.get.sessionLocalTimeZone
+ return child.execute().mapPartitionsInternal { rowIterator =>
+ val context = TaskContext.get()
+ val allocator = ArrowUtils.getDefaultAllocator
+ val bytesIterator = ArrowConverters
+ .toBatchIterator(rowIterator, localSchema, maxRecordsPerBatch,
timeZoneId, context)
Review comment:
@BryanCutler , my goal here is to convert Iterator[InternalRow] to
Iterator[ColumnarBatch] with ArrowColumnVectors as internal ColumnVector. The
reason that I converted it to byte arrays first is inspired by @sunchao in
previous comment, Quoted:
> instead of having a writable Arrow vector, currently other projects such
as Iceberg and Spark RAPIDS choose to first construct the Arrow ValueVectors
and then wrap them to Spark with ArrowColumnVector."
So I figured when I need to read from internalRow and write to arrow
columnar format, there will be two ways to make that happen, 1) one is to read
and write to ArrowRecordBatch and load as ColumnarBatch; 2) the other option is
to use Arrow JAVA put APIs to write.
And from @sunchao , I think option 1 would make more sense since Spark
already has `ArrowConverters.toBatchIterator` function
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
