xuechendi commented on a change in pull request #34396:
URL: https://github.com/apache/spark/pull/34396#discussion_r747988830
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala
##########
@@ -458,6 +462,34 @@ case class RowToColumnarExec(child: SparkPlan) extends
RowToColumnarTransition {
// This avoids calling `schema` in the RDD closure, so that we don't need
to include the entire
// plan (this) in the closure.
val localSchema = this.schema
+ if (enableArrowColumnVector) {
+ val maxRecordsPerBatch = SQLConf.get.arrowMaxRecordsPerBatch
+ val timeZoneId = SQLConf.get.sessionLocalTimeZone
+ return child.execute().mapPartitionsInternal { rowIterator =>
+ val context = TaskContext.get()
+ val allocator = ArrowUtils.getDefaultAllocator
+ val bytesIterator = ArrowConverters
+ .toBatchIterator(rowIterator, localSchema, maxRecordsPerBatch,
timeZoneId, context)
Review comment:
I see, I am not sure if you are familiar with
[toArrowBatchRdd](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L3857-L3872)
in Dataset, it really looks like what `RowsToArrowExec` is.
And if your are looking for
> converts 2. More specifically Rows -> 3 -> 2 transition.
I think that is what I am proposing in `RowToColumnarExec`, so if I think we
can add a parameter in `RowToColumnarExec` to specify Arrow as backend, it
already gets what we wanted.
After all, if you think adding `RowToArrowExec` won't be duplicated to
`Dataset.toArrowBatchRdd`, I am ok to add these two new physical plans, then do
a combination of physical plan of converting Row -> 3 -> 2. I can do it in a
seperate PR , but I still hope to make sure adding two new physical plans is a
desired way. @cloud-fan @sunchao @BryanCutler
So the proposal here will be
1. add `ArrowToColumnarExec` which converts `RDD[ArrowRecordBatch]` to
`RDD[ColumnarBatch backend by Arrow]`
2. add `ColumnarToArrowExec` which converts `RDD[ColumnarBatch backend by
Arrow]` to `RDD[ArrowRecordBatch]`
3. add `RowToArrowExec` doing what current
[toArrowBatchRdd](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L3857-L3872)
in Dataset does
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
