BryanCutler commented on a change in pull request #34396:
URL: https://github.com/apache/spark/pull/34396#discussion_r745202340
##########
File path:
sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ArrowColumnVector.java
##########
@@ -38,6 +43,29 @@
private final ArrowVectorAccessor accessor;
private ArrowColumnVector[] childColumns;
+ public static ArrowColumnVector[] loadColumns(StructType schema,
+ ArrowRecordBatch recordBatch) {
Review comment:
I'm not really sure this makes sense here, it seems like it's mainly to
convert an `ArrowRecordBatch` to Spark `ColumnarBatch`. That is basically
what's done in `ArrowConverters.fromBatchIterator`, maybe you could move that
to it's own function there?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala
##########
@@ -458,6 +462,34 @@ case class RowToColumnarExec(child: SparkPlan) extends
RowToColumnarTransition {
// This avoids calling `schema` in the RDD closure, so that we don't need
to include the entire
// plan (this) in the closure.
val localSchema = this.schema
+ if (enableArrowColumnVector) {
+ val maxRecordsPerBatch = SQLConf.get.arrowMaxRecordsPerBatch
+ val timeZoneId = SQLConf.get.sessionLocalTimeZone
+ return child.execute().mapPartitionsInternal { rowIterator =>
+ val context = TaskContext.get()
+ val allocator = ArrowUtils.getDefaultAllocator
+ val bytesIterator = ArrowConverters
+ .toBatchIterator(rowIterator, localSchema, maxRecordsPerBatch,
timeZoneId, context)
Review comment:
I'm not quite sure of your use case for this code, could you explain a
bit more what this does for you, especially why serialize the record batch to a
byte array?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
