zhengruifeng commented on code in PR #38468:
URL: https://github.com/apache/spark/pull/38468#discussion_r1019118121
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowConverters.scala:
##########
@@ -128,6 +128,92 @@ private[sql] object ArrowConverters extends Logging {
}
}
+ private[sql] def toArrowBatchIterator(
Review Comment:
`toArrowBatchIterator` also write schema before each record batch, while
`toBatchIterator ` just output record batch.
and I am going to update `toArrowBatchIterator` in a follow-up to control
each batch size < 4MB as per the suggestions
https://github.com/apache/spark/pull/38468#discussion_r1018951362
https://github.com/apache/spark/pull/38468#discussion_r1013186031
I think we can deduplicate the codes then
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
