Yicong-Huang commented on code in PR #53822:
URL: https://github.com/apache/spark/pull/53822#discussion_r2696994087
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowWriter.scala:
##########
@@ -387,6 +387,11 @@ private[arrow] class ArrayWriter(
val valueVector: ListVector,
val elementWriter: ArrowFieldWriter) extends ArrowFieldWriter {
+ // SPARK-55056: Arrow format requires ListArray offset buffer to have N+1
entries.
+ // Even when N=0, the buffer must contain [0]. Initialize offset buffer at
construction
+ // to ensure it exists even if no elements are written.
+ valueVector.getOffsetBuffer.setInt(0, 0)
Review Comment:
It should never be empty. according to [arrow's columnar
documentation](https://arrow.apache.org/docs/format/Columnar.html#list-layout)
and its example, the List offset is always N+1. As a list contains at least 0
elements, its offset is at least 1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
