nssalian commented on code in PR #1238:
URL: https://github.com/apache/iceberg-go/pull/1238#discussion_r3444742486
##########
table/arrow_utils.go:
##########
@@ -1190,8 +1178,33 @@ func (a *arrowProjectionVisitor) Primitive(_
iceberg.PrimitiveType, arr arrow.Ar
return arr
}
+// Variant reassembles a shredded variant back into the unshredded layout so
the
+// projected schema matches; without it the scanner fails to assemble the
table.
Review Comment:
I looked at it for a bit. It's not really a bug: the reader builds the
variant array's storage straight from the file's physical schema, so a shredded
file comes back as a shredded VariantArray (with Shredded()/UntypedValues() for
typed-column access), and Value(i) reassembles each row on demand. What's
missing is a way to collapse a whole shredded array into the plain layout,
which is what the scan needs since it can't mix shredded and non-shredded
arrays in one column.
I prototyped that as a small helper in arrow-go and used it here. It passes
the tests locally. I'll put it up as an arrow-go PR (will tag you for a review)
then update this PR to use it and drop the manual reassembly here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
