nssalian commented on code in PR #18851:
URL: https://github.com/apache/iceberg/pull/18851#discussion_r4556352483
##########
arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedReaderBuilder.java:
##########
@@ -163,14 +170,38 @@ public ParquetVariantVisitor<VectorizedReader<?>>
variantVisitor() {
return null;
}
+ if (isShredded(variant.asGroupType())) {
+ // shredded variants are read by the row-based reader built in variant()
+ return null;
+ }
+
return new VectorizedVariantVisitor(
currentPath(), parquetSchema, icebergSchema, rootAllocator,
setArrowValidityVector);
}
@Override
+ @SuppressWarnings("unchecked")
public VectorizedReader<?> variant(
Types.VariantType iVariant, GroupType variant, VectorizedReader<?>
result) {
- return result;
+ if (iVariant == null || variant.getId() == null || !isShredded(variant)) {
Review Comment:
Makes sense. variant() now returns early when the variant is under a list or
map (max repetition level > 0), the same as primitive().
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]