huaxingao commented on code in PR #18851:
URL: https://github.com/apache/iceberg/pull/18851#discussion_r4545101630


##########
arrow/src/test/java/org/apache/iceberg/arrow/vectorized/TestVectorizedReaderBuilder.java:
##########
@@ -171,14 +174,47 @@ void projectingShreddedVariantUnsupported() {
             .addField(shreddedVariant(Type.Repetition.OPTIONAL, 2, "data"))
             .named("table");
 
+    VectorizedReaderBuilder builder =
+        new VectorizedReaderBuilder(
+            icebergSchema, parquetSchema, false, ImmutableMap.of(), readers -> 
readers.get(1));
+
+    VectorizedReader<?> reader =
+        TypeWithSchemaVisitor.visit(icebergSchema.asStruct(), parquetSchema, 
builder);
+
+    
assertThat(reader).isInstanceOf(VectorizedArrowReader.RowBasedVariantReader.class);
+  }
+
+  @Test
+  void projectedShreddedListElementWithoutFieldIdSkipped() {
+    Schema icebergSchema =
+        new Schema(
+            NestedField.required(1, "id", IntegerType.get()),
+            NestedField.optional(2, "list", ListType.ofOptional(3, 
VariantType.get())));
+
+    MessageType parquetSchema =
+        Types.buildMessage()
+            .addField(
+                Types.primitive(PrimitiveTypeName.INT32, 
Type.Repetition.REQUIRED)
+                    .id(1)
+                    .named("id"))
+            .addField(
+                Types.buildGroup(Type.Repetition.OPTIONAL)
+                    .as(LogicalTypeAnnotation.listType())
+                    .addField(
+                        Types.buildGroup(Type.Repetition.REPEATED)
+                            
.addField(shreddedVariant(Type.Repetition.OPTIONAL, "element"))

Review Comment:
   This variant has no field id, so it was already skipped before this PR, and 
the test passes on main too. Could we add a field id here so the test covers 
the new code?



##########
arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedReaderBuilder.java:
##########
@@ -163,14 +170,38 @@ public ParquetVariantVisitor<VectorizedReader<?>> 
variantVisitor() {
       return null;
     }
 
+    if (isShredded(variant.asGroupType())) {
+      // shredded variants are read by the row-based reader built in variant()
+      return null;
+    }
+
     return new VectorizedVariantVisitor(
         currentPath(), parquetSchema, icebergSchema, rootAllocator, 
setArrowValidityVector);
   }
 
   @Override
+  @SuppressWarnings("unchecked")
   public VectorizedReader<?> variant(
       Types.VariantType iVariant, GroupType variant, VectorizedReader<?> 
result) {
-    return result;
+    if (iVariant == null || variant.getId() == null || !isShredded(variant)) {

Review Comment:
   Should we skip variants inside a list here? `primitive()` skips list columns 
with `getMaxRepetitionLevel() > 0`, but this method doesn't. So for a shredded 
variant in a list, we build a reader that is never used.



##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/sql/TestSparkVariantRead.java:
##########
@@ -557,6 +552,65 @@ public void testReadShreddedWithMetricsDisabled(String 
metricsMode)
     sql("DROP TABLE IF EXISTS %s", noStatsTable);
   }
 
+  @Test
+  void readShreddedArrayEnabledBySessionConf()
+      throws IOException, NoSuchTableException, ParseException {
+    String table = CATALOG + ".default.var_shredded_array";
+    sql("DROP TABLE IF EXISTS %s", table);
+    sql(
+        "CREATE TABLE %s (id BIGINT, v VARIANT) USING iceberg "

Review Comment:
   The new tests only use an optional variant. Could we also test a required 
one (`v VARIANT NOT NULL`)? For a required variant, `option()` in 
`VectorizedReaderBuilder.variant()` returns the reader without the null check, 
so that path is not tested yet.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to