Fokko commented on code in PR #3876:
URL: https://github.com/apache/parquet-java/pull/3876#discussion_r4444434085


##########
parquet-avro/src/main/java/org/apache/parquet/avro/AvroReadSupport.java:
##########
@@ -281,7 +281,9 @@ private static Boolean allListStructuresAreThreeLevel(Type 
type) {
       return null;
     }
     GroupType group = type.asGroupType();
-    if (group.getLogicalTypeAnnotation() instanceof 
LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+    if (group.getLogicalTypeAnnotation() instanceof 
LogicalTypeAnnotation.ListLogicalTypeAnnotation
+        || group.getLogicalTypeAnnotation()
+            instanceof 
LogicalTypeAnnotation.FiniteVectorLogicalTypeAnnotation) {

Review Comment:
   Nit, I think it makes sense to rip out the `getLogicalTypeAnnotation` in a 
variable, to make it easier to read:
   ```suggestion
       LogicalTypeAnnotation annotation = group.getLogicalTypeAnnotation();
       if (annotation instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation
           || annotation instanceof 
LogicalTypeAnnotation.FiniteVectorLogicalTypeAnnotation) {
   ```



##########
parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/List3Levels.java:
##########
@@ -43,6 +43,8 @@ class List3Levels {
    */
   public List3Levels(GroupType list) {
     if (list.getLogicalTypeAnnotation() != LogicalTypeAnnotation.listType()
+            && !(list.getLogicalTypeAnnotation()
+                instanceof 
LogicalTypeAnnotation.FiniteVectorLogicalTypeAnnotation)

Review Comment:
   One of the rare occasions to use Morgan's law to simplify:
   ```suggestion
   LogicalTypeAnnotation annotation = list.getLogicalTypeAnnotation();
   if (!(annotation == LogicalTypeAnnotation.listType()
         || annotation instanceof 
LogicalTypeAnnotation.FiniteVectorLogicalTypeAnnotation)
   ```



##########
parquet-column/src/main/java/org/apache/parquet/schema/TypeUtil.java:
##########
@@ -18,8 +18,106 @@
  */
 package org.apache.parquet.schema;
 
+import 
org.apache.parquet.schema.LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+import 
org.apache.parquet.schema.LogicalTypeAnnotation.FiniteVectorLogicalTypeAnnotation;
+import 
org.apache.parquet.schema.LogicalTypeAnnotation.Float16LogicalTypeAnnotation;
+import 
org.apache.parquet.schema.LogicalTypeAnnotation.IntLogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName;
+import org.apache.parquet.schema.Type.Repetition;
+
 public class TypeUtil {
+  public static boolean checkValidFiniteVectorSchema(Type type) {
+    boolean found = type.getLogicalTypeAnnotation() instanceof 
FiniteVectorLogicalTypeAnnotation;
+    if (found) {
+      if (type.isPrimitive()) {
+        throw new InvalidSchemaException("Invalid FINITE_VECTOR schema: " + 
type);
+      }
+      checkVector(type.asGroupType());
+    } else if (!type.isPrimitive()) {
+      for (Type child : type.asGroupType().getFields()) {
+        found |= checkValidFiniteVectorSchema(child);
+      }
+    }
+    return found;
+  }
+
+  private static void checkVector(GroupType vector) {
+    if (vector.getRepetition() == Repetition.REPEATED || 
vector.getFieldCount() != 1) {
+      throw new InvalidSchemaException("Invalid FINITE_VECTOR schema: " + 
vector);
+    }
+    Type listType = vector.getType(0);
+    if (listType.isPrimitive()
+        || !listType.getName().equals("list")
+        || listType.getRepetition() != Repetition.REPEATED
+        || listType.getLogicalTypeAnnotation() != null) {
+      throw new InvalidSchemaException("Invalid FINITE_VECTOR schema: " + 
vector);
+    }
+    GroupType list = listType.asGroupType();
+    if (list.getFieldCount() != 1) {
+      throw new InvalidSchemaException("Invalid FINITE_VECTOR schema: " + 
vector);
+    }
+    Type elementType = list.getType(0);
+    if (!elementType.isPrimitive()
+        || !elementType.getName().equals("element")
+        || elementType.getRepetition() != Repetition.REQUIRED) {
+      throw new InvalidSchemaException("Invalid FINITE_VECTOR schema: " + 
vector);
+    }
+    PrimitiveType element = elementType.asPrimitiveType();
+    PrimitiveTypeName physical = element.getPrimitiveTypeName();
+    LogicalTypeAnnotation logical = element.getLogicalTypeAnnotation();
+    boolean allowed;
+    switch (physical) {
+      case BOOLEAN:
+      case FLOAT:
+      case DOUBLE:
+        allowed = logical == null;
+        break;
+      case INT32:
+      case INT64:
+        allowed = logical == null
+            || logical instanceof IntLogicalTypeAnnotation
+            || logical instanceof DecimalLogicalTypeAnnotation;
+        break;
+      case FIXED_LEN_BYTE_ARRAY:
+        allowed = logical instanceof Float16LogicalTypeAnnotation
+            || logical instanceof DecimalLogicalTypeAnnotation;
+        break;
+      default:
+        allowed = false;
+    }

Review Comment:
   We're on JDK17, so we can use the sweet enhanced switch:
   ```suggestion
       boolean allowed = switch (physical) {
         case BOOLEAN, FLOAT, DOUBLE -> logical == null;
         case INT32, INT64 -> logical == null
             || logical instanceof IntLogicalTypeAnnotation
             || logical instanceof DecimalLogicalTypeAnnotation;
         case FIXED_LEN_BYTE_ARRAY -> logical instanceof 
Float16LogicalTypeAnnotation
             || logical instanceof DecimalLogicalTypeAnnotation;
         default -> false;
       };
   ```
   So much cleaner 🤤 



##########
parquet-column/src/main/java/org/apache/parquet/io/FilteredRecordReader.java:
##########
@@ -84,16 +84,19 @@ private void skipToMatch() {
       State currentState = getState(0);
       do {
         ColumnReader columnReader = currentState.column;
+        int d = columnReader.getCurrentDefinitionLevel();

Review Comment:
   ```suggestion
           int depth = columnReader.getCurrentDefinitionLevel();
   ```



##########
parquet-column/src/main/java/org/apache/parquet/schema/LogicalTypeAnnotation.java:
##########
@@ -500,6 +511,50 @@ public int hashCode() {
     }
   }
 
+  public static class FiniteVectorLogicalTypeAnnotation extends 
LogicalTypeAnnotation {

Review Comment:
   I think the key here is:
   
   
   ```suggestion
     public static class FiniteVectorLogicalTypeAnnotation extends 
ListLogicalTypeAnnotation {
   ```



##########
parquet-column/src/main/java/org/apache/parquet/io/ColumnIOFactory.java:
##########
@@ -53,6 +54,7 @@ private ColumnIOCreatorVisitor(
 
     @Override
     public void visit(MessageType messageType) {
+      TypeUtil.checkValidFiniteVectorSchema(messageType);

Review Comment:
   I see a lot of random validation, shouldn't we check this much earlier when 
the `MessageType` is being constructed?



##########
parquet-avro/src/main/java/org/apache/parquet/avro/AvroSchemaConverter.java:
##########
@@ -470,7 +471,9 @@ public Optional<Schema> visit(
                 if (!repeatedType.isRepetition(REPEATED)) {
                   throw new UnsupportedOperationException("Invalid list type " 
+ parquetGroupType);
                 }
-                if (isElementType(repeatedType, parquetGroupType.getName())) {
+                if (!(logicalTypeAnnotation
+                        instanceof 
LogicalTypeAnnotation.FiniteVectorLogicalTypeAnnotation)
+                    && isElementType(repeatedType, 
parquetGroupType.getName())) {

Review Comment:
   Do we really need this addition specifically for 
`FiniteVectorLogicalTypeAnnotation`? Feels awkward to me. Shouldn't 
`isElementType` handle this?



##########
parquet-hadoop/src/test/resources/vector/vector_invalid_v1_dictionary.parquet:
##########


Review Comment:
   Eventually, these should go in https://github.com/apache/parquet-testing/



##########
parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/SchemaConverter.java:
##########
@@ -486,6 +487,13 @@ public Optional<TypeMapping> visit(
               return of(new ListTypeMapping(arrowField, list3Levels, child));
             }
 
+            @Override
+            public Optional<TypeMapping> visit(
+                LogicalTypeAnnotation.FiniteVectorLogicalTypeAnnotation 
vectorLogicalType) {
+              TypeUtil.checkValidFiniteVectorSchema(type);
+              return visit(LogicalTypeAnnotation.listType());

Review Comment:
   For Arrow, we want o make sure that we map to the Fixed:
   
   
   ```suggestion
                 List3Levels list3Levels = new List3Levels(type);
                 TypeMapping child = fromParquet(
                     list3Levels.getElement(),
                     null,
                     list3Levels.getElement().getRepetition());
                 ArrowType arrowType = new 
ArrowType.FixedSizeList(vectorLogicalType.getNumElements());
                 Field arrowField = new Field(
                     name,
                     type.isRepetition(OPTIONAL)
                         ? FieldType.nullable(arrowType)
                         : FieldType.notNullable(arrowType),
                     Collections.singletonList(child.getArrowField()));
                 return of(new ListTypeMapping(arrowField, list3Levels, child));
   ```



##########
parquet-column/src/main/java/org/apache/parquet/schema/TypeUtil.java:
##########
@@ -18,8 +18,106 @@
  */
 package org.apache.parquet.schema;
 
+import 
org.apache.parquet.schema.LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+import 
org.apache.parquet.schema.LogicalTypeAnnotation.FiniteVectorLogicalTypeAnnotation;
+import 
org.apache.parquet.schema.LogicalTypeAnnotation.Float16LogicalTypeAnnotation;
+import 
org.apache.parquet.schema.LogicalTypeAnnotation.IntLogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName;
+import org.apache.parquet.schema.Type.Repetition;
+
 public class TypeUtil {
+  public static boolean checkValidFiniteVectorSchema(Type type) {
+    boolean found = type.getLogicalTypeAnnotation() instanceof 
FiniteVectorLogicalTypeAnnotation;
+    if (found) {
+      if (type.isPrimitive()) {
+        throw new InvalidSchemaException("Invalid FINITE_VECTOR schema: " + 
type);
+      }
+      checkVector(type.asGroupType());
+    } else if (!type.isPrimitive()) {
+      for (Type child : type.asGroupType().getFields()) {
+        found |= checkValidFiniteVectorSchema(child);
+      }
+    }
+    return found;
+  }
+
+  private static void checkVector(GroupType vector) {
+    if (vector.getRepetition() == Repetition.REPEATED || 
vector.getFieldCount() != 1) {
+      throw new InvalidSchemaException("Invalid FINITE_VECTOR schema: " + 
vector);
+    }
+    Type listType = vector.getType(0);
+    if (listType.isPrimitive()
+        || !listType.getName().equals("list")
+        || listType.getRepetition() != Repetition.REPEATED
+        || listType.getLogicalTypeAnnotation() != null) {
+      throw new InvalidSchemaException("Invalid FINITE_VECTOR schema: " + 
vector);
+    }
+    GroupType list = listType.asGroupType();
+    if (list.getFieldCount() != 1) {
+      throw new InvalidSchemaException("Invalid FINITE_VECTOR schema: " + 
vector);
+    }
+    Type elementType = list.getType(0);
+    if (!elementType.isPrimitive()
+        || !elementType.getName().equals("element")
+        || elementType.getRepetition() != Repetition.REQUIRED) {
+      throw new InvalidSchemaException("Invalid FINITE_VECTOR schema: " + 
vector);
+    }
+    PrimitiveType element = elementType.asPrimitiveType();
+    PrimitiveTypeName physical = element.getPrimitiveTypeName();
+    LogicalTypeAnnotation logical = element.getLogicalTypeAnnotation();
+    boolean allowed;
+    switch (physical) {
+      case BOOLEAN:
+      case FLOAT:
+      case DOUBLE:
+        allowed = logical == null;
+        break;
+      case INT32:
+      case INT64:
+        allowed = logical == null
+            || logical instanceof IntLogicalTypeAnnotation
+            || logical instanceof DecimalLogicalTypeAnnotation;
+        break;
+      case FIXED_LEN_BYTE_ARRAY:
+        allowed = logical instanceof Float16LogicalTypeAnnotation
+            || logical instanceof DecimalLogicalTypeAnnotation;
+        break;
+      default:
+        allowed = false;
+    }
+    if (!allowed) {
+      throw new InvalidSchemaException("Invalid FINITE_VECTOR schema: " + 
vector);
+    }
+    try {
+      Types.PrimitiveBuilder<PrimitiveType> builder = 
Types.primitive(physical, element.getRepetition());
+      if (physical == PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY) {
+        builder.length(element.getTypeLength());
+      }
+      if (logical != null) {
+        builder.as(logical);
+      }
+      builder.named(element.getName());

Review Comment:
   I think this shows the issue. For other types we validate it once we try to 
build it, we want something similar for the `FINITE_VECTOR`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to