Fokko commented on code in PR #3876:
URL: https://github.com/apache/parquet-java/pull/3876#discussion_r4580318998


##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1158,6 +1165,57 @@ public PageReadStore readNextRowGroup() throws 
IOException {
     return currentRowGroup;
   }
 
+  private record VectorColumn(ColumnDescriptor descriptor, int length) {}
+
+  private static Map<ColumnPath, VectorColumn> vectorColumns(MessageType 
schema) {
+    if (!TypeUtil.hasFiniteVector(schema)) return Collections.emptyMap();
+    Map<ColumnPath, VectorColumn> columns = new HashMap<>();
+    for (ColumnDescriptor column : schema.getColumns()) {
+      FiniteVectorLogicalTypeAnnotation vector = 
TypeUtil.getFiniteVectorType(schema, column.getPath());
+      if (vector != null) {
+        columns.put(ColumnPath.get(column.getPath()), new VectorColumn(column, 
vector.getNumElements()));
+      }
+    }
+    return columns;
+  }
+
+  private static void checkVectorCounts(ColumnDescriptor path, int 
numElements, long rows, long values, long nulls) {
+    long declaredNulls = nulls;
+    boolean invalid = rows < 0 || values < 0 || nulls < -1 || nulls > values;
+    if (!invalid && path.getMaxRepetitionLevel() == 1) {
+      if (path.getMaxDefinitionLevel() == 1) {
+        invalid = nulls > 0;
+        nulls = 0;
+      }
+      if (nulls >= 0) {
+        long present = values - nulls;
+        invalid |= nulls > rows || present % numElements != 0 || present / 
numElements != rows - nulls;
+      } else {
+        long extra = values - rows;
+        invalid |= values < rows
+            || (numElements == 1
+                ? values != rows
+                : extra % (numElements - 1) != 0 || extra / (numElements - 1) 
> rows);
+      }
+    } else if (!invalid && nulls >= 0) {
+      invalid = (values - nulls) % numElements != 0;
+    }
+    if (invalid) {
+      throw new ParquetDecodingException("Invalid FINITE_VECTOR counts for " + 
Arrays.toString(path.getPath())
+          + ": rows=" + rows + ", values=" + values + ", nulls=" + 
declaredNulls
+          + ", length=" + numElements);
+    }
+  }
+
+  private void checkVectorColumnCounts(ColumnChunkMetaData column, long rows) {
+    if (vectorColumns.isEmpty()) return;
+    VectorColumn vector = vectorColumns.get(column.getPath());
+    if (vector == null) return;
+    org.apache.parquet.column.statistics.Statistics<?> stats = 
column.getStatistics();
+    long nulls = stats != null && stats.isNumNullsSet() ? stats.getNumNulls() 
: -1;

Review Comment:
   This turns `null_count` from the statistics into a hard read failure. 
Statistics are optional, and we've had writers producing incorrect ones before, 
so a file with valid data but a bad `null_count` becomes unreadable. One more 
reason to drop the count checks (see above).



##########
parquet-column/src/main/java/org/apache/parquet/schema/LogicalTypeAnnotation.java:
##########
@@ -513,6 +524,45 @@ public int hashCode() {
     }
   }
 
+  public static class FiniteVectorLogicalTypeAnnotation extends 
ListLogicalTypeAnnotation {

Review Comment:
   The spec says every non-null vector MUST have exactly `num_elements` 
elements and every element MUST be finite. Since parquet-java doesn't enforce 
either (`testVectorValuesAreNotValidated` writes NaN/Inf), could we add a 
Javadoc here saying so? Then users know it's up to them to write valid vectors.



##########
parquet-protobuf/src/main/java/org/apache/parquet/proto/ProtoMessageConverter.java:
##########
@@ -268,6 +268,12 @@ public Optional<Converter> 
visit(LogicalTypeAnnotation.ListLogicalTypeAnnotation
             return of(new ListConverter(parentBuilder, fieldDescriptor, 
parquetType));
           }
 
+          @Override
+          public Optional<Converter> visit(
+              LogicalTypeAnnotation.FiniteVectorLogicalTypeAnnotation 
vectorLogicalType) {
+            return visit(LogicalTypeAnnotation.listType());
+          }

Review Comment:
   The default `visit(FiniteVectorLogicalTypeAnnotation)` already falls back to 
`visit(ListLogicalTypeAnnotation)`, which dispatches to the override above, so 
I think we can drop this:
   
   ```suggestion
   ```



##########
parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/SchemaConverter.java:
##########
@@ -486,6 +486,21 @@ public Optional<TypeMapping> visit(
               return of(new ListTypeMapping(arrowField, list3Levels, child));
             }
 
+            @Override
+            public Optional<TypeMapping> visit(
+                LogicalTypeAnnotation.FiniteVectorLogicalTypeAnnotation 
vectorLogicalType) {
+              List3Levels list3Levels = new List3Levels(type);
+              Type element = list3Levels.getElement();
+              TypeMapping child = fromParquet(element, null, 
element.getRepetition());
+              ArrowType.FixedSizeList fixedSizeList =
+                  new 
ArrowType.FixedSizeList(vectorLogicalType.getNumElements());

Review Comment:
   Nice, this maps to `FixedSizeList` now 👍 The other direction still loses the 
length: `fromArrow` maps `ArrowType.FixedSizeList` to a plain `LIST` 
(`createListTypeMapping()`), so `FINITE_VECTOR` → Arrow → Parquet gives a 
`LIST`. Should we map it back to `FINITE_VECTOR(listSize)` when the child is a 
non-nullable supported primitive, or do we keep that for a follow-up?



##########
parquet-column/src/main/java/org/apache/parquet/schema/LogicalTypeAnnotation.java:
##########
@@ -503,7 +514,7 @@ LogicalTypeToken getType() {
 
     @Override
     public boolean equals(Object obj) {
-      return obj instanceof ListLogicalTypeAnnotation;
+      return obj instanceof ListLogicalTypeAnnotation && !(obj instanceof 
FiniteVectorLogicalTypeAnnotation);

Review Comment:
   Nit: this way the parent needs to know about its subclass. How about:
   
   ```suggestion
         return obj != null && obj.getClass() == 
ListLogicalTypeAnnotation.class;
   ```



##########
parquet-hadoop/pom.xml:
##########
@@ -58,6 +58,12 @@
       <type>test-jar</type>
       <scope>test</scope>
     </dependency>
+    <dependency>
+      <groupId>org.apache.parquet</groupId>
+      <artifactId>parquet-encoding</artifactId>

Review Comment:
   This came in with the commit that removed the validator. Is it still needed? 
`parquet-encoding` is already on the classpath through `parquet-column`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to