Fokko commented on code in PR #3876:
URL: https://github.com/apache/parquet-java/pull/3876#discussion_r4580318998
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1158,6 +1165,57 @@ public PageReadStore readNextRowGroup() throws
IOException {
return currentRowGroup;
}
+ private record VectorColumn(ColumnDescriptor descriptor, int length) {}
+
+ private static Map<ColumnPath, VectorColumn> vectorColumns(MessageType
schema) {
+ if (!TypeUtil.hasFiniteVector(schema)) return Collections.emptyMap();
+ Map<ColumnPath, VectorColumn> columns = new HashMap<>();
+ for (ColumnDescriptor column : schema.getColumns()) {
+ FiniteVectorLogicalTypeAnnotation vector =
TypeUtil.getFiniteVectorType(schema, column.getPath());
+ if (vector != null) {
+ columns.put(ColumnPath.get(column.getPath()), new VectorColumn(column,
vector.getNumElements()));
+ }
+ }
+ return columns;
+ }
+
+ private static void checkVectorCounts(ColumnDescriptor path, int
numElements, long rows, long values, long nulls) {
+ long declaredNulls = nulls;
+ boolean invalid = rows < 0 || values < 0 || nulls < -1 || nulls > values;
+ if (!invalid && path.getMaxRepetitionLevel() == 1) {
+ if (path.getMaxDefinitionLevel() == 1) {
+ invalid = nulls > 0;
+ nulls = 0;
+ }
+ if (nulls >= 0) {
+ long present = values - nulls;
+ invalid |= nulls > rows || present % numElements != 0 || present /
numElements != rows - nulls;
+ } else {
+ long extra = values - rows;
+ invalid |= values < rows
+ || (numElements == 1
+ ? values != rows
+ : extra % (numElements - 1) != 0 || extra / (numElements - 1)
> rows);
+ }
+ } else if (!invalid && nulls >= 0) {
+ invalid = (values - nulls) % numElements != 0;
+ }
+ if (invalid) {
+ throw new ParquetDecodingException("Invalid FINITE_VECTOR counts for " +
Arrays.toString(path.getPath())
+ + ": rows=" + rows + ", values=" + values + ", nulls=" +
declaredNulls
+ + ", length=" + numElements);
+ }
+ }
+
+ private void checkVectorColumnCounts(ColumnChunkMetaData column, long rows) {
+ if (vectorColumns.isEmpty()) return;
+ VectorColumn vector = vectorColumns.get(column.getPath());
+ if (vector == null) return;
+ org.apache.parquet.column.statistics.Statistics<?> stats =
column.getStatistics();
+ long nulls = stats != null && stats.isNumNullsSet() ? stats.getNumNulls()
: -1;
Review Comment:
This turns `null_count` from the statistics into a hard read failure.
Statistics are optional, and we've had writers producing incorrect ones before,
so a file with valid data but a bad `null_count` becomes unreadable. One more
reason to drop the count checks (see above).
##########
parquet-column/src/main/java/org/apache/parquet/schema/LogicalTypeAnnotation.java:
##########
@@ -513,6 +524,45 @@ public int hashCode() {
}
}
+ public static class FiniteVectorLogicalTypeAnnotation extends
ListLogicalTypeAnnotation {
Review Comment:
The spec says every non-null vector MUST have exactly `num_elements`
elements and every element MUST be finite. Since parquet-java doesn't enforce
either (`testVectorValuesAreNotValidated` writes NaN/Inf), could we add a
Javadoc here saying so? Then users know it's up to them to write valid vectors.
##########
parquet-protobuf/src/main/java/org/apache/parquet/proto/ProtoMessageConverter.java:
##########
@@ -268,6 +268,12 @@ public Optional<Converter>
visit(LogicalTypeAnnotation.ListLogicalTypeAnnotation
return of(new ListConverter(parentBuilder, fieldDescriptor,
parquetType));
}
+ @Override
+ public Optional<Converter> visit(
+ LogicalTypeAnnotation.FiniteVectorLogicalTypeAnnotation
vectorLogicalType) {
+ return visit(LogicalTypeAnnotation.listType());
+ }
Review Comment:
The default `visit(FiniteVectorLogicalTypeAnnotation)` already falls back to
`visit(ListLogicalTypeAnnotation)`, which dispatches to the override above, so
I think we can drop this:
```suggestion
```
##########
parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/SchemaConverter.java:
##########
@@ -486,6 +486,21 @@ public Optional<TypeMapping> visit(
return of(new ListTypeMapping(arrowField, list3Levels, child));
}
+ @Override
+ public Optional<TypeMapping> visit(
+ LogicalTypeAnnotation.FiniteVectorLogicalTypeAnnotation
vectorLogicalType) {
+ List3Levels list3Levels = new List3Levels(type);
+ Type element = list3Levels.getElement();
+ TypeMapping child = fromParquet(element, null,
element.getRepetition());
+ ArrowType.FixedSizeList fixedSizeList =
+ new
ArrowType.FixedSizeList(vectorLogicalType.getNumElements());
Review Comment:
Nice, this maps to `FixedSizeList` now 👍 The other direction still loses the
length: `fromArrow` maps `ArrowType.FixedSizeList` to a plain `LIST`
(`createListTypeMapping()`), so `FINITE_VECTOR` → Arrow → Parquet gives a
`LIST`. Should we map it back to `FINITE_VECTOR(listSize)` when the child is a
non-nullable supported primitive, or do we keep that for a follow-up?
##########
parquet-column/src/main/java/org/apache/parquet/schema/LogicalTypeAnnotation.java:
##########
@@ -503,7 +514,7 @@ LogicalTypeToken getType() {
@Override
public boolean equals(Object obj) {
- return obj instanceof ListLogicalTypeAnnotation;
+ return obj instanceof ListLogicalTypeAnnotation && !(obj instanceof
FiniteVectorLogicalTypeAnnotation);
Review Comment:
Nit: this way the parent needs to know about its subclass. How about:
```suggestion
return obj != null && obj.getClass() ==
ListLogicalTypeAnnotation.class;
```
##########
parquet-hadoop/pom.xml:
##########
@@ -58,6 +58,12 @@
<type>test-jar</type>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.parquet</groupId>
+ <artifactId>parquet-encoding</artifactId>
Review Comment:
This came in with the commit that removed the validator. Is it still needed?
`parquet-encoding` is already on the classpath through `parquet-column`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]