Fokko commented on code in PR #3876:
URL: https://github.com/apache/parquet-java/pull/3876#discussion_r4485350719
##########
parquet-column/src/main/java/org/apache/parquet/schema/LogicalTypeAnnotation.java:
##########
@@ -1536,6 +1591,10 @@ default Optional<T> visit(ListLogicalTypeAnnotation
listLogicalType) {
return empty();
}
+ default Optional<T> visit(FiniteVectorLogicalTypeAnnotation
vectorLogicalType) {
+ return empty();
+ }
Review Comment:
I think we can do the fallback here:
```suggestion
default Optional<T> visit(FiniteVectorLogicalTypeAnnotation
vectorLogicalType) {
return visit((ListLogicalTypeAnnotation) vectorLogicalType);
}
```
##########
parquet-column/src/main/java/org/apache/parquet/io/MessageColumnIO.java:
##########
@@ -508,7 +509,9 @@ public void flush() {
public RecordConsumer getRecordWriter(ColumnWriteStore columns) {
RecordConsumer recordWriter = new MessageColumnIORecordConsumer(columns);
if (DEBUG) recordWriter = new RecordConsumerLoggingWrapper(recordWriter);
- return validating ? new ValidatingRecordConsumer(recordWriter, getType())
: recordWriter;
+ return validating || TypeUtil.hasFiniteVector(getType())
Review Comment:
Based on the discussion on the dev-list, we could also drop the validation.
One of the goals of the VECTOR is performance, so I think we can also delegate
this to the engine.
##########
parquet-column/src/main/java/org/apache/parquet/schema/Types.java:
##########
@@ -846,10 +850,46 @@ public THIS addFields(Type... types) {
@Override
protected GroupType build(String name) {
- if (newLogicalTypeSet) {
- return new GroupType(repetition, name, logicalTypeAnnotation, fields,
id);
- } else {
- return new GroupType(repetition, name, getOriginalType(), fields, id);
+ GroupType group = newLogicalTypeSet
+ ? new GroupType(repetition, name, logicalTypeAnnotation, fields, id)
+ : new GroupType(repetition, name, getOriginalType(), fields, id);
+ if (logicalTypeAnnotation instanceof FiniteVectorLogicalTypeAnnotation) {
+ checkFiniteVector(group);
Review Comment:
Thanks for moving this into the builder! There's still a public way around
it: `GroupType.withNewFields(...)` keeps the annotation and calls the
constructor directly. I tried it: a `FINITE_VECTOR` group with an `optional
float element` (via `withNewFields`) writes fine, and then
`ParquetFileReader.open` on that same file fails with `Invalid FINITE_VECTOR
schema in file metadata`. So we can produce files we can't read ourselves.
Can we also call `checkFiniteVector` from `TypeUtil.checkValidWriteSchema`?
`ParquetFileWriter` already calls that for every write, so it closes the gap
without touching the `GroupType` constructors.
##########
parquet-column/src/main/java/org/apache/parquet/schema/LogicalTypeAnnotation.java:
##########
@@ -513,6 +524,50 @@ public int hashCode() {
}
}
+ public static class FiniteVectorLogicalTypeAnnotation extends
ListLogicalTypeAnnotation {
+ private final int numElements;
+
+ private FiniteVectorLogicalTypeAnnotation(int numElements) {
+ Preconditions.checkArgument(numElements > 0, "Invalid FINITE_VECTOR
length: %s", numElements);
+ this.numElements = numElements;
+ }
+
+ public int getNumElements() {
+ return numElements;
+ }
+
+ @Override
+ public OriginalType toOriginalType() {
+ return OriginalType.LIST;
+ }
+
Review Comment:
Nit: this is inherited from `ListLogicalTypeAnnotation`, so we can drop it:
```suggestion
```
Less is more 👍
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java:
##########
@@ -1158,6 +1160,45 @@ public PageReadStore readNextRowGroup() throws
IOException {
return currentRowGroup;
}
+ private static void checkVectorCounts(ColumnDescriptor path, int
numElements, long rows, long values, long nulls) {
+ long declaredNulls = nulls;
+ boolean invalid = rows < 0 || values < 0 || nulls < -1 || nulls > values;
+ if (!invalid && path.getMaxRepetitionLevel() == 1) {
+ if (path.getMaxDefinitionLevel() == 1) {
+ invalid = nulls > 0;
+ nulls = 0;
+ }
+ if (nulls >= 0) {
+ long present = values - nulls;
+ invalid |= nulls > rows || present % numElements != 0 || present /
numElements != rows - nulls;
+ } else {
+ long extra = values - rows;
+ invalid |= values < rows
+ || (numElements == 1
+ ? values != rows
+ : extra % (numElements - 1) != 0 || extra / (numElements - 1)
> rows);
+ }
+ } else if (!invalid && nulls >= 0) {
+ invalid = (values - nulls) % numElements != 0;
+ }
+ if (invalid) {
+ throw new ParquetDecodingException("Invalid FINITE_VECTOR counts for " +
Arrays.toString(path.getPath())
+ + ": rows=" + rows + ", values=" + values + ", nulls=" +
declaredNulls
+ + ", length=" + numElements);
+ }
+ }
+
+ private void checkVectorColumnCounts(ColumnChunkMetaData column, long rows) {
+ MessageType fileSchema = getFileMetaData().getSchema();
+ String[] path = column.getPath().toArray();
+ FiniteVectorLogicalTypeAnnotation vector =
TypeUtil.getFiniteVectorType(fileSchema, path);
Review Comment:
Small perf thing: this walks the schema (`getFiniteVectorType` +
`getColumnDescription`) for every column chunk of every row group, and again in
`readAllPages`. We could compute a `Map<ColumnPath, Integer>` of vector lengths
once when the reader is constructed; for files without vectors that's a single
`isEmpty()` check.
##########
parquet-column/src/main/java/org/apache/parquet/io/RecordReaderImplementation.java:
##########
@@ -413,6 +433,7 @@ public T read() {
do {
ColumnReader columnReader = currentState.column;
int d = columnReader.getCurrentDefinitionLevel();
+ currentState.validateVector(d);
Review Comment:
This validates every vector value on the record read path, which costs
~15–25% on vector-heavy reads in my benchmark. Based on the point raised on the
dev-list, I'd remove this check for now. C++ does not validate this either.
##########
parquet-hadoop/src/test/resources/vector/README.md:
##########
@@ -0,0 +1,92 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# FINITE_VECTOR reference files
+
+These files use the draft [FINITE_VECTOR
proposal](https://github.com/apache/parquet-format/pull/624)
+at revision `8ad956d543e3074e1d9c8d4faa194199e06a8be4`. They are
+parquet-java-generated examples, not independent interoperability evidence.
Runtime
+mixed-file tests exercise the writer; these fixed bytes exercise reader
compatibility.
+The valid files are intended for a later parquet-testing/data export.
Malformed 4/6,
+content, and count cases and Java exception policy are excluded from portable
conformance.
+
+These bytes were generated before VECTOR was renamed to FINITE_VECTOR. The
rename
+preserves Thrift field 20 and its parameter, so the files and hashes are
unchanged.
+The malformed file's `VectorInteropFileGenerator` source blob was
+`f5217bb32e86abfac965a3eebab9709adbaa549d`. The three valid files came from
+the Round 2 generator source `2d496dad2213216ca6e75f1ab0b4249822829421`.
+Both used Java 17, Maven 3.9.16, and Thrift 0.25.0.
+
+## Mixed files
+
+For each row `i=0..5`, `id=i`,
`f=[i+0.5f,-0.0f,Float.MIN_VALUE,-0.25f,100.0f]`,
+`o` is absent for even `i` and `[7,8]` for odd `i`, and `a` is absent for
`i=1,4`.
+When present, `a.h=[0080,0100]` in hex. There are `i%3` `outer` instances,
+each with `d=[-0.0d,Double.MIN_VALUE]`. Every row has `b=[true,false]`,
+`l=[9L]`, `s=[-7]`, `u=[9L]`, `dec=[125]` (logical 1.25), and
+`bytes=[00007c00]` in hex (logical 317.44).
+
+## Split file
+
+The two rows are `[0f,1f,2f,3f,4f]` and `[5f,6f,7f,8f,9f]`. Their stored RLs
+are `[0,1,1,1,1,0,1,1,1,1]`, with DL=1 throughout. Three unindexed V1 pages
+contain 3/4/3 values. The first and last pages use RLE/RLE levels; the middle
+uses BIT_PACKED/BIT_PACKED levels. Values are PLAIN, compression is
UNCOMPRESSED,
+and column and offset indexes are absent.
+
+## Malformed file
+
+`vector_invalid_v1_dictionary.parquet` is deliberately invalid FINITE_VECTOR
data. Its
+ordinary V1 writer used a canonical LIST schema with `id=0..3`, then changed
only
+the footer to annotate eight LIST groups as FINITE_VECTOR. `short_v` has
lengths
+`[4,6,5,5]`, `long_v` has `[6,4,5,5]`, `empty_v` has `[0,5,5,5]`, nested `v`
+has per-row instance lengths `[[4,6],[],[5],[5]]`, and partial `v` has
+`[[5],[5],[5],[3]]`. These groups declare FINITE_VECTOR(5). Their finite
element `k`
+in instance `j` of row `i` is `100*i+10*j+k` as FLOAT.
+
+`nf_f`, `nf_d`, and `nf_h` declare FINITE_VECTOR(8). Each row contains seven
ones;
+element zero is zero except in row zero, where the fields contain FLOAT NaN,
+DOUBLE positive infinity, and FLOAT16 negative infinity respectively. The
FLOAT16
+values use little-endian bytes `00fc`, `0000`, and `003c` for negative
infinity,
+zero, and one. The file is UNCOMPRESSED with a page row limit of two. `nf_f`
and
+`nf_d` each have a dictionary and two V1 PLAIN_DICTIONARY pages of 16 values;
+`nf_h` has no dictionary and two V1 PLAIN pages of 16 values. All column-chunk
+null counts are unset, and column and offset index references are absent.
+Each projected malformed FINITE_VECTOR field must be rejected.
+
+## SHA-256
+
+```text
+c0e182b7f019d394a26bf863626cd58fed3916d930195ab0741fe15b41afb6e8
vector_mixed_v1_plain.parquet
+939ce11a1cbe84c68d867deeab85546162d04cab8fd191d5a8efe203bfbb0765
vector_mixed_v2_dictionary.parquet
+a598252a6f5715742bdb0e52c86be992f609189096e1cdc518701f4520f4c657
vector_split_v1.parquet
+a87042f46c3d56bb9b236795c9c44f13bd02eecb4107040d6bb4b245a73dd7fd
vector_invalid_v1_dictionary.parquet
+```
+
+## Reproduce
+
+From the repository root, use a new empty `target/vector-reference` directory.
+An occupied directory fails by design; the generator does not overwrite files.
+The pinned source and toolchain reproduce the documented rows and layout;
future
+writer changes need not produce byte-identical files.
+
+```bash
+JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
/home/div.arora/.m2/wrapper/dists/apache-maven-3.9.16/c51baac8/bin/mvn
-Dthrift.executable=/home/div.arora/parquet-java-worktrees/vector/target/vector-tools/bin/thrift
-pl parquet-hadoop -am -DskipTests package dependency:build-classpath
-DincludeScope=test -Dmdep.outputFile=target/vector-test-classpath.txt
-Dmdep.regenerateFile=true
Review Comment:
This contains local paths (`/home/...`, a local Thrift binary) and some
review history (pre-rename notes, "Round 2" generator). Could we trim it down
to what the files contain and how to regenerate them with the repo's standard
build? Once the files move to parquet-testing, this can go there too.
##########
parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java:
##########
@@ -2017,13 +2031,20 @@ private static ColumnPath getPath(ColumnMetaData
metaData) {
MessageType fromParquetSchema(List<SchemaElement> schema, List<ColumnOrder>
columnOrders) {
Iterator<SchemaElement> iterator = schema.iterator();
SchemaElement root = iterator.next();
+ if (root.isSetLogicalType() && root.logicalType.isSetFINITE_VECTOR()) {
+ throw new ParquetDecodingException("Invalid FINITE_VECTOR metadata for "
+ root.name);
+ }
Types.MessageTypeBuilder builder = Types.buildMessage();
if (root.isSetField_id()) {
builder.id(root.field_id);
}
Iterator<ColumnOrder> columnOrderIterator = columnOrders == null ? null :
columnOrders.iterator();
- buildChildren(builder, iterator, root.getNum_children(),
columnOrderIterator);
- return builder.named(root.name);
+ try {
+ buildChildren(builder, iterator, root.getNum_children(),
columnOrderIterator);
+ return builder.named(root.name);
+ } catch (InvalidSchemaException e) {
+ throw new ParquetDecodingException("Invalid FINITE_VECTOR schema in file
metadata", e);
Review Comment:
This wraps *every* `InvalidSchemaException` from the builders with a
FINITE_VECTOR-specific message. That's true today, but as soon as another type
adds builder validation the message is misleading. Could we keep the original
message, or only rethrow with this message when the cause came from the vector
check?
##########
parquet-column/src/main/java/org/apache/parquet/io/FiniteVectorValidator.java:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.io;
+
+import java.util.Arrays;
+import org.apache.parquet.column.ColumnReader;
+import org.apache.parquet.io.api.Binary;
+import
org.apache.parquet.schema.LogicalTypeAnnotation.FiniteVectorLogicalTypeAnnotation;
+import
org.apache.parquet.schema.LogicalTypeAnnotation.Float16LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName;
+
+final class FiniteVectorValidator {
+ private final ColumnReader column;
+ private final String[] fieldPath;
+ private final PrimitiveTypeName primitive;
+ private final int maxDefinitionLevel;
+ private final int maxRepetitionLevel;
+ private final int length;
+ private final boolean float16;
+ private long count;
+
+ FiniteVectorValidator(PrimitiveColumnIO columnIO, ColumnReader column,
FiniteVectorLogicalTypeAnnotation vector) {
+ this.column = column;
+ this.fieldPath = columnIO.getFieldPath();
+ this.primitive =
columnIO.getType().asPrimitiveType().getPrimitiveTypeName();
+ this.maxDefinitionLevel = columnIO.getDefinitionLevel();
+ this.maxRepetitionLevel = columnIO.getRepetitionLevel();
+ this.length = vector.getNumElements();
+ this.float16 = columnIO.getType().getLogicalTypeAnnotation() instanceof
Float16LogicalTypeAnnotation;
+ }
+
+ void validate(int definitionLevel) {
+ int repetitionLevel = column.getCurrentRepetitionLevel();
+ if (definitionLevel < 0
+ || definitionLevel > maxDefinitionLevel
+ || repetitionLevel < 0
+ || repetitionLevel > maxRepetitionLevel) {
+ throw invalidVector("invalid levels");
+ }
+ if (count == 0 && repetitionLevel == maxRepetitionLevel) {
+ throw invalidVector("invalid continuation");
+ }
+ if (definitionLevel < maxDefinitionLevel) {
+ if (count > 0) throw invalidVector("null within vector");
+ if (definitionLevel == maxDefinitionLevel - 1) throw
invalidVector("empty vector");
+ return;
+ }
+ count++;
+ if (count > length) {
+ throw invalidVector("expected " + length + " elements, got " + count);
+ }
+ if (primitive == PrimitiveTypeName.FLOAT &&
!Float.isFinite(column.getFloat())
+ || primitive == PrimitiveTypeName.DOUBLE &&
!Double.isFinite(column.getDouble())) {
+ throw invalidVector("non-finite element");
+ }
+ if (float16) {
+ Binary value = column.getBinary();
+ if (value.length() != 2) {
+ throw invalidVector("invalid FLOAT16 element length: " +
value.length());
+ }
+ if ((value.get2BytesLittleEndian() & 0x7c00) == 0x7c00) {
Review Comment:
The same FLOAT16 bit check is also in `ValidatingRecordConsumer#addBinary`.
How about adding `Float16.isFinite(short)` next to the existing
`Float16.isNaN(short)` and using that in both places?
More broadly, this class is already the state machine for "a valid vector
level stream". If the writer also checks at the column level (see my comment on
`MessageColumnIO`), both sides can share it, and the vector logic in
`ValidatingRecordConsumer` can go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]