Fokko commented on code in PR #3876:
URL: https://github.com/apache/parquet-java/pull/3876#discussion_r4444434085
##########
parquet-avro/src/main/java/org/apache/parquet/avro/AvroReadSupport.java:
##########
@@ -281,7 +281,9 @@ private static Boolean allListStructuresAreThreeLevel(Type
type) {
return null;
}
GroupType group = type.asGroupType();
- if (group.getLogicalTypeAnnotation() instanceof
LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+ if (group.getLogicalTypeAnnotation() instanceof
LogicalTypeAnnotation.ListLogicalTypeAnnotation
+ || group.getLogicalTypeAnnotation()
+ instanceof
LogicalTypeAnnotation.FiniteVectorLogicalTypeAnnotation) {
Review Comment:
Nit, I think it makes sense to rip out the `getLogicalTypeAnnotation` in a
variable, to make it easier to read:
```suggestion
LogicalTypeAnnotation annotation = group.getLogicalTypeAnnotation();
if (annotation instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation
|| annotation instanceof
LogicalTypeAnnotation.FiniteVectorLogicalTypeAnnotation) {
```
##########
parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/List3Levels.java:
##########
@@ -43,6 +43,8 @@ class List3Levels {
*/
public List3Levels(GroupType list) {
if (list.getLogicalTypeAnnotation() != LogicalTypeAnnotation.listType()
+ && !(list.getLogicalTypeAnnotation()
+ instanceof
LogicalTypeAnnotation.FiniteVectorLogicalTypeAnnotation)
Review Comment:
One of the rare occasions to use Morgan's law to simplify:
```suggestion
LogicalTypeAnnotation annotation = list.getLogicalTypeAnnotation();
if (!(annotation == LogicalTypeAnnotation.listType()
|| annotation instanceof
LogicalTypeAnnotation.FiniteVectorLogicalTypeAnnotation)
```
##########
parquet-column/src/main/java/org/apache/parquet/schema/TypeUtil.java:
##########
@@ -18,8 +18,106 @@
*/
package org.apache.parquet.schema;
+import
org.apache.parquet.schema.LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+import
org.apache.parquet.schema.LogicalTypeAnnotation.FiniteVectorLogicalTypeAnnotation;
+import
org.apache.parquet.schema.LogicalTypeAnnotation.Float16LogicalTypeAnnotation;
+import
org.apache.parquet.schema.LogicalTypeAnnotation.IntLogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName;
+import org.apache.parquet.schema.Type.Repetition;
+
public class TypeUtil {
+ public static boolean checkValidFiniteVectorSchema(Type type) {
+ boolean found = type.getLogicalTypeAnnotation() instanceof
FiniteVectorLogicalTypeAnnotation;
+ if (found) {
+ if (type.isPrimitive()) {
+ throw new InvalidSchemaException("Invalid FINITE_VECTOR schema: " +
type);
+ }
+ checkVector(type.asGroupType());
+ } else if (!type.isPrimitive()) {
+ for (Type child : type.asGroupType().getFields()) {
+ found |= checkValidFiniteVectorSchema(child);
+ }
+ }
+ return found;
+ }
+
+ private static void checkVector(GroupType vector) {
+ if (vector.getRepetition() == Repetition.REPEATED ||
vector.getFieldCount() != 1) {
+ throw new InvalidSchemaException("Invalid FINITE_VECTOR schema: " +
vector);
+ }
+ Type listType = vector.getType(0);
+ if (listType.isPrimitive()
+ || !listType.getName().equals("list")
+ || listType.getRepetition() != Repetition.REPEATED
+ || listType.getLogicalTypeAnnotation() != null) {
+ throw new InvalidSchemaException("Invalid FINITE_VECTOR schema: " +
vector);
+ }
+ GroupType list = listType.asGroupType();
+ if (list.getFieldCount() != 1) {
+ throw new InvalidSchemaException("Invalid FINITE_VECTOR schema: " +
vector);
+ }
+ Type elementType = list.getType(0);
+ if (!elementType.isPrimitive()
+ || !elementType.getName().equals("element")
+ || elementType.getRepetition() != Repetition.REQUIRED) {
+ throw new InvalidSchemaException("Invalid FINITE_VECTOR schema: " +
vector);
+ }
+ PrimitiveType element = elementType.asPrimitiveType();
+ PrimitiveTypeName physical = element.getPrimitiveTypeName();
+ LogicalTypeAnnotation logical = element.getLogicalTypeAnnotation();
+ boolean allowed;
+ switch (physical) {
+ case BOOLEAN:
+ case FLOAT:
+ case DOUBLE:
+ allowed = logical == null;
+ break;
+ case INT32:
+ case INT64:
+ allowed = logical == null
+ || logical instanceof IntLogicalTypeAnnotation
+ || logical instanceof DecimalLogicalTypeAnnotation;
+ break;
+ case FIXED_LEN_BYTE_ARRAY:
+ allowed = logical instanceof Float16LogicalTypeAnnotation
+ || logical instanceof DecimalLogicalTypeAnnotation;
+ break;
+ default:
+ allowed = false;
+ }
Review Comment:
We're on JDK17, so we can use the sweet enhanced switch:
```suggestion
boolean allowed = switch (physical) {
case BOOLEAN, FLOAT, DOUBLE -> logical == null;
case INT32, INT64 -> logical == null
|| logical instanceof IntLogicalTypeAnnotation
|| logical instanceof DecimalLogicalTypeAnnotation;
case FIXED_LEN_BYTE_ARRAY -> logical instanceof
Float16LogicalTypeAnnotation
|| logical instanceof DecimalLogicalTypeAnnotation;
default -> false;
};
```
So much cleaner 🤤
##########
parquet-column/src/main/java/org/apache/parquet/io/FilteredRecordReader.java:
##########
@@ -84,16 +84,19 @@ private void skipToMatch() {
State currentState = getState(0);
do {
ColumnReader columnReader = currentState.column;
+ int d = columnReader.getCurrentDefinitionLevel();
Review Comment:
```suggestion
int depth = columnReader.getCurrentDefinitionLevel();
```
##########
parquet-column/src/main/java/org/apache/parquet/schema/LogicalTypeAnnotation.java:
##########
@@ -500,6 +511,50 @@ public int hashCode() {
}
}
+ public static class FiniteVectorLogicalTypeAnnotation extends
LogicalTypeAnnotation {
Review Comment:
I think the key here is:
```suggestion
public static class FiniteVectorLogicalTypeAnnotation extends
ListLogicalTypeAnnotation {
```
##########
parquet-column/src/main/java/org/apache/parquet/io/ColumnIOFactory.java:
##########
@@ -53,6 +54,7 @@ private ColumnIOCreatorVisitor(
@Override
public void visit(MessageType messageType) {
+ TypeUtil.checkValidFiniteVectorSchema(messageType);
Review Comment:
I see a lot of random validation, shouldn't we check this much earlier when
the `MessageType` is being constructed?
##########
parquet-avro/src/main/java/org/apache/parquet/avro/AvroSchemaConverter.java:
##########
@@ -470,7 +471,9 @@ public Optional<Schema> visit(
if (!repeatedType.isRepetition(REPEATED)) {
throw new UnsupportedOperationException("Invalid list type "
+ parquetGroupType);
}
- if (isElementType(repeatedType, parquetGroupType.getName())) {
+ if (!(logicalTypeAnnotation
+ instanceof
LogicalTypeAnnotation.FiniteVectorLogicalTypeAnnotation)
+ && isElementType(repeatedType,
parquetGroupType.getName())) {
Review Comment:
Do we really need this addition specifically for
`FiniteVectorLogicalTypeAnnotation`? Feels awkward to me. Shouldn't
`isElementType` handle this?
##########
parquet-hadoop/src/test/resources/vector/vector_invalid_v1_dictionary.parquet:
##########
Review Comment:
Eventually, these should go in https://github.com/apache/parquet-testing/
##########
parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/SchemaConverter.java:
##########
@@ -486,6 +487,13 @@ public Optional<TypeMapping> visit(
return of(new ListTypeMapping(arrowField, list3Levels, child));
}
+ @Override
+ public Optional<TypeMapping> visit(
+ LogicalTypeAnnotation.FiniteVectorLogicalTypeAnnotation
vectorLogicalType) {
+ TypeUtil.checkValidFiniteVectorSchema(type);
+ return visit(LogicalTypeAnnotation.listType());
Review Comment:
For Arrow, we want o make sure that we map to the Fixed:
```suggestion
List3Levels list3Levels = new List3Levels(type);
TypeMapping child = fromParquet(
list3Levels.getElement(),
null,
list3Levels.getElement().getRepetition());
ArrowType arrowType = new
ArrowType.FixedSizeList(vectorLogicalType.getNumElements());
Field arrowField = new Field(
name,
type.isRepetition(OPTIONAL)
? FieldType.nullable(arrowType)
: FieldType.notNullable(arrowType),
Collections.singletonList(child.getArrowField()));
return of(new ListTypeMapping(arrowField, list3Levels, child));
```
##########
parquet-column/src/main/java/org/apache/parquet/schema/TypeUtil.java:
##########
@@ -18,8 +18,106 @@
*/
package org.apache.parquet.schema;
+import
org.apache.parquet.schema.LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+import
org.apache.parquet.schema.LogicalTypeAnnotation.FiniteVectorLogicalTypeAnnotation;
+import
org.apache.parquet.schema.LogicalTypeAnnotation.Float16LogicalTypeAnnotation;
+import
org.apache.parquet.schema.LogicalTypeAnnotation.IntLogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName;
+import org.apache.parquet.schema.Type.Repetition;
+
public class TypeUtil {
+ public static boolean checkValidFiniteVectorSchema(Type type) {
+ boolean found = type.getLogicalTypeAnnotation() instanceof
FiniteVectorLogicalTypeAnnotation;
+ if (found) {
+ if (type.isPrimitive()) {
+ throw new InvalidSchemaException("Invalid FINITE_VECTOR schema: " +
type);
+ }
+ checkVector(type.asGroupType());
+ } else if (!type.isPrimitive()) {
+ for (Type child : type.asGroupType().getFields()) {
+ found |= checkValidFiniteVectorSchema(child);
+ }
+ }
+ return found;
+ }
+
+ private static void checkVector(GroupType vector) {
+ if (vector.getRepetition() == Repetition.REPEATED ||
vector.getFieldCount() != 1) {
+ throw new InvalidSchemaException("Invalid FINITE_VECTOR schema: " +
vector);
+ }
+ Type listType = vector.getType(0);
+ if (listType.isPrimitive()
+ || !listType.getName().equals("list")
+ || listType.getRepetition() != Repetition.REPEATED
+ || listType.getLogicalTypeAnnotation() != null) {
+ throw new InvalidSchemaException("Invalid FINITE_VECTOR schema: " +
vector);
+ }
+ GroupType list = listType.asGroupType();
+ if (list.getFieldCount() != 1) {
+ throw new InvalidSchemaException("Invalid FINITE_VECTOR schema: " +
vector);
+ }
+ Type elementType = list.getType(0);
+ if (!elementType.isPrimitive()
+ || !elementType.getName().equals("element")
+ || elementType.getRepetition() != Repetition.REQUIRED) {
+ throw new InvalidSchemaException("Invalid FINITE_VECTOR schema: " +
vector);
+ }
+ PrimitiveType element = elementType.asPrimitiveType();
+ PrimitiveTypeName physical = element.getPrimitiveTypeName();
+ LogicalTypeAnnotation logical = element.getLogicalTypeAnnotation();
+ boolean allowed;
+ switch (physical) {
+ case BOOLEAN:
+ case FLOAT:
+ case DOUBLE:
+ allowed = logical == null;
+ break;
+ case INT32:
+ case INT64:
+ allowed = logical == null
+ || logical instanceof IntLogicalTypeAnnotation
+ || logical instanceof DecimalLogicalTypeAnnotation;
+ break;
+ case FIXED_LEN_BYTE_ARRAY:
+ allowed = logical instanceof Float16LogicalTypeAnnotation
+ || logical instanceof DecimalLogicalTypeAnnotation;
+ break;
+ default:
+ allowed = false;
+ }
+ if (!allowed) {
+ throw new InvalidSchemaException("Invalid FINITE_VECTOR schema: " +
vector);
+ }
+ try {
+ Types.PrimitiveBuilder<PrimitiveType> builder =
Types.primitive(physical, element.getRepetition());
+ if (physical == PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY) {
+ builder.length(element.getTypeLength());
+ }
+ if (logical != null) {
+ builder.as(logical);
+ }
+ builder.named(element.getName());
Review Comment:
I think this shows the issue. For other types we validate it once we try to
build it, we want something similar for the `FINITE_VECTOR`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]