divjotarora commented on code in PR #3608:
URL: https://github.com/apache/parquet-java/pull/3608#discussion_r4387478733
##########
parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java:
##########
@@ -2184,7 +2194,15 @@ && getLogicalTypeAnnotation(schemaElement.logicalType)
!= null)
childBuilder.id(schemaElement.field_id);
}
- childBuilder.named(schemaElement.name);
+ try {
+ childBuilder.named(schemaElement.name);
+ } catch (IllegalArgumentException e) {
+ if (!fileLogicalType) {
+ throw e;
+ }
+ LOG.warn("Dropping malformed FILE logical type annotation on group
{}", schemaElement.name, e);
+ childBuilder.as((LogicalTypeAnnotation)
null).named(schemaElement.name);
+ }
Review Comment:
Why do we suppress errors here and drop the type annotation? For other
logical types, we only validate that the logical/physical type combination is
valid. If the combination is valid but the annotation's parameters are invalid
or corrupted, we don't drop the annotation and instead raise that error. For
example, if `DECIMAL` annotates a `BOOLEAN` column, that's an invalid
combination and gets suppressed. But if `DECIMAL` annotates an FLBA column but
declares an invalid precision, that's a valid combination with invalid
parameters and throws.
IMO we should do the same here: as long as FILE annotates a struct column,
we should error if the FILE schema validation fails.
##########
parquet-column/src/main/java/org/apache/parquet/schema/LogicalTypeAnnotation.java:
##########
Review Comment:
We should have a simple test case to validate parsing of a text FILE schema
in
`parquet-column/src/test/java/org/apache/parquet/parser/TestParquetParser.java`.
The schema can be something like
```
optional group file (FILE) {
optional binary uri (STRING);
}
```
##########
parquet-column/src/test/java/org/apache/parquet/io/api/TestFileValueWriter.java:
##########
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.io.api;
+
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.BINARY;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.INT64;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.util.ArrayDeque;
+import java.util.Arrays;
+import java.util.Deque;
+import org.apache.parquet.io.ExpectationValidatingRecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.Types;
+import org.junit.jupiter.api.Test;
+
+public class TestFileValueWriter {
+ private static final GroupType ALL_FIELDS_SCHEMA = Types.optionalGroup()
+ .as(LogicalTypeAnnotation.fileType())
+ .optional(BINARY)
+ .as(LogicalTypeAnnotation.stringType())
+ .named("uri")
+ .optional(INT64)
+ .named("offset")
+ .optional(INT64)
+ .named("size")
+ .optional(BINARY)
+ .as(LogicalTypeAnnotation.stringType())
+ .named("content_type")
+ .optional(BINARY)
+ .as(LogicalTypeAnnotation.stringType())
+ .named("checksum")
+ .optional(BINARY)
+ .named("inline")
+ .named("file");
+
+ @Test
+ public void testWritesSetFieldsInSchemaOrder() {
Review Comment:
[nit] This case orders the FILE fields in the same order as the spec, so a
bug in the writer that causes it to use the canonical order rather than the
supplied one would still pass. Can we reorder the fields on the `FileValue`
builder to exercise this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]