brkyvz commented on code in PR #3608:
URL: https://github.com/apache/parquet-java/pull/3608#discussion_r4426857500
##########
parquet-column/src/main/java/org/apache/parquet/schema/LogicalTypeAnnotation.java:
##########
@@ -1229,6 +1240,151 @@ public boolean equals(Object obj) {
}
}
+ /**
+ * File logical type annotation. Annotates a group (struct) that represents
a reference to a
+ * range of bytes, which may be stored inline in the value or in an external
file. Every field is
+ * optional, both in the schema (a writer may omit any field from the group
definition) and in the
+ * data (any field that is present has a field repetition type of {@code
OPTIONAL}). Fields are
+ * identified by name (case sensitively), not by field order. A group need
only define the fields
+ * it uses. The group may contain the following fields:
+ * <ul>
+ * <li>{@code uri} (STRING): a URI-reference (RFC 3986) that identifies an
external file, for
+ * example {@code s3://bucket/file.jpg}.</li>
+ * <li>{@code offset} (INT64): start of the byte range within the external
file identified by
+ * {@code uri}; if not set, treated as 0. Must not be negative and may
only be set together
+ * with {@code uri}.</li>
+ * <li>{@code size} (INT64): byte length of the referenced data within the
external file. May be
+ * omitted for a whole-file external reference, in which case the
range runs to the end of
+ * the referenced file. May only be set together with {@code uri},
must be set whenever
+ * {@code offset} is set, and must not be negative.</li>
+ * <li>{@code content_type} (STRING): the media (MIME) type (RFC 2046) of
the resolved bytes;
+ * when not set, {@code application/octet-stream} is assumed.</li>
+ * <li>{@code checksum} (STRING): a self-describing integrity token for
the resolved bytes, of
+ * the form {@code <algorithm>:<digest>}.</li>
+ * <li>{@code inline} (BYTE_ARRAY): the referenced bytes stored inline in
the value.</li>
+ * </ul>
+ * No fields with names other than the above are permitted, and field names
must be unique. Each
+ * declared field must match its required physical and logical types. Rules
involving which fields
+ * are set and their values are enforced per value rather than by the schema.
+ */
+ public static class FileLogicalTypeAnnotation extends LogicalTypeAnnotation {
+ private static final FileLogicalTypeAnnotation INSTANCE = new
FileLogicalTypeAnnotation();
+
+ /** Field name holding the URI-reference of an external file. */
+ public static final String URI_FIELD = "uri";
+
+ /** Field name holding the start of the byte range. */
+ public static final String OFFSET_FIELD = "offset";
+
+ /** Field name holding the byte length of the referenced data. */
+ public static final String SIZE_FIELD = "size";
+
+ /** Field name holding the media (MIME) type of the resolved bytes. */
+ public static final String CONTENT_TYPE_FIELD = "content_type";
+
+ /** Field name holding the integrity token for the resolved bytes. */
+ public static final String CHECKSUM_FIELD = "checksum";
+
+ /** Field name holding the referenced bytes stored inline. */
+ public static final String INLINE_FIELD = "inline";
+
+ /** All recognized field names in a FILE-annotated group. All fields are
optional. */
+ public static final Set<String> FIELD_NAMES =
+ Set.of(URI_FIELD, OFFSET_FIELD, SIZE_FIELD, CONTENT_TYPE_FIELD,
CHECKSUM_FIELD, INLINE_FIELD);
+
+ private FileLogicalTypeAnnotation() {}
+
+ /** Validates the fields of a FILE-annotated group. */
+ public void validate(GroupType group) {
+ validateFields(group.getName(), group.getFields());
+ }
+
+ void validateFields(String groupName, List<Type> fields) {
Review Comment:
Schema validation now has a single construction-time path: the [GroupType
constructor](https://github.com/apache/parquet-java/blob/bab237a1f7b57457052f520f7cd5dae84b211e16/parquet-co
lumn/src/main/java/org/apache/parquet/schema/GroupType.java#L117)
calls `FileLogicalTypeAnnotation.validateFields`. Removed the repeated
schema-validation calls from `FileValueValida
tor`, `Types`, and metadata serialization.
[FileValueValidator.validateValue](https://github.com/apache/parquet-java/blob/bab237a1f7b57457052f520f7cd5dae84b211e16/parquet-column/src/
main/java/org/apache/parquet/file/FileValueValidator.java#L35)
retains the value-dependent locator/range checks shared by the reader and
writer; its schema check now only requires th
e FILE annotation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]