brkyvz commented on code in PR #3608:
URL: https://github.com/apache/parquet-java/pull/3608#discussion_r4426857500


##########
parquet-column/src/main/java/org/apache/parquet/schema/LogicalTypeAnnotation.java:
##########
@@ -1229,6 +1240,151 @@ public boolean equals(Object obj) {
     }
   }
 
+  /**
+   * File logical type annotation. Annotates a group (struct) that represents 
a reference to a
+   * range of bytes, which may be stored inline in the value or in an external 
file. Every field is
+   * optional, both in the schema (a writer may omit any field from the group 
definition) and in the
+   * data (any field that is present has a field repetition type of {@code 
OPTIONAL}). Fields are
+   * identified by name (case sensitively), not by field order. A group need 
only define the fields
+   * it uses. The group may contain the following fields:
+   * <ul>
+   *   <li>{@code uri} (STRING): a URI-reference (RFC 3986) that identifies an 
external file, for
+   *       example {@code s3://bucket/file.jpg}.</li>
+   *   <li>{@code offset} (INT64): start of the byte range within the external 
file identified by
+   *       {@code uri}; if not set, treated as 0. Must not be negative and may 
only be set together
+   *       with {@code uri}.</li>
+   *   <li>{@code size} (INT64): byte length of the referenced data within the 
external file. May be
+   *       omitted for a whole-file external reference, in which case the 
range runs to the end of
+   *       the referenced file. May only be set together with {@code uri}, 
must be set whenever
+   *       {@code offset} is set, and must not be negative.</li>
+   *   <li>{@code content_type} (STRING): the media (MIME) type (RFC 2046) of 
the resolved bytes;
+   *       when not set, {@code application/octet-stream} is assumed.</li>
+   *   <li>{@code checksum} (STRING): a self-describing integrity token for 
the resolved bytes, of
+   *       the form {@code <algorithm>:<digest>}.</li>
+   *   <li>{@code inline} (BYTE_ARRAY): the referenced bytes stored inline in 
the value.</li>
+   * </ul>
+   * No fields with names other than the above are permitted, and field names 
must be unique. Each
+   * declared field must match its required physical and logical types. Rules 
involving which fields
+   * are set and their values are enforced per value rather than by the schema.
+   */
+  public static class FileLogicalTypeAnnotation extends LogicalTypeAnnotation {
+    private static final FileLogicalTypeAnnotation INSTANCE = new 
FileLogicalTypeAnnotation();
+
+    /** Field name holding the URI-reference of an external file. */
+    public static final String URI_FIELD = "uri";
+
+    /** Field name holding the start of the byte range. */
+    public static final String OFFSET_FIELD = "offset";
+
+    /** Field name holding the byte length of the referenced data. */
+    public static final String SIZE_FIELD = "size";
+
+    /** Field name holding the media (MIME) type of the resolved bytes. */
+    public static final String CONTENT_TYPE_FIELD = "content_type";
+
+    /** Field name holding the integrity token for the resolved bytes. */
+    public static final String CHECKSUM_FIELD = "checksum";
+
+    /** Field name holding the referenced bytes stored inline. */
+    public static final String INLINE_FIELD = "inline";
+
+    /** All recognized field names in a FILE-annotated group. All fields are 
optional. */
+    public static final Set<String> FIELD_NAMES =
+        Set.of(URI_FIELD, OFFSET_FIELD, SIZE_FIELD, CONTENT_TYPE_FIELD, 
CHECKSUM_FIELD, INLINE_FIELD);
+
+    private FileLogicalTypeAnnotation() {}
+
+    /** Validates the fields of a FILE-annotated group. */
+    public void validate(GroupType group) {
+      validateFields(group.getName(), group.getFields());
+    }
+
+    void validateFields(String groupName, List<Type> fields) {

Review Comment:
   Schema validation now has a single construction-time path: the [GroupType 
constructor](https://github.com/apache/parquet-java/blob/bab237a1f7b57457052f520f7cd5dae84b211e16/parquet-co
           lumn/src/main/java/org/apache/parquet/schema/GroupType.java#L117) 
calls `FileLogicalTypeAnnotation.validateFields`. Removed the repeated 
schema-validation calls from `FileValueValidator`, `Types`, and metadata 
serialization. 
[FileValueValidator.validateValue](https://github.com/apache/parquet-java/blob/bab237a1f7b57457052f520f7cd5dae84b211e16/parquet-column/src/main/java/org/apache/parquet/file/FileValueValidator.java#L35)
 retains the value-dependent locator/range checks shared by the reader and 
writer; its schema check now only requires the FILE annotation.



##########
parquet-column/src/main/java/org/apache/parquet/schema/LogicalTypeAnnotation.java:
##########
@@ -1229,6 +1240,151 @@ public boolean equals(Object obj) {
     }
   }
 
+  /**
+   * File logical type annotation. Annotates a group (struct) that represents 
a reference to a
+   * range of bytes, which may be stored inline in the value or in an external 
file. Every field is
+   * optional, both in the schema (a writer may omit any field from the group 
definition) and in the
+   * data (any field that is present has a field repetition type of {@code 
OPTIONAL}). Fields are
+   * identified by name (case sensitively), not by field order. A group need 
only define the fields
+   * it uses. The group may contain the following fields:
+   * <ul>
+   *   <li>{@code uri} (STRING): a URI-reference (RFC 3986) that identifies an 
external file, for
+   *       example {@code s3://bucket/file.jpg}.</li>
+   *   <li>{@code offset} (INT64): start of the byte range within the external 
file identified by
+   *       {@code uri}; if not set, treated as 0. Must not be negative and may 
only be set together
+   *       with {@code uri}.</li>
+   *   <li>{@code size} (INT64): byte length of the referenced data within the 
external file. May be
+   *       omitted for a whole-file external reference, in which case the 
range runs to the end of
+   *       the referenced file. May only be set together with {@code uri}, 
must be set whenever
+   *       {@code offset} is set, and must not be negative.</li>
+   *   <li>{@code content_type} (STRING): the media (MIME) type (RFC 2046) of 
the resolved bytes;
+   *       when not set, {@code application/octet-stream} is assumed.</li>
+   *   <li>{@code checksum} (STRING): a self-describing integrity token for 
the resolved bytes, of
+   *       the form {@code <algorithm>:<digest>}.</li>
+   *   <li>{@code inline} (BYTE_ARRAY): the referenced bytes stored inline in 
the value.</li>
+   * </ul>
+   * No fields with names other than the above are permitted, and field names 
must be unique. Each
+   * declared field must match its required physical and logical types. Rules 
involving which fields
+   * are set and their values are enforced per value rather than by the schema.
+   */
+  public static class FileLogicalTypeAnnotation extends LogicalTypeAnnotation {
+    private static final FileLogicalTypeAnnotation INSTANCE = new 
FileLogicalTypeAnnotation();
+
+    /** Field name holding the URI-reference of an external file. */
+    public static final String URI_FIELD = "uri";
+
+    /** Field name holding the start of the byte range. */
+    public static final String OFFSET_FIELD = "offset";
+
+    /** Field name holding the byte length of the referenced data. */
+    public static final String SIZE_FIELD = "size";
+
+    /** Field name holding the media (MIME) type of the resolved bytes. */
+    public static final String CONTENT_TYPE_FIELD = "content_type";
+
+    /** Field name holding the integrity token for the resolved bytes. */
+    public static final String CHECKSUM_FIELD = "checksum";
+
+    /** Field name holding the referenced bytes stored inline. */
+    public static final String INLINE_FIELD = "inline";
+
+    /** All recognized field names in a FILE-annotated group. All fields are 
optional. */
+    public static final Set<String> FIELD_NAMES =
+        Set.of(URI_FIELD, OFFSET_FIELD, SIZE_FIELD, CONTENT_TYPE_FIELD, 
CHECKSUM_FIELD, INLINE_FIELD);
+
+    private FileLogicalTypeAnnotation() {}
+
+    /** Validates the fields of a FILE-annotated group. */
+    public void validate(GroupType group) {
+      validateFields(group.getName(), group.getFields());
+    }
+
+    void validateFields(String groupName, List<Type> fields) {

Review Comment:
   Schema validation now has a single construction-time path: the [GroupType 
constructor](https://github.com/apache/parquet-java/blob/bab237a1f7b57457052f520f7cd5dae84b211e16/parquet-column/src/main/java/org/apache/parquet/schema/GroupType.java#L117)
 calls `FileLogicalTypeAnnotation.validateFields`. Removed the repeated 
schema-validation calls from `FileValueValidator`, `Types`, and metadata 
serialization. 
[FileValueValidator.validateValue](https://github.com/apache/parquet-java/blob/bab237a1f7b57457052f520f7cd5dae84b211e16/parquet-column/src/main/java/org/apache/parquet/file/FileValueValidator.java#L35)
 retains the value-dependent locator/range checks shared by the reader and 
writer; its schema check now only requires the FILE annotation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to