This is an automated email from the ASF dual-hosted git repository.
emkornfield pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/parquet-format.git
The following commit(s) were added to refs/heads/master by this push:
new c6a6967 Introduces a new LogicalType: FILE (#585)
c6a6967 is described below
commit c6a6967f53906174567eca7d66e9abede1424723
Author: Burak Yavuz <[email protected]>
AuthorDate: Sun Jul 26 01:29:46 2026 -0700
Introduces a new LogicalType: FILE (#585)
* Add FILE type definitions
---
LogicalTypes.md | 182 ++++++++++++++++++++++++++++++++++++++++-
src/main/thrift/parquet.thrift | 14 ++++
2 files changed, 195 insertions(+), 1 deletion(-)
diff --git a/LogicalTypes.md b/LogicalTypes.md
index 690ae3f..fae874c 100644
--- a/LogicalTypes.md
+++ b/LogicalTypes.md
@@ -635,7 +635,187 @@ The type has two type parameters:
The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max
statistics should be saved for this type and if such non-compliant statistics
-are found during reading, they must be ignored.
+are found during reading, they must be ignored.
+
+### FILE
+
+`FILE` annotates a group that represents a reference to a range of bytes,
which may
+be stored inline in the value, elsewhere within the current file, or in an
external file. It
+is intended for use cases such as storing file inventories, manifests, and
unstructured
+data references (e.g., images or audio files stored in object storage).
+
+The annotated group may contain the following fields, identified by name case
sensitively,
+not by field order. Field IDs, if they exist, may also be used for projection.
Every field
+is optional both in the schema and in the data: a writer may omit any field
from the group
+definition, and any field that is present has a field repetition type of
`OPTIONAL`.
+A group need only define the fields it uses (for example, an inline-only group
may define
+just `inline`, and an external reference may define just `uri`).
+
+| Field | Type |
+|----------------|------------|
+| `uri` | STRING |
+| `offset` | INT64 |
+| `size` | INT64 |
+| `content_type` | STRING |
+| `checksum` | STRING |
+| `inline` | BYTE_ARRAY |
+
+A value resolves to bytes determined by `inline` / `uri` / `offset` / `size`;
+`content_type` and `checksum` are metadata describing whatever is resolved.
+
+#### Fields
+
+For the descriptions below, a field is *set* when it is present in the `FILE`
group
+and its value is non-null (and, for string fields, non-empty[1]). A field is
*not set*
+when it is absent from the group, or is present but null or empty.
+
+[1] Implementations are not expected to treat empty strings as null
+
+##### uri
+
+A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g.,
`s3://bucket/file.jpg`).
+The URI may be absolute or relative. No additional encoding (e.g., URI
encoding) is applied on top
+of the user-provided data. If `uri` is not set, the value refers to the
current file
+(a self-reference).
+
+##### offset
+
+A byte offset indicating the start of the byte range within the referenced
data.
+If not set, readers must treat the value as 0.
+If set and non-zero, readers must seek to this offset to retrieve the
referenced data.
+`offset` must be set for a self-reference (`uri` not set); it is optional for
an
+external reference (`uri` set). `offset` must not be < 0.
+
+##### size
+
+The byte length of the referenced data. Must be zero or a positive integer if
set; a
+value of 0 indicates empty referenced data. `size` must be set whenever
`offset` is set.
+It may be omitted only for a whole-file external reference (`uri` set,
`offset` not set),
+in which case the range runs to the end of the referenced file. Because a
self-reference
+always sets `offset`, it always sets `size` as well.
+
+##### content_type
+
+The media type (MIME type), as defined by RFC 2046, of the resolved bytes
(e.g., `image/png`).
+When not set, the type can be assumed as `application/octet-stream`.
+
+##### checksum
+
+A self-describing integrity token for the resolved bytes, of the form
+`<algorithm>:<digest>`, where `<digest>` is encoded according to the `Encoding`
+column below. Readers should ignore unknown algorithms. The recognized
algorithms
+are:
+
+| Algorithm | Encoding | Notes
|
+|-----------|---------------|----------------------------------------------------------|
+| `ETAG` | opaque | the object-store eTag, not recomputable
|
+| `MD5` | lowercase hex | as defined in RFC 1321 represented as 32 hex
characters |
+| `CRC32` | lowercase hex | as defined in RFC 2083, represented as 8 hex
characters |
+| `CRC32C` | lowercase hex | as defined in RFC 3385, represented as 8 hex
characters |
+| `SHA-256` | lowercase hex | as defined in RFC 6234, represented as 64 hex
characters |
+
+`<digest>` encodings are:
+
+* `lowercase hex`: the digest bytes rendered as lowercase hexadecimal, two
+ characters per byte and no separators (e.g.
`MD5:d41d8cd98f00b204e9800998ecf8427e`).
+* `opaque`: the token supplied verbatim by the object store, used only for
+ equality comparison and not otherwise interpreted.
+
+`checksum` applies to the resolved bytes, except for `ETAG`, which is the
+object-store eTag for the whole file referenced by `uri`.
+
+##### inline
+
+The referenced bytes stored inline in the value. If `inline` is set, it
supplies the
+bytes and any locator fields (`uri`, `offset`, `size`) that are set are
provenance
+only.
+
+#### Resolution
+
+A value resolves to bytes based on which of `inline`, `uri`, `offset`, and
`size` are
+set:
+
+| `inline` | `uri` | `offset` | `size` | Resolves to
|
+|----------|-------|----------|--------|-------------------------------------------------------|
+| set | - | - | - | the inline bytes
|
+| - | set | - | - | whole external file at `uri`
|
+| - | set | set | - | invalid
|
+| - | set | - | set | external `uri`, `[0, size)`
|
+| - | set | set | set | external `uri`, `[offset, offset +
size)` |
+| - | - | set | - | invalid
|
+| - | - | - | set | invalid
|
+| - | - | set | set | this file, `[offset, offset + size)`
(self-reference) |
+| - | - | - | - | nothing - invalid
|
+
+`size` must be set whenever `offset` is set, so any offset-based read always
carries an
+explicit `size`. A self-reference (`uri` not set) must set `offset`, and
therefore also
+`size`. `size` may be omitted only for a whole-file external reference, where
the range
+runs to the end of the referenced file.
+
+A self-reference points within the same Parquet file using `offset` and `size`
(both
+required). A self-reference is when `uri` is not set. A file containing
self-references
+can be renamed or relocated as a single unit.
+
+Parquet files containing self-references must not use Parquet modular
encryption.
+Self-referenced byte ranges are not Parquet encryption modules and therefore
cannot
+be encrypted or authenticated independently. Encryption of external files
referenced
+by `uri` is outside the scope of the Parquet format.
+
+#### Validation
+
+* A value must resolve to some referenced data. It resolves only if `inline`,
`uri`, or
+ `offset` is set; if none of them are set, the value does not resolve and is
invalid, even
+ if `size` is set.
+* A self-reference (`uri` not set) must set `offset`. A value with neither
`uri` nor
+ `offset` set (and not `inline`) does not resolve and is invalid.
+* `size` must be set whenever `offset` is set. A value that sets `offset`
without `size`
+ is invalid. Because a self-reference must set `offset`, it must also set
`size`.
+* If `inline` is set, it supplies the bytes for readers; producers may treat
`inline` and the
+ locator fields as mutually exclusive.
+* Field names within a `FILE`-annotated group must not be renamed.
+* Additional metadata about the file (e.g., modification timestamp) must
+ be stored adjacent to this group by engines or table formats, not inside it.
+* If a reader comes across an invalid file reference, the reader may return a
`null` file reference
+ for that row.
+
+Statistics may be collected for the individual fields of a `FILE`-annotated
group
+according to the sort order defined in each field's logical type.
+
+This is an example of a `FILE`-annotated group that defines all fields:
+
+```
+optional group my_file (FILE) {
+ optional binary uri (STRING);
+ optional int64 offset;
+ optional int64 size;
+ optional binary content_type (STRING);
+ optional binary checksum (STRING);
+ optional binary inline;
+}
+```
+
+Because every field is optional, a group need only define the fields it uses.
A group
+whose values are always stored inline may define just `inline` and optionally
`content_type`
+as additional metadata:
+
+```
+optional group inline_file (FILE) {
+ optional binary inline;
+ optional binary content_type (STRING);
+}
+```
+
+A group whose values are always whole external files may define just `uri` and
optionally
+`content_type` and `checksum` for validation:
+
+```
+optional group external_file (FILE) {
+ optional binary uri (STRING);
+ optional binary content_type (STRING);
+ optional binary checksum (STRING);
+}
+```
+
## Nested Types
diff --git a/src/main/thrift/parquet.thrift b/src/main/thrift/parquet.thrift
index c057966..a5922bb 100644
--- a/src/main/thrift/parquet.thrift
+++ b/src/main/thrift/parquet.thrift
@@ -468,6 +468,18 @@ struct GeographyType {
2: optional EdgeInterpolationAlgorithm algorithm;
}
+/**
+ * File logical type annotation
+ *
+ * Annotates a group that represents a reference to a file, or to a range of
+ * bytes that may be stored inline, elsewhere in this file, or in an external
+ * file.
+ *
+ * See LogicalTypes.md for details.
+ */
+struct FileType {
+}
+
/**
* LogicalType annotations to replace ConvertedType.
*
@@ -501,6 +513,7 @@ union LogicalType {
16: VariantType VARIANT // no compatible ConvertedType
17: GeometryType GEOMETRY // no compatible ConvertedType
18: GeographyType GEOGRAPHY // no compatible ConvertedType
+ 19: FileType FILE // no compatible ConvertedType
}
/**
@@ -1108,6 +1121,7 @@ union ColumnOrder {
* VARIANT - undefined
* GEOMETRY - undefined
* GEOGRAPHY - undefined
+ * FILE - undefined
*
* In the absence of logical types, the sort order is determined by the
physical type:
* BOOLEAN - false, true
