alkis commented on code in PR #603:
URL: https://github.com/apache/parquet-format/pull/603#discussion_r3814407052
##########
LogicalTypes.md:
##########
@@ -735,41 +732,35 @@ only.
A value resolves to bytes based on which of `inline`, `uri`, `offset`, and
`size` are
set:
-| `inline` | `uri` | `offset` | `size` | Resolves to
|
-|----------|-------|----------|--------|-------------------------------------------------------|
-| set | - | - | - | the inline bytes
|
-| - | set | - | - | whole external file at `uri`
|
-| - | set | set | - | invalid
|
-| - | set | - | set | external `uri`, `[0, size)`
|
-| - | set | set | set | external `uri`, `[offset, offset +
size)` |
-| - | - | set | - | invalid
|
-| - | - | - | set | invalid
|
-| - | - | set | set | this file, `[offset, offset + size)`
(self-reference) |
-| - | - | - | - | nothing - invalid
|
+| `inline` | `uri` | `offset` | `size` | Resolves to
|
+|----------|-------|----------|--------|-------------------------------------------|
+| set | - | - | - | the inline bytes
|
+| - | set | - | - | whole external file at `uri`
|
+| - | set | set | - | invalid
|
+| - | set | - | set | external `uri`, `[0, size)`
|
+| - | set | set | set | external `uri`, `[offset, offset +
size)` |
+| - | - | set | - | invalid
|
+| - | - | - | set | invalid
|
+| - | - | set | set | invalid
|
+| - | - | - | - | nothing - invalid
|
`size` must be set whenever `offset` is set, so any offset-based read always
carries an
-explicit `size`. A self-reference (`uri` not set) must set `offset`, and
therefore also
-`size`. `size` may be omitted only for a whole-file external reference, where
the range
-runs to the end of the referenced file.
+explicit `size`. `size` may be omitted only for a whole-file external
reference, where
+the range runs to the end of the referenced file. A byte range within the
current file
+cannot be referenced: `offset` and `size` apply only to data referenced by
`uri`.
-A self-reference points within the same Parquet file using `offset` and `size`
(both
-required). A self-reference is when `uri` is not set. A file containing
self-references
-can be renamed or relocated as a single unit.
-
-Parquet files containing self-references must not use Parquet modular
encryption.
-Self-referenced byte ranges are not Parquet encryption modules and therefore
cannot
-be encrypted or authenticated independently. Encryption of external files
referenced
-by `uri` is outside the scope of the Parquet format.
+Encryption of external files referenced by `uri` is outside the scope of the
Parquet
+format.
Review Comment:
Yes, and worth stating explicitly — the neighbouring sentence about external
files being out of scope invites the wrong inference. Added in f4b79fe:
> Encryption of external files referenced by `uri` is outside the scope of
the Parquet format. The fields of a `FILE`-annotated group are ordinary columns
and are encoded, compressed, and encrypted like any other column, `inline`
included.
So the two are separate concerns: the columns holding the reference get full
modular encryption like any other column, while the bytes sitting in some
external object store are not something Parquet can speak to. `inline` is
called out because it is the case where the payload itself lives in a Parquet
column and therefore is covered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
