Hi all, While preparing support for the FILE logical type, we found that a value with both inline and uri set can resolve to different bytes in different engines while all of them remain spec compliant. For example, if the object at uri is overwritten after the value is written, one reader returns the inline bytes and another returns the new object. We want our behavior to match other engines, so we'd like the spec to define which bytes such a value resolves to.
We propose restoring the meaning from the original FILE PR [1]: if inline is set, it supplies the bytes, and any locator fields that are set (uri, offset, size) are provenance only. This keeps the clarification from [2] that inline and a locator may both be set, so inline can still serve as a cached copy of the referenced bytes. We'd also like to allow size with inline alone, as its length, so readers can get it without reading the inline bytes. When inline and uri are both set, we'd specify that checksum, like the locator, describes the source at uri as it was when inline was copied from it, which the spec currently leaves open. I opened a draft PR with the corresponding changes at [3]. We also considered making inline and the locator fields mutually exclusive, but that would make values invalid that current writers already produce (for example, Hardwood [4] and the proposed parquet-testing fixtures [5]) and would remove the only way to record where an inline copy came from, for use cases like finding every copy of a file for a deletion request or explicitly refreshing an inline copy from its source. "Inline wins" invalidates no data and only narrows what readers may return. Looking forward to hearing your thoughts. Thanks, Stefan [1] https://github.com/apache/parquet-format/pull/585 [2] https://github.com/apache/parquet-format/pull/603 [3] https://github.com/apache/parquet-format/pull/635 [4] https://github.com/hardwood-hq/hardwood/pull/1494 [5] https://github.com/apache/parquet-testing/pull/128
