I think this is reasonable, but it feels like a murky area of the compatibility story. FILE is merged and released, is it forwards/backwards compatible to change the spec on how to read it?
-- Div On Tue, Nov 3, 2026 at 2:20 PM Stefan Grafberger via dev < [email protected]> wrote: > Hi all, > > While preparing support for the FILE logical type, we found that a value > with both inline and uri set can resolve to different bytes in different > engines while all of them remain spec compliant. For example, if the object > at uri is overwritten after the value is written, one reader returns the > inline bytes and another returns the new object. We want our behavior to > match other engines, so we'd like the spec to define which bytes such a > value resolves to. > > We propose restoring the meaning from the original FILE PR [1]: if inline > is set, it supplies the bytes, and any locator fields that are set (uri, > offset, size) are provenance only. This keeps the clarification from [2] > that inline and a locator may both be set, so inline can still serve as a > cached copy of the referenced bytes. We'd also like to allow size with > inline alone, as its length, so readers can get it without reading the > inline bytes. When inline and uri are both set, we'd specify that checksum, > like the locator, describes the source at uri as it was when inline was > copied from it, which the spec currently leaves open. I opened a draft PR > with the corresponding changes at [3]. > > We also considered making inline and the locator fields mutually exclusive, > but that would make values invalid that current writers already produce > (for example, Hardwood [4] and the proposed parquet-testing fixtures [5]) > and would remove the only way to record where an inline copy came from, for > use cases like finding every copy of a file for a deletion request or > explicitly refreshing an inline copy from its source. "Inline wins" > invalidates no data and only narrows what readers may return. > > Looking forward to hearing your thoughts. > > Thanks, > Stefan > > [1] https://github.com/apache/parquet-format/pull/585 > [2] https://github.com/apache/parquet-format/pull/603 > [3] https://github.com/apache/parquet-format/pull/635 > [4] https://github.com/hardwood-hq/hardwood/pull/1494 > [5] https://github.com/apache/parquet-testing/pull/128 >
