This is a case where we under defined the behavior leading to a different result depending on the reader. Even if we try hard to avoid this situation, this sometimes happens. To fix this we'll have to pick which behavior is correct as we can not stay in a situation where different readers return a different value. This will lead to declare the other behaviors incorrect and releasing bugfixes for those. >From there we'll maintain compatibility from the spec as it is after clarification.
On Tue, Nov 3, 2026 at 10:17 AM Divjot Arora via dev <[email protected]> wrote: > I think this is reasonable, but it feels like a murky area of the > compatibility story. FILE is merged and released, > is it forwards/backwards compatible to change the spec on how to read it? > > -- Div > > On Tue, Nov 3, 2026 at 2:20 PM Stefan Grafberger via dev < > [email protected]> wrote: > > > Hi all, > > > > While preparing support for the FILE logical type, we found that a value > > with both inline and uri set can resolve to different bytes in different > > engines while all of them remain spec compliant. For example, if the > object > > at uri is overwritten after the value is written, one reader returns the > > inline bytes and another returns the new object. We want our behavior to > > match other engines, so we'd like the spec to define which bytes such a > > value resolves to. > > > > We propose restoring the meaning from the original FILE PR [1]: if inline > > is set, it supplies the bytes, and any locator fields that are set (uri, > > offset, size) are provenance only. This keeps the clarification from [2] > > that inline and a locator may both be set, so inline can still serve as a > > cached copy of the referenced bytes. We'd also like to allow size with > > inline alone, as its length, so readers can get it without reading the > > inline bytes. When inline and uri are both set, we'd specify that > checksum, > > like the locator, describes the source at uri as it was when inline was > > copied from it, which the spec currently leaves open. I opened a draft PR > > with the corresponding changes at [3]. > > > > We also considered making inline and the locator fields mutually > exclusive, > > but that would make values invalid that current writers already produce > > (for example, Hardwood [4] and the proposed parquet-testing fixtures [5]) > > and would remove the only way to record where an inline copy came from, > for > > use cases like finding every copy of a file for a deletion request or > > explicitly refreshing an inline copy from its source. "Inline wins" > > invalidates no data and only narrows what readers may return. > > > > Looking forward to hearing your thoughts. > > > > Thanks, > > Stefan > > > > [1] https://github.com/apache/parquet-format/pull/585 > > [2] https://github.com/apache/parquet-format/pull/603 > > [3] https://github.com/apache/parquet-format/pull/635 > > [4] https://github.com/hardwood-hq/hardwood/pull/1494 > > [5] https://github.com/apache/parquet-testing/pull/128 > > >
