Hey all, >From what I can see, the spec leaves it open intentionally how values are resolved:
"A reader may resolve the value from inline or from the locator, whichever suits it." So I'm not sure whether there's actually anything to be done here from a spec perspective? It's an application concern whether to read inline or referenced bytes, or is there any Parquet parser which actually resolves bytes itself (I don't think there is; if this was the intention, the spec would also have to touch on authentication)? As for the proposed size change, I think that should be a separate concern. The spec clearly states "offset and size apply only to data referenced by uri" and in the resolution footnote for inline: "The locator fields may all be unset. Otherwise, fields set alongside inline must form a locator valid on its own, so offset requires uri and size." AFAICS, the spec treats size strictly as a locator field. Hence a value setting size without uri is invalid today, and readers that validate per the spec reject it, so this would be a breaking change rather than merely a clarification. Best, --Gunnar On Wed, 4 Nov 2026 at 00:51, Julien Le Dem via dev <[email protected]> wrote: > This is a case where we under defined the behavior leading to a different > result depending on the reader. > Even if we try hard to avoid this situation, this sometimes happens. > To fix this we'll have to pick which behavior is correct as we can not stay > in a situation where different readers return a different value. > This will lead to declare the other behaviors incorrect and releasing > bugfixes for those. > From there we'll maintain compatibility from the spec as it is after > clarification. > > > On Tue, Nov 3, 2026 at 10:17 AM Divjot Arora via dev < > [email protected]> > wrote: > > > I think this is reasonable, but it feels like a murky area of the > > compatibility story. FILE is merged and released, > > is it forwards/backwards compatible to change the spec on how to read it? > > > > -- Div > > > > On Tue, Nov 3, 2026 at 2:20 PM Stefan Grafberger via dev < > > [email protected]> wrote: > > > > > Hi all, > > > > > > While preparing support for the FILE logical type, we found that a > value > > > with both inline and uri set can resolve to different bytes in > different > > > engines while all of them remain spec compliant. For example, if the > > object > > > at uri is overwritten after the value is written, one reader returns > the > > > inline bytes and another returns the new object. We want our behavior > to > > > match other engines, so we'd like the spec to define which bytes such a > > > value resolves to. > > > > > > We propose restoring the meaning from the original FILE PR [1]: if > inline > > > is set, it supplies the bytes, and any locator fields that are set > (uri, > > > offset, size) are provenance only. This keeps the clarification from > [2] > > > that inline and a locator may both be set, so inline can still serve > as a > > > cached copy of the referenced bytes. We'd also like to allow size with > > > inline alone, as its length, so readers can get it without reading the > > > inline bytes. When inline and uri are both set, we'd specify that > > checksum, > > > like the locator, describes the source at uri as it was when inline was > > > copied from it, which the spec currently leaves open. I opened a draft > PR > > > with the corresponding changes at [3]. > > > > > > We also considered making inline and the locator fields mutually > > exclusive, > > > but that would make values invalid that current writers already produce > > > (for example, Hardwood [4] and the proposed parquet-testing fixtures > [5]) > > > and would remove the only way to record where an inline copy came from, > > for > > > use cases like finding every copy of a file for a deletion request or > > > explicitly refreshing an inline copy from its source. "Inline wins" > > > invalidates no data and only narrows what readers may return. > > > > > > Looking forward to hearing your thoughts. > > > > > > Thanks, > > > Stefan > > > > > > [1] https://github.com/apache/parquet-format/pull/585 > > > [2] https://github.com/apache/parquet-format/pull/603 > > > [3] https://github.com/apache/parquet-format/pull/635 > > > [4] https://github.com/hardwood-hq/hardwood/pull/1494 > > > [5] https://github.com/apache/parquet-testing/pull/128 > > > > > >
