Hi all, I agree with Gunnar. It was intentionally left open. In my mind, the inline field with a locator would mostly act as a "cache" of the value for very small data in case we want to do sequential scans. However, I can still just send the URI information to remote services instead of sending the bytes over the wire.
In the use case I'm describing, the inline data can go stale in case the underlying files are mutable. That's when the eTag values would come into play, but that would also mean that you can't avoid an RPC to the cloud. Regarding "using inline to store the data and storing the uri as historical provenance fields", why couldn't the data be stored in separate columns for this use case? I'd love to hear more thoughts around this. Best, Burak On Sun, Nov 8, 2026 at 7:57 AM Gunnar Morling via dev < [email protected]> wrote: > Hey all, > > From what I can see, the spec leaves it open intentionally how values are > resolved: > > "A reader may resolve the value from inline or from the locator, whichever > suits it." > > So I'm not sure whether there's actually anything to be done here from a > spec perspective? It's an application concern whether to read inline or > referenced bytes, or is there any Parquet parser which actually resolves > bytes itself (I don't think there is; if this was the intention, the spec > would also have to touch on authentication)? > > As for the proposed size change, I think that should be a separate concern. > The spec clearly states "offset and size apply only to data referenced by > uri" and in the resolution footnote for inline: "The locator fields may all > be unset. Otherwise, fields set alongside inline must form a locator valid > on its own, so offset requires uri and size." > > AFAICS, the spec treats size strictly as a locator field. Hence a value > setting size without uri is invalid today, and readers that validate per > the spec reject it, so this would be a breaking change rather than merely a > clarification. > > Best, > > --Gunnar > > > On Wed, 4 Nov 2026 at 00:51, Julien Le Dem via dev <[email protected] > > > wrote: > > > This is a case where we under defined the behavior leading to a different > > result depending on the reader. > > Even if we try hard to avoid this situation, this sometimes happens. > > To fix this we'll have to pick which behavior is correct as we can not > stay > > in a situation where different readers return a different value. > > This will lead to declare the other behaviors incorrect and releasing > > bugfixes for those. > > From there we'll maintain compatibility from the spec as it is after > > clarification. > > > > > > On Tue, Nov 3, 2026 at 10:17 AM Divjot Arora via dev < > > [email protected]> > > wrote: > > > > > I think this is reasonable, but it feels like a murky area of the > > > compatibility story. FILE is merged and released, > > > is it forwards/backwards compatible to change the spec on how to read > it? > > > > > > -- Div > > > > > > On Tue, Nov 3, 2026 at 2:20 PM Stefan Grafberger via dev < > > > [email protected]> wrote: > > > > > > > Hi all, > > > > > > > > While preparing support for the FILE logical type, we found that a > > value > > > > with both inline and uri set can resolve to different bytes in > > different > > > > engines while all of them remain spec compliant. For example, if the > > > object > > > > at uri is overwritten after the value is written, one reader returns > > the > > > > inline bytes and another returns the new object. We want our behavior > > to > > > > match other engines, so we'd like the spec to define which bytes > such a > > > > value resolves to. > > > > > > > > We propose restoring the meaning from the original FILE PR [1]: if > > inline > > > > is set, it supplies the bytes, and any locator fields that are set > > (uri, > > > > offset, size) are provenance only. This keeps the clarification from > > [2] > > > > that inline and a locator may both be set, so inline can still serve > > as a > > > > cached copy of the referenced bytes. We'd also like to allow size > with > > > > inline alone, as its length, so readers can get it without reading > the > > > > inline bytes. When inline and uri are both set, we'd specify that > > > checksum, > > > > like the locator, describes the source at uri as it was when inline > was > > > > copied from it, which the spec currently leaves open. I opened a > draft > > PR > > > > with the corresponding changes at [3]. > > > > > > > > We also considered making inline and the locator fields mutually > > > exclusive, > > > > but that would make values invalid that current writers already > produce > > > > (for example, Hardwood [4] and the proposed parquet-testing fixtures > > [5]) > > > > and would remove the only way to record where an inline copy came > from, > > > for > > > > use cases like finding every copy of a file for a deletion request or > > > > explicitly refreshing an inline copy from its source. "Inline wins" > > > > invalidates no data and only narrows what readers may return. > > > > > > > > Looking forward to hearing your thoughts. > > > > > > > > Thanks, > > > > Stefan > > > > > > > > [1] https://github.com/apache/parquet-format/pull/585 > > > > [2] https://github.com/apache/parquet-format/pull/603 > > > > [3] https://github.com/apache/parquet-format/pull/635 > > > > [4] https://github.com/hardwood-hq/hardwood/pull/1494 > > > > [5] https://github.com/apache/parquet-testing/pull/128 > > > > > > > > > >
