I believe the intent is that the inline values always match the referenced part of an external file.
I don't think we should try to specify what happens if someone unintentionally (or intentionally) changes the referenced file so the two representations are inconsistent. This does leave a scenario where they can be inconsistent, but I believe that is an error state. There's no guarantee that the value at the referenced external location is even readable after overwrite, so overwrite can cause issues regardless of the presence of inline data. We shouldn't require readers validate since that would undermine the benefits of the inline storage. The checksum exists if a reader chooses to compare, but the behavior in this scenario is undefined because it normally wouldn't be detected. I don't really think we need to do anything beyond recognize this is possible and encourage implementations to not overwrite referenced files. Implementations can provide ways to detect or correct, but I don't think it's something that requires a spec change to "fix". -Dan On Sun, Nov 8, 2026 at 12:35 PM Burak Yavuz <[email protected]> wrote: > Hi all, > > I agree with Gunnar. It was intentionally left open. In my mind, the inline > field with a locator would mostly act as a "cache" of the value for very > small data in case we want to do sequential scans. However, I can still > just send the URI information to remote services instead of sending the > bytes over the wire. > > In the use case I'm describing, the inline data can go stale in case the > underlying files are mutable. That's when the eTag values would come into > play, but that would also mean that you can't avoid an RPC to the cloud. > > Regarding "using inline to store the data and storing the uri as historical > provenance fields", why couldn't the data be stored in separate columns for > this use case? > > I'd love to hear more thoughts around this. > > Best, > Burak > > On Sun, Nov 8, 2026 at 7:57 AM Gunnar Morling via dev < > [email protected]> wrote: > > > Hey all, > > > > From what I can see, the spec leaves it open intentionally how values are > > resolved: > > > > "A reader may resolve the value from inline or from the locator, > whichever > > suits it." > > > > So I'm not sure whether there's actually anything to be done here from a > > spec perspective? It's an application concern whether to read inline or > > referenced bytes, or is there any Parquet parser which actually resolves > > bytes itself (I don't think there is; if this was the intention, the spec > > would also have to touch on authentication)? > > > > As for the proposed size change, I think that should be a separate > concern. > > The spec clearly states "offset and size apply only to data referenced by > > uri" and in the resolution footnote for inline: "The locator fields may > all > > be unset. Otherwise, fields set alongside inline must form a locator > valid > > on its own, so offset requires uri and size." > > > > AFAICS, the spec treats size strictly as a locator field. Hence a value > > setting size without uri is invalid today, and readers that validate per > > the spec reject it, so this would be a breaking change rather than > merely a > > clarification. > > > > Best, > > > > --Gunnar > > > > > > On Wed, 4 Nov 2026 at 00:51, Julien Le Dem via dev < > [email protected] > > > > > wrote: > > > > > This is a case where we under defined the behavior leading to a > different > > > result depending on the reader. > > > Even if we try hard to avoid this situation, this sometimes happens. > > > To fix this we'll have to pick which behavior is correct as we can not > > stay > > > in a situation where different readers return a different value. > > > This will lead to declare the other behaviors incorrect and releasing > > > bugfixes for those. > > > From there we'll maintain compatibility from the spec as it is after > > > clarification. > > > > > > > > > On Tue, Nov 3, 2026 at 10:17 AM Divjot Arora via dev < > > > [email protected]> > > > wrote: > > > > > > > I think this is reasonable, but it feels like a murky area of the > > > > compatibility story. FILE is merged and released, > > > > is it forwards/backwards compatible to change the spec on how to read > > it? > > > > > > > > -- Div > > > > > > > > On Tue, Nov 3, 2026 at 2:20 PM Stefan Grafberger via dev < > > > > [email protected]> wrote: > > > > > > > > > Hi all, > > > > > > > > > > While preparing support for the FILE logical type, we found that a > > > value > > > > > with both inline and uri set can resolve to different bytes in > > > different > > > > > engines while all of them remain spec compliant. For example, if > the > > > > object > > > > > at uri is overwritten after the value is written, one reader > returns > > > the > > > > > inline bytes and another returns the new object. We want our > behavior > > > to > > > > > match other engines, so we'd like the spec to define which bytes > > such a > > > > > value resolves to. > > > > > > > > > > We propose restoring the meaning from the original FILE PR [1]: if > > > inline > > > > > is set, it supplies the bytes, and any locator fields that are set > > > (uri, > > > > > offset, size) are provenance only. This keeps the clarification > from > > > [2] > > > > > that inline and a locator may both be set, so inline can still > serve > > > as a > > > > > cached copy of the referenced bytes. We'd also like to allow size > > with > > > > > inline alone, as its length, so readers can get it without reading > > the > > > > > inline bytes. When inline and uri are both set, we'd specify that > > > > checksum, > > > > > like the locator, describes the source at uri as it was when inline > > was > > > > > copied from it, which the spec currently leaves open. I opened a > > draft > > > PR > > > > > with the corresponding changes at [3]. > > > > > > > > > > We also considered making inline and the locator fields mutually > > > > exclusive, > > > > > but that would make values invalid that current writers already > > produce > > > > > (for example, Hardwood [4] and the proposed parquet-testing > fixtures > > > [5]) > > > > > and would remove the only way to record where an inline copy came > > from, > > > > for > > > > > use cases like finding every copy of a file for a deletion request > or > > > > > explicitly refreshing an inline copy from its source. "Inline wins" > > > > > invalidates no data and only narrows what readers may return. > > > > > > > > > > Looking forward to hearing your thoughts. > > > > > > > > > > Thanks, > > > > > Stefan > > > > > > > > > > [1] https://github.com/apache/parquet-format/pull/585 > > > > > [2] https://github.com/apache/parquet-format/pull/603 > > > > > [3] https://github.com/apache/parquet-format/pull/635 > > > > > [4] https://github.com/hardwood-hq/hardwood/pull/1494 > > > > > [5] https://github.com/apache/parquet-testing/pull/128 > > > > > > > > > > > > > > >
