Hi Jiayi, I took a pass reviewing. Overall, I like the design. I left a bunch of comments on the footer proto, which I think is pretty dense and could use a cleanup/augmenting of documentation so readers can better piece together how everything fits together.
Thanks, Micah On Tue, Nov 3, 2026 at 4:21 PM Julien Le Dem via dev <[email protected]> wrote: > Thank you Jiayi for leading this effort and everyone who contributed. > I would encourage folks on the list to review as it is fairly > transformational for the project. > I expect that after that we'll avoid changing the footer format again for a > while. > > > On Fri, Oct 30, 2026 at 7:20 AM Jiayi Wang <[email protected]> wrote: > > > Hi everyone, > > > > As mentioned previously, we have revisited the proposal [1] to make it > > more flexible and also address existing feedback. I’d like to share the > new > > version of the proposal [2], which is included as a new tab in the same > > document. A summary of what changed and why: > > Flexible Data Layouts > > > > Per discussions on evolving Parquet data layout (e.g. the “non-contiguous > > pages” proposal), our goal is to have the new footer handle both existing > > and potential future layouts seamlessly. The updated proposal doesn’t > > assume that row groups are physical boundaries. It preserves the wording > of > > “column chunks”, but these are simply defined to mean a contiguous set of > > pages for one column. Different columns may have different numbers of > > chunks in a file. The “row group” wording is preserved as well, but row > > group metadata is made optional. Lastly, stats are re-defined based on > row > > ranges rather than row groups; different columns may publish stats at > > different granularities. > > Page-based Representation > > > > A big feedback point on the initial proposal was that the EncodedArray > > concept was opaque/confusing. It was added to efficiently model arrays in > > Thrift and provide random access, at the expense of readability. The > > updated proposal uses Parquet pages to represent modules, which yields a > > self-describing layout that evolves naturally as Parquet gains new > > encodings. It also allows for optional per-page compression. > > > > Overall, the new proposal yields a footer that is comparable in size and > > performance to the original proposal, but is simpler and more flexible > for > > handling future Parquet layout changes. We appreciate any feedback on the > > proposal. > > > > Best, > > > > Jiayi > > > > [1] > > > > > https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?tab=t.o42wlmqbbdma#heading=h.kfs863ye4rzw > > > > [2] > > > > > https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?tab=t.cbl5qtgc5xhz > > > > > > > > Jiayi Wang <[email protected]> 于2026年10月19日周一 20:49写道： > > > > > Hi everyone, > > > > > > > > > I'd like to share an update on the progress for the modular footer > > > proposal [1]. We've received some valuable feedback and one recurring > > > suggestion was to make the footer more future-proof by reusing Parquet > > > encodings. > > > > > > > > > We previously ruled out this approach because existing Parquet > encodings > > > don't support random access. However, with PFOR under development and > > > potentially more random-access-friendly encodings coming in the future, > > we > > > think it may be worth revisiting this idea. > > > > > > > > > Another frequently raised point was compression, especially for cases > > > where the footer becomes large enough that its size is a major > bottleneck > > > for cold reads. If we reuse Parquet pages to store footer metadata, we > > > could also get compression support for free, potentially reducing > footer > > > size even further. > > > > > > > > > At the same time, we want to make the modular footer flexible enough to > > > accommodate future changes to the Parquet data layout. For example, the > > > previously proposed non-contiguous pages layout [2] would require the > > > optional offset index to be present, and we'd like to make its > > > representation more efficient. Our goal is to design a footer that can > > > support potential layout changes like this while still representing the > > > current layout efficiently. > > > > > > > > > We're still exploring these directions and will share updates as we > learn > > > more. In the meantime, please feel free to share any thoughts, > concerns, > > or > > > alternative ideas! > > > > > > > > > Thanks again to everyone who reviewed the proposal and provided > feedback! > > > > > > > > > Best, > > > Jiayi > > > > > > > > > [1] > > > > > > https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?tab=t.o42wlmqbbdma > > > > > > [2] > > > > > > https://docs.google.com/document/d/1nntcYM98PFSkHT70RexSBPtCnWqg1uRJ5_7m--ZgbsA/edit?tab=t.0#heading=h.k4r8orckhbx0 > > > > > > Jiayi Wang <[email protected]> 于2026年9月21日周一 17:26写道： > > > > > >> Hi everyone, > > >> > > >> We’ve been discussing the Modular Footer for quite a while, including > > >> through a dedicated footer working group meeting every two weeks, and > we > > >> finally have a proposal ready for broader review. > > >> > > >> The proposal covers the motivation, format design, compatibility > > >> considerations, and benchmark results. Huge thanks to everyone who has > > >> contributed to the project and the discussions so far. > > >> > > >> The proposal doc, parquet-format PR, and tracking issue are all > > available > > >> here: > > >> > > >> Proposal doc: > > >> > > > https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?usp=sharing > > >> > > >> PR: https://github.com/apache/parquet-format/pull/623/changes > > >> > > >> Parquet-format issue: > > https://github.com/apache/parquet-format/issues/622 > > >> > > >> The proposal introduces several significant conceptual changes, so I > > >> expect there will be a lot of discussion and feedback. For the first > > round > > >> of review, I’d suggest focusing primarily on the high-level direction > > and > > >> design choices, and leaving more detailed questions (for example, > > whether a > > >> particular field can be deprecated) for later discussion. I think this > > will > > >> help us converge on the major design questions first and keep the > review > > >> process moving efficiently. > > >> > > >> Thanks! > > >> > > >> Best, > > >> Jiayi > > >> > > > > > Jiayi Wang <[email protected]> 于2026年10月19日周一 20:49写道： > > > > > Hi everyone, > > > > > > > > > I'd like to share an update on the progress for the modular footer > > > proposal [1]. We've received some valuable feedback and one recurring > > > suggestion was to make the footer more future-proof by reusing Parquet > > > encodings. > > > > > > > > > We previously ruled out this approach because existing Parquet > encodings > > > don't support random access. However, with PFOR under development and > > > potentially more random-access-friendly encodings coming in the future, > > we > > > think it may be worth revisiting this idea. > > > > > > > > > Another frequently raised point was compression, especially for cases > > > where the footer becomes large enough that its size is a major > bottleneck > > > for cold reads. If we reuse Parquet pages to store footer metadata, we > > > could also get compression support for free, potentially reducing > footer > > > size even further. > > > > > > > > > At the same time, we want to make the modular footer flexible enough to > > > accommodate future changes to the Parquet data layout. For example, the > > > previously proposed non-contiguous pages layout [2] would require the > > > optional offset index to be present, and we'd like to make its > > > representation more efficient. Our goal is to design a footer that can > > > support potential layout changes like this while still representing the > > > current layout efficiently. > > > > > > > > > We're still exploring these directions and will share updates as we > learn > > > more. In the meantime, please feel free to share any thoughts, > concerns, > > or > > > alternative ideas! > > > > > > > > > Thanks again to everyone who reviewed the proposal and provided > feedback! > > > > > > > > > Best, > > > Jiayi > > > > > > > > > [1] > > > > > > https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?tab=t.o42wlmqbbdma > > > > > > [2] > > > > > > https://docs.google.com/document/d/1nntcYM98PFSkHT70RexSBPtCnWqg1uRJ5_7m--ZgbsA/edit?tab=t.0#heading=h.k4r8orckhbx0 > > > > > > Jiayi Wang <[email protected]> 于2026年9月21日周一 17:26写道： > > > > > >> Hi everyone, > > >> > > >> We’ve been discussing the Modular Footer for quite a while, including > > >> through a dedicated footer working group meeting every two weeks, and > we > > >> finally have a proposal ready for broader review. > > >> > > >> The proposal covers the motivation, format design, compatibility > > >> considerations, and benchmark results. Huge thanks to everyone who has > > >> contributed to the project and the discussions so far. > > >> > > >> The proposal doc, parquet-format PR, and tracking issue are all > > available > > >> here: > > >> > > >> Proposal doc: > > >> > > > https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?usp=sharing > > >> > > >> PR: https://github.com/apache/parquet-format/pull/623/changes > > >> > > >> Parquet-format issue: > > https://github.com/apache/parquet-format/issues/622 > > >> > > >> The proposal introduces several significant conceptual changes, so I > > >> expect there will be a lot of discussion and feedback. For the first > > round > > >> of review, I’d suggest focusing primarily on the high-level direction > > and > > >> design choices, and leaving more detailed questions (for example, > > whether a > > >> particular field can be deprecated) for later discussion. I think this > > will > > >> help us converge on the major design questions first and keep the > review > > >> process moving efficiently. > > >> > > >> Thanks! > > >> > > >> Best, > > >> Jiayi > > >> > > > > > >
