Notes are available here
<https://docs.google.com/document/d/e/2PACX-1vSDHW7gvG8eO6aIxaIVPrZSqYYhtRDb5W1imnbpM4QRYNPsTwEO1fU5z7SEhVIFa4YqWJeSRJ9tcXYS/pub>
.
Thank you all who participated.
| Nov 4, 2026 Apache Parquet Community Sync.
Attendees:
- {attendee}: {org}, {topic}
-
- Philipp Fischbeck: Snowflake, vector
- Sebastian Baunsgaard: Databricks, listening in
- Fokko Driesprong: Databricks, Listening in
- Andrew Lamb: InfluxData, None
- Kurtis Wright: AWS, Vector, Versioning, Footer
- Rok Mihevc: G-Research/Arctos Alliance, vectors, footers
- Connor Tsui: Spiral, listening in
- Martin Prammer: Spiral, listening in
- Mashrafee Aryan: MLH, listening in
- Jiayi Wang: Databricks, footer
- Russell Spitzer: Snowflake - Vector, Encodings, Footer, Automated
Release 😀
- Alkis Evlogimenos: Databricks, footer, vector, encodings, versioning
- Dave Bunten, CU Anschutz, listening in
- Talat Uyarer, Google, listening in
- Daniel Weeks, Databricks: footer, vector, decfloat
- Prateek Gaur : Snowflake : footer, encodings
- Julien Le Dem: Datadog, versioning
- Daniel Tenedorio: Databricks, vectors, footer work, encodings
- Jigao Luo, a PhD student at TU Darmstadt, listening in
Agenda/Notes
- (Rok) Vector Type
- Vote in progress - ML thread
<https://www.google.com/url?q=https://lists.apache.org/thread/fm2373y1d0pk20x6rw8jdkl6kwvfoc32&sa=D&source=editors&ust=1793837552659309&usg=AOvVaw2C1fTdFbygrVj9s29tZVrC>
- Discussion about “Writers MUST enforce [..] finite-element
requirements.” See comment
<https://www.google.com/url?q=https://github.com/apache/parquet-format/pull/624%23discussion_r4466168398&sa=D&source=editors&ust=1793837552659790&usg=AOvVaw1SG1QkmE6RNLekN38VAGQm>
- Minus one vote. => adjust wording accordingly and proceed with vote.
- (Dan) Decfloat:
- Dev List Thread
<https://www.google.com/url?q=https://lists.apache.org/thread.html/h2qc2hh464095lnzkzmoj2blqwm32v75&sa=D&source=editors&ust=1793837552660444&usg=AOvVaw3RI4HAOP_6byQdL-LwY7Vk>
- Public Proposal: Extensible Decimal Floating-Point Type
<https://www.google.com/url?q=https://docs.google.com/document/d/1cHzlsYH_-JBkcvHLXKSJXMAZ0zwRz3jcnIl4izE1UFQ/edit?tab%3Dt.0%23heading%3Dh.s59uie4uvuwk&sa=D&source=editors&ust=1793837552660695&usg=AOvVaw3JD25VUEnykS82ko8MauZZ>
- IEEE semantics
- Concern on physical representation: wastes a few bits here and there.
We can come up with a more compact representation in the future.
- When you’re forced to use a decfloat because of size, not being the
most efficient is not the main concern anymore.
- Please give another round of review so we can proceed with the next
step.
- Encodings:
- Prateek: requested evaluation.specific frontiers (performance,
compression, …)
- Adding encodings that enable faster decompression and random access.
- ALP (added to spec) is for floating point values
- FSST/OnPair (in progress) is for Strings
- 4 decisions needed for PFoR:
- PFoR (Patched Frame-of-Reference) is for integers (similar to FASTLANES
<https://www.google.com/url?q=https://www.vldb.org/pvldb/vol16/p2132-afroozeh.pdf&sa=D&source=editors&ust=1793837552663081&usg=AOvVaw1jRuoxbfXjMQBPPuYTKqvl>
)
- frame only => slam dunk
- Additionally: adding Delta mode is desirable
- Pluses: pseudo random access to the block, helps with patching
framework with outliers. => better compression.
- Layout questions:
- After analysis we are able to leverage the existing encoding (delta
binary bit packed)
- Interleaving (“Unified Transposed Layout”) or not?
- 80% speedup as long as output fits in L1/L2 caches.
- To not overcomplicate now, option to add it in the future.
- Layout for delta:
- Recommended as a follow up.
- Need more experiments.
- Questions:
- Prateek focused on C++. What are other prototypes?
- RUST: 2x slower with a quick experiment
- Hard to get the autovectorization that fastlanes benefits from.
- Unknowns on how this will work in java.
- Need language experts for this. Please raise you hand. Want to
validate for 3+ language implementations
- Martin Prammer - Rust. Spiral has a Rust crate we’re happy to help get
integrated: https://github.com/spiraldb/fastlanes
- Any delta encoding will benefit from fastlane.
- FSST/OnPair:
- Still open questions.
- Where to put the symbol table (page vs column chunk)
- (Jiayi) Footer update Modular Footer Format
<https://www.google.com/url?q=https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?tab%3Dt.cbl5qtgc5xhz&sa=D&source=editors&ust=1793837552667581&usg=AOvVaw1V_sbBv-OwU_pz0r3LmDWe>
- Metadata stored in parquet pages in the footer. New random access
encodings are a benefit.
- Footer modular: don’t have to decode everything
- Compact footer
- Stats: changed from opaque bytes to their physical types => better
compression
- Relaxing data layout:
- Allow non contiguous column chunks within row groups.
- Asymmetric tables: dense columns => more column chunks, vs sparse
columns => fewer column chunks.
- Compatible with old layout. (row group stats and page stats)
- Stats can be decoupled from row groups and pages.
- Especially since some encodings allow random access.
- We’ll need to have clear guidelines before we open this up.
- To start with, will align this with column chunks and pages.
- Benchmarks:
- Much smaller than the previous proposal.
- 10x smaller, + compression another 5x.
- Decoding speed: very fast.
- Please review => need to start implementation.
- (Julien) Versioning discussion Parquet Versioning proposal
<https://www.google.com/url?q=https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab%3Dt.8veepidgez8a&sa=D&source=editors&ust=1793837552671206&usg=AOvVaw0DViTXsk96WGI1GJF0LmBr>
- Good idea to categorize the changes we made over the past year.
- TODO: Martin: take a first stab documenting previous changes going
through the decision tree. (add tab in the doc)
- Automated release - Prepare Output - Prepare 1.8.2-RC1
<https://www.google.com/url?q=https://github.com/apache/parquet-java/actions/runs/41768120561&sa=D&source=editors&ust=1793837552671884&usg=AOvVaw1gmGEF1IM5CoeDgEMCE8Nf>
Cancel Output - Cancel RC0
<https://www.google.com/url?q=https://github.com/apache/parquet-java/actions/runs/41759590601/job/126343843490&sa=D&source=editors&ust=1793837552672121&usg=AOvVaw1LQm7EQplq4byK-n0wFMVs>
- Ironing out the bugs.
- Do we need a dedicated sync for encodings?
- Prateek to propose an encoding sync on the list.
On Tue, Nov 3, 2026 at 4:31 PM Julien Le Dem <[email protected]> wrote:
> The next Parquet sync is tomorrow Wednesday Nov 4th at 10am PT - 1pm ET -
> 7pm CET
>
> Everybody is welcome, bring your topic or just listen in.
> As usual, we'll build the agenda at the beginning of the meeting.
> On my end, I'm planning to review the change taxonomy
> <https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.8veepidgez8a>
> that
> I have added to the versioning doc following the conversation last time.
> This should lay a solid common foundation to finish the versioning
> discussion.
> (Feel free to respond to this thread if you need to nudge people to review
> something in advance of the call)
>
> To join the invite, join the group:
> https://groups.google.com/g/apache-parquet-community-sync
>
>
>
