Please find notes from our sync today below
Sep 9, 2026
Attendees:
-
Andrew Lamb, InfluxData, ALP Update, versioning
-
Daniel Weeks, Databricks, Vectors, Footers, etc.
-
Lukas Nalezenec,, listening in
-
Kevin Liu, Microsoft, listening in
-
Alkis Evlogimenos, Databricks, versioning, vector, footer
-
Jiayi Wang, Databricks, footer
-
Philipp Fischbeck: Snowflake, vectors
-
Vinoo Ganesh: Kepler, Java ALP Impl
-
Kosta Tarasov, listening in
-
Russell Spitzer: Snowflake, Versioning, vectorTypes, ALP, FSST, PFOR
-
Ryan Blue: Databricks, versioning, vector types, encodings
-
Neelesh Salian: Apple, versioning, listening in for other topics
-
Jigao Luo, a PhD student at TU Darmstadt, listening in
-
Div Arora: Databricks, vector
-
Connor Tsui, Spiral: listening in
-
Rok Mihevc: G-Research/Arctos Alliance, vectors, footers
-
William Hyun, listening in
-
Will Edwards, listening in
-
Arnav Balyan, listening in
-
Fokko Driesprong, Databricks; versioning
-
Prateek Gaur, Encodings, footer,
-
Benjamin Owad, Snowflake: listening in
-
Kurtis Wright: AWS, Types, Footer
Agenda/Notes:
-
ALP Update:
-
Java implementation is being reviewed, waiting to get merged.
-
C++ implementation: a few rounds of review, waiting on approval to
get it merged
-
Golang implementation
-
Merged Rust implementation:
https://github.com/apache/arrow-rs/pull/9372
-
Parquet-version vote it outstanding:
https://lists.apache.org/thread/ovo7vhh2gp31v32xzq6wm5f0grk97bs3
-
Blog is ready to publish once parquet-version is merged. Reviews
appreciated: https://github.com/apache/parquet-site/pull/195
-
Java implementation -
https://github.com/apache/parquet-java/pull/3397
-
Vectors
-
We had a focused call on Vector logical type, notes
<https://docs.google.com/document/d/1rPTjEJTwN97UfHvNlvdriX9XBQsW8PGwIHMXHvy-XlU/edit?tab=t.0>
and have a followup ML thread
<https://lists.apache.org/thread/kgmgn2vmrz49thbfhv8b66lglsyd81xt>
-
Open questions are roughly
-
Where do we put finite-element information
-
What to do about vectors that legitimately contain nulls, nans and
infinities
-
We vaguely agreed on:
-
Finite-element requirement is a MUST. Stats can help readers when
processing untrusted sources, but we don’t depend on them.
-
LIST seems like a more popular option, didn’t get to consensus
point
-
-
Footers
-
Working on more benchmarks, and integration in one open sourced
parquet reader
-
Size benchmark on public datasets (result
<https://github.com/Jiayi-Wang-db/parquet-footer-bench/blob/main/real-footer-size/README.md>
)
-
Addresses feedback on previous proposals
-
Benchmarks on public data
-
Benchmarks with integrating with a public parquet reader
-
Will publish a proposal to the mailing list once it is refined
more – join the sync if interested.
-
FSST Update:
-
Initial implementations of Rust, Java (thanks Prateek!), Arrow ready
-
Initial proposal has 8-bit and 16 bit codes (similar to OnPair)
-
Waiting on some final comments on the FSST Spec: Parquet FSST
Support: Specification
<https://docs.google.com/document/d/1Xg2b8HR19QnI3nhtQUDWZJhCLwJzW6y9tU1ziiLFZrM/edit?tab=t.0#heading=h.a9r0tnd6fhtq>
-
Looking for feedback on the spec, and once addressed will
-
Preparing to start vote on the mailing list.
-
FYI: vote thread
<https://lists.apache.org/thread/v11pw7sbnkdxzjhkmcbsx06clyvjv0xx> for
FLBA(12) timestamps
-
Hoping to close the vote this week
-
Git Action Based Release Jobs
-
Russel has been working through it for parquet-java, has right
secrets from ASF infra
-
To prepare jars, etc you would just trigger a job as a committer
-
Run a second job that would actually close the vote and publish the
artifacts
-
PFOR
-
Looking for feedback on the doc showcasing performance numbers
-
[External] Parquet PFOR : Numeric Compression
<https://docs.google.com/document/d/1ZZOtxmq6K8pNU0npijfSglTJVkspXL5GLDKPSGj9HlA>
-
Community ask for comparison with Fastlanes
-
Trying to publish the numbers for that next. Kosta expressed
interest so will work with him for this.
-
Considering a delta mode
-
Looking into effectiveness for dictionary index coding
-
To try and setup a meeting with the fast lanes authors to discuss
(Andrew will try and help organize a meeting)
-
Versioning
-
Mailing list discussion thread:
https://lists.apache.org/thread/bwtprwvc4jgv5d4tmfzh5c9qrv02qfyn
-
Assumption about writing a version into a file was not universally
held
-
Was pushback on writing versions into files
-
Differing ideas on what reader behavior should be on encountering
unknown versions
-
Also, discussion on list vs in sync
-
Discussion: should we allow / not allow the version used to write a
file to control the reader behavior.
-
Dan Weeks will summarize the discussion and send to the mailing list
