See notes here:
https://docs.google.com/document/d/e/2PACX-1vSDHW7gvG8eO6aIxaIVPrZSqYYhtRDb5W1imnbpM4QRYNPsTwEO1fU5z7SEhVIFa4YqWJeSRJ9tcXYS/pub
I listed in *bold *the action items and requests for reviews. Attendees:
-
Julien Le Dem: Datadog. Versioning discussion
-
Neelesh Salian: Apple, Versioning discussion
-
Div Arora: Databricks, Versioning, adopting new sort orders
-
Jiayi Wang: Databricks, modular footer
-
Alkis Evlogimenos: Databricks, FILE amend, vector, encodings, footer,
sort orders, timestamp-nanos
-
Daniel Weeks: Databricks, all the things
-
Benjamin Owad : Snowflake, listening in
-
Prateek Gaur : Snowflake
-
Kosta Tarasov: listening in
-
Connor Tsui: Spiral, listening in
-
Lukas Nalezenec, listening in
-
Rok Mihevc: G-Research/Arctos Alliance, vector type proposal, new footer
-
Kevin Liu: Microsoft, backport variant support + versioning!!
-
Philipp Fischbeck: Snowflake, vector type
-
Jigao Luo: TU Darmstadt PhD, external contributor to cuDF, Microsoft
Intern, listening in
-
Daniel Tenedorio: Databricks, FILE type, vector type, also listening in
-
Yan Yan: Databricks, listening in
-
Arnav Balyan: Fsst/composite encodings.
-
Russell Spitzer: Snowflake, Vectors, Encodings, Versions, Automated
Releases
-
Shawn Chang: AWS, listening in
Agenda/Notes:
-
Versioning discussion
-
Parquet Versioning proposal
<https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.6vhlejvz7fj#heading=h.8v8711vu0p1i>
-
Parquet versioning voting thread
<https://docs.google.com/document/d/1OXEtJ6f1g_8PQGhbhHwoEsrh3XCex8-Ftb44798EqwY/edit?tab=t.0>
-
Another 24h to give feedback on this and then I’ll send a voting
thread
-
Encodings
-
ALP
-
We started and closed the vote last week.
-
People left some comments on the spec doc, addressed most of them.
Few remain which I’ll try to wrap up today.
-
Don’t think the c++ PR needs any updates based on the spec updates
so far (minus one feedback). Will send back the PR for review
by the end of
the week.
-
Vinoo and Arnav for Java and Golang PR’s.
-
PFOR
-
[External] Parquet PFOR : Numeric Compression
<https://docs.google.com/document/d/1ZZOtxmq6K8pNU0npijfSglTJVkspXL5GLDKPSGj9HlA/edit?tab=t.0>
-
Antoine requested a few more data points/encoding compression
algorithms to the sheet. Which I have done.
-
Does anyone have more feedback?
-
FastLanes
-
Early numbers don’t show promise.
-
Alkis hinted towards this.
-
If people missed it Antoine made a big improvement to bitunpack
decoding speed (Feb 2026). In the presence of that, the
interleaved lane
layout doesn’t look like a direction to pursue.
-
Kosta is also experimenting with it. We’ll try to sync and get
back on this.
-
[TODO] link to branch
-
FSST:
-
Spec is complete, all open comments are addressed. Parquet FSST
Support: Specification
<https://docs.google.com/document/d/1Xg2b8HR19QnI3nhtQUDWZJhCLwJzW6y9tU1ziiLFZrM/edit?tab=t.0>
-
Discussion on training data being an implementation detail was
incorporated in spec
-
Requesting additional reviews
-
Composite encodings
-
[Proposal] Introduce Cascaded/Composite Encodings in Parquet
<https://docs.google.com/document/d/1Yi5JwpKEsRFw7D8-iETguRDPtjlyiKITCguYUrrzEVY/edit?tab=t.0#heading=h.2eyxl5kkyzy7>
-
Review requested
-
Adopting new sort orders: is emitting a new sort order considered a
breaking change for a writer / need to wait until the next major version
release? Parquet-java reference PR
<https://github.com/apache/parquet-java/pull/3393#issuecomment-5105403373>
-
Discussed: New sorting order and NaN count. (used for pruning)
-
Parquet java does NaN count but not new sort order.
-
Only forward compatibles if using the new sort order with NaN counts
-
Before: was not outputting stats in that case.
-
Using a new enum value will create error in old reader. (discussion
that this is no longer the case)
-
Action: need to resolve the discrepancy on format saying it’s forward
compatible and implementation wanting to the next major release.
-
Conclusion: revert the non-forward compatible change not yet
released (INT96 sort order)
-
Unclear what to do for IEEE sort order as it landed in
parquet-format v2.13.0
-
Action item: Divjot to start a thread to resolve this.
-
New types:
-
FILE amendment:
https://lists.apache.org/thread/zrzc7t9fccg92rx3h4fw3ndw3bdo5xr7
-
TLDR make self-reference blobs inherit compression codec from
inline column
-
Discussion of per entry flag vs per page flag on whether we
compress.
-
Action: finalize discussion on this. Alkis to reply on the thread
with summary of the two options and gauge other opinions on this.
-
Who’s the consumer and how is it used?
-
When do we decide on encryption?
-
Prateek: Agree with Alkis here. That the mechanism is not
infinitely scalable and that the writer will have to be smart
to switch to
new page if it wants to use a new compression/encoding
scheme. And it might
be fine because you are self referencing it so likely the
value is large
and it is okay to have only a few values per page. Though i do feel
Russell has a point, that `in future` we may see use cases
where each value
in the blob might need a different compression scheme.
-
Vector
-
Recommended LIST reader/writer optimizations (ML discussion state
https://lists.apache.org/thread/xot5f3ghhtc82n1bf0wdl9zqwlrzqks3)
-
LIST optimization favoured for short term availability and
compatibility (
https://www.morling.dev/blog/fast-path-for-fixed-length-lists-in-parquet/
)
-
Longer term solution - new repetition level proposal (ML thread
https://lists.apache.org/thread/04s1w3qwoqv3ww9bx982fxg7kyw5lmf2)
-
Short term: Optimize reader for fixed sized list.
-
Medium term: logical type.
-
Open question of other fixed sized.
-
OQ: Floats: nulls and ints?
-
Similar discussion on the iceberg sync.
-
Not only ML vector, also scientific use cases.
-
Do we want separate types for Vector store vs other uses? Can
have a more narrow definition for specific vector store use cases.
-
Action: Need to follow up on the mailing list,
-
Rok to drive the discussion.
-
timesta mp-nanos: parquet-format PR
<https://github.com/apache/parquet-format/pull/601>, parquet-java PR
<https://github.com/apache/parquet-java/pull/3680> please review
-
Modular footer proposal is out, Modular parquet footer
<https://docs.google.com/document/d/15J-MVkwFTfq3vUipvhrf69YJVDlfD6siR3yUmo3V-E0/edit?usp=sharing>,
please review and comment
-
TL;DR: Explains why we need a breaking change and drop the FB option
-
Next steps listed, please review.
-
Logical type forward compatibility (Kevin)
-
Should we backport variant support in 1.15.x?
https://github.com/apache/parquet-java/issues/3633 please review
-
Parquet-java LICENSE/NOTICE
-
https://github.com/apache/parquet-java/pull/3694 please review
-
Automated releases
-
Please look at the PRs. [TODO link to the PRs.]
