Notes: https://docs.google.com/document/d/e/2PACX-1vSDHW7gvG8eO6aIxaIVPrZSqYYhtRDb5W1imnbpM4QRYNPsTwEO1fU5z7SEhVIFa4YqWJeSRJ9tcXYS/pub
Aug 12, 2026 | Apache Parquet Community Sync. <https://www.google.com/url?q=https://www.google.com/calendar/event?eid%3DMnJrazhwdDYwMmNuMjZrYWN1NjJwN2lmYTZfMjAyNjA4MTJUMTcwMDAwWiBqdWxpZW4ubGVkZW1AbQ&sa=D&source=editors&ust=1786579593405592&usg=AOvVaw3rlqjTaCtCt4dXH03xDd2D> Attendees: - Neelesh Salian: Apple, Versioning mechanism, listening in for other topics - Julien Le Dem: Datadog, FSST - Andrew Lamb, InfluxData, ALP update - Kenny Daniel, Hyperparam, listening - Gunnar Morling, Confluent, listening in - Lars Volker, listening in - Daniel Tenedorio, Databricks, vector type, listening in - Yan Yan, vector type, listening in - Will Edwards, Spotify, listening in - Talat Uyarer, Google, listening in - Jared Yu, listening in - Connor Tsui, Spiral, listening in - Jigao Luo, a PhD student at TU Darmstadt, listening in - Russell Spitzer, Snowflake, Release Automation, Vector Type, File type? Amendment, ALP, FSST - Ismaël Mejía, Microsoft, Parquet performance update - Arnav Balyan: FSST, Composite Encodings - Ryan Blue, Databricks: new encodings - Rok Mihevc: G-Research/Arctos Alliance, listening in today :) - Prateek Gaur : ALP, Listening in - Kevin Liu, Microsoft, listening in - Kurtis Wright: AWS, listening in - Alkis Evlogimenos, Databricks, vector *logical* type, encodings, File type amendment - Daniel Weeks, Databricks, File type, vector, etc Agenda/Notes: - ALP: status update - Format change approved and merged: https://github.com/apache/parquet-format/pull/557 <https://www.google.com/url?q=https://github.com/apache/parquet-format/pull/557&sa=D&source=editors&ust=1786579593409511&usg=AOvVaw3UFIY4p8xGRNHtUGSLikGA> - Should we mark it in preview? https://github.com/apache/parquet-format/pull/604 <https://www.google.com/url?q=https://github.com/apache/parquet-format/pull/604&sa=D&source=editors&ust=1786579593409896&usg=AOvVaw2bcOOE10o5lXHBRyAp1cLK> - Please review - Proposed parquet-testing data file: https://github.com/apache/parquet-testing/pull/119 <https://www.google.com/url?q=https://github.com/apache/parquet-testing/pull/119&sa=D&source=editors&ust=1786579593410331&usg=AOvVaw2LA159vnAQ1PqYF7sCWtRy> - Please review - Kosta and Andrew are working on a Blog post for parquet site: https://github.com/apache/parquet-site/pull/195 <https://www.google.com/url?q=https://github.com/apache/parquet-site/pull/195&sa=D&source=editors&ust=1786579593410803&usg=AOvVaw3WYh8ielJ0qqoxgxJ3wHvJ> - (not really ready for review) - Prateek will contribute - FSST: - Spec looking good: Parquet FSST Support: Specification <https://www.google.com/url?q=https://docs.google.com/document/d/1Xg2b8HR19QnI3nhtQUDWZJhCLwJzW6y9tU1ziiLFZrM/edit&sa=D&source=editors&ust=1786579593411312&usg=AOvVaw1Yvaz1VimaO3qktXwibwyA> - Mailing list update: https://lists.apache.org/thread/yhv0vp5w1cy08n5n6q2vry98hmw00gnj <https://www.google.com/url?q=https://lists.apache.org/thread/yhv0vp5w1cy08n5n6q2vry98hmw00gnj&sa=D&source=editors&ust=1786579593411644&usg=AOvVaw3LS9fEGyu-PD922ziXHhFz> - Added numbers on GPUs => please review - If no more comments => moving to the implementation phase. - Composite encoding: [Proposal] Introduce Cascaded/Composite Encodings in Parquet <https://www.google.com/url?q=https://docs.google.com/document/u/1/d/1Yi5JwpKEsRFw7D8-iETguRDPtjlyiKITCguYUrrzEVY/edit?tab%3Dt.0%23heading%3Dh.2eyxl5kkyzy7&sa=D&source=editors&ust=1786579593412326&usg=AOvVaw2JuvX1P8AAbainAJZF7_UJ> - Added details and benchmark. - Bumping on the ML for comments. - RLE + Delta. - Ex: Monotonic sequences => constant delta + RLE => high compression. - AKA: composite encodings in literature. - Russel will read the doc and give feedback. - Potentially: FSST encoding dictionaries. - Versioning: - Voting passed: Yeah! - Let’s use ALP to implement this - ALP has a feature flag to turn the feature early. (not on by default) - The writer needs to know what version they are writing. - Need to formalize: TODO - feature flag for - new forward incompatible feature - What version i used to writing - How the file knows what version it is - Behavior of reader based on file version - path_in_schema: will also be using the process. - Existing features not forward compatible: - Existing feature list: https://parquet.apache.org/docs/file-format/versions/ <https://www.google.com/url?q=https://parquet.apache.org/docs/file-format/versions/&sa=D&source=editors&ust=1786579593415076&usg=AOvVaw3rm1gSdfCj3ZndkK-cjylk> - Vector type: - Preparing logical vector type that can annotate LIST or (hopefully) future physical FIXED_SIZE_LIST type - No other progress to share - Current best candidate out of 3: - 1st: Logical type to annotate list (existing variable size encoding) - 2nd: Corresponding physical type for fixed size list - What constraints do we allow to physically represent? - Vector database notion of Vector has additional constraints to a general fixed length array: no NaN, no infinite, no nulls. - Potential mechanisms: different type, parameters in logical type, stats on has-nans etc. - Ryan’s preference: different type specifically for mathematical vector (no-nans, no-infinite, no-nulls) - Rok: propose to make a generic type that supports this use case as well - The constraints might be even more complex depending on the use case (quantization, …) - Action: Rok, Edward, Daniel and Ryan to collect use cases for the type and propose a solution. - File type: - Proposal in the works: https://github.com/apache/parquet-format/pull/603 <https://www.google.com/url?q=https://github.com/apache/parquet-format/pull/603&sa=D&source=editors&ust=1786579593417575&usg=AOvVaw1K9Q3il7pxfSgP-jhrr3VC> - How do we handle encryption and compression? Large value in file instead of inline. - We need feedback on this. - Russell: let’s not make it a special case. - Potential: Limit of size for compression. Many values. But not practically a limit? - Goal: Not causing problems with memory pressure. - Release automation: - Scripts merged, waiting on keys provided by infra for committers to trigger release by pushing a button. - 3 actions: - Prepare (creates RC) - Cancel (if vote doesn’t pass) - Release (if vote passes) - Performance: En/Decoders and Compression optimizations: Progress! - PARQUET <https://www.google.com/url?q=https://github.com/apache/parquet-java/issues/3530&sa=D&source=editors&ust=1786579593419309&usg=AOvVaw1nrPmz4UCcdSkkN0prPhqt> 3/7 Merged. Need review for dictionary and (de)compression. - SPARK-56897 <https://www.google.com/url?q=https://issues.apache.org/jira/browse/SPARK-56897&sa=D&source=editors&ust=1786579593419669&usg=AOvVaw2X9Q6jwl4If2leRkLJbDLk> 6/8 Merged. Pending DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY - VELOX Writes <https://www.google.com/url?q=https://github.com/facebookincubator/velox/issues/17988&sa=D&source=editors&ust=1786579593419950&usg=AOvVaw1pZy3rDxPtpDCqJ0nQjWpL> 3/3 Completed. - VELOX Reads <https://www.google.com/url?q=https://github.com/facebookincubator/velox/issues/17994&sa=D&source=editors&ust=1786579593420149&usg=AOvVaw2faJrEOhYkVQ0wGjw-MVBA> 4/7 Merged. Pending skip and bulk paths for booleans and int64 - What’s next? ALP for Java On Tue, Aug 11, 2026 at 2:19 PM Julien Le Dem <[email protected]> wrote: > The next Parquet sync is tomorrow Wednesday Aug 12th at 10am PT - 1pm ET > - 7pm CET > > To join the invite, join the group: > https://groups.google.com/g/apache-parquet-community-sync > > Everybody is welcome, bring your topic or just listen in. >
