A brief update here: The current implementation status page[1] is looking quite nice. I think the amount of green on that page illustrates visually that despite rumors to the contrary the state of Parquet compatibility is actually good.
Thanks to everyone who has contributed status for their implementations [2][3][4] and more! Andrew [1]: https://parquet.apache.org/docs/file-format/implementationstatus/ [2]: https://github.com/apache/parquet-site/pull/104 [3]: https://github.com/apache/parquet-site/pull/101 [4]: https://github.com/apache/parquet-site/pull/100 On Fri, Feb 7, 2025 at 8:57 AM Andrew Lamb <andrewlam...@gmail.com> wrote: > Thank you!. Once that gets in I will find time (or really try and find > someone else to find time) to fill it out for parquet-rs > > On Thu, Feb 6, 2025 at 12:04 AM Gang Wu <ust...@gmail.com> wrote: > >> Just a heads-up that I've created a PR [1] to update the matrix >> for parquet-cpp and parquet-java. Let me know if you have any >> suggestion. Thanks! >> >> [1] https://github.com/apache/parquet-site/pull/100 >> >> >> On Tue, Feb 4, 2025 at 10:11 PM Andrew Lamb <andrewlam...@gmail.com> >> wrote: >> >> > I wanted to highlight that there is a PR[1] to fill out this matrix for >> > cuda . Super exciting to see this starting >> > >> > [1] https://github.com/apache/parquet-site/pull/99# >> > >> > On Thu, Jul 4, 2024 at 2:57 PM Andrew Lamb <andrewlam...@gmail.com> >> wrote: >> > >> > > Thank you! A small step perhaps but a major step forward >> > > >> > > On Thu, Jul 4, 2024 at 1:03 PM Micah Kornfield <emkornfi...@gmail.com >> > >> > > wrote: >> > > >> > >> I just merged it. >> > >> >> > >> Cheers, >> > >> Micah >> > >> >> > >> On Thu, Jul 4, 2024 at 2:32 AM Andrew Lamb <andrewlam...@gmail.com> >> > >> wrote: >> > >> >> > >> > The initial feature matrix PR[1], has all outstanding comments >> > addressed >> > >> > and is ready to merge from my opinion. >> > >> > >> > >> > Could one of the committers consider merging it so we can start >> > filling >> > >> the >> > >> > matrix out? >> > >> > >> > >> > Thanks, >> > >> > Andrew >> > >> > >> > >> > [1]: https://github.com/apache/parquet-site/pull/34 >> > >> > >> > >> > >> > >> > On Wed, Jun 26, 2024 at 11:22 AM Andrew Lamb < >> andrewlam...@gmail.com> >> > >> > wrote: >> > >> > >> > >> > > I filed a ticket [1] to discuss integration testing / feature >> > >> > > compatibility testing >> > >> > > >> > >> > > [1]: https://github.com/apache/parquet-format/issues/441 >> > >> > > >> > >> > > On Wed, Jun 26, 2024 at 10:03 AM Andrew Lamb < >> > andrewlam...@gmail.com> >> > >> > > wrote: >> > >> > > >> > >> > >> I will file a ticket for discussion shortly >> > >> > >> >> > >> > >> On Wed, Jun 26, 2024, 06:55 Julien Le Dem <jul...@apache.org> >> > wrote: >> > >> > >> >> > >> > >>> I agree with Antoine for the separate thread/ticket for >> > integration >> > >> > >>> testing >> > >> > >>> >> > >> > >>> For the « implementation status » PR, I agree we can merge and >> > >> > iterate. I >> > >> > >>> added a couple of comments that can easily be addressed >> > afterwards. >> > >> > >>> Julien >> > >> > >>> >> > >> > >>> On Wed, Jun 26, 2024 at 10:43 Antoine Pitrou < >> anto...@python.org> >> > >> > wrote: >> > >> > >>> >> > >> > >>> > >> > >> > >>> > IMHO, we should either start a dedicated discussion thread >> for >> > >> > >>> > integration testing, or open a GH issue and discuss it there. >> > >> > >>> > >> > >> > >>> > Regards >> > >> > >>> > >> > >> > >>> > Antoine. >> > >> > >>> > >> > >> > >>> > >> > >> > >>> > >> > >> > >>> > On Wed, 26 Jun 2024 09:21:33 +0200 >> > >> > >>> > Alkis Evlogimenos >> > >> > >>> > <alkis.evlogime...@databricks.com.INVALID> >> > >> > >>> > wrote: >> > >> > >>> > > It would be nice if the integration suite specifies how a >> > driver >> > >> > can >> > >> > >>> be >> > >> > >>> > > executed. Then each implementation can provide a driver and >> > the >> > >> > suite >> > >> > >>> > will >> > >> > >>> > > use that for validation. >> > >> > >>> > > >> > >> > >>> > > By specifying both reads and writes for the driver we get a >> > lot >> > >> > more >> > >> > >>> > power. >> > >> > >>> > > Given an input we can roundtrip all combination of >> > >> readers/writers >> > >> > >>> and >> > >> > >>> > > verify they can roundtrip. >> > >> > >>> > > >> > >> > >>> > > On Tue, Jun 25, 2024 at 6:42 PM Andrew Lamb < >> > >> > >>> > andrewlamb11-re5jqeeqqe8avxtiumw...@public.gmane.org> wrote: >> > >> > >>> > > >> > >> > >>> > > > FWIW I started hacking up a prototype[1] of what a >> > >> > parquet-testing >> > >> > >>> > > > integration suite might look like if anyone is interested >> > >> > >>> > > > >> > >> > >>> > > > >> > >> > >>> > > > >> > >> > >>> > > > [1]: https://github.com/apache/arrow-rs/pull/5956 >> > >> > >>> > > > >> > >> > >>> > > > On Tue, Jun 18, 2024 at 10:39 AM Alkis Evlogimenos >> > >> > >>> > > > <alkis.evlogime...@databricks.com.invalid> wrote: >> > >> > >>> > > > >> > >> > >>> > > > > +1. >> > >> > >>> > > > > >> > >> > >>> > > > > I would suggest you address the comments first? I went >> > >> through >> > >> > >>> the >> > >> > >>> > open >> > >> > >>> > > > > ones and most of them make sense to me (and left few >> > >> additional >> > >> > >>> > > > comments). >> > >> > >>> > > > > >> > >> > >>> > > > > On Tue, Jun 18, 2024 at 12:42 PM Andrew Lamb < >> > >> > >>> andrewlam...@gmail.com >> > >> > >>> > > >> > >> > >>> > > > > wrote: >> > >> > >>> > > > > >> > >> > >>> > > > > > Thank you >> > >> > >>> > > > > > >> > >> > >>> > > > > > On Mon, Jun 17, 2024 at 11:40 PM Micah Kornfield < >> > >> > >>> > > > emkornfi...@gmail.com> >> > >> > >>> > > > > > wrote: >> > >> > >>> > > > > > >> > >> > >>> > > > > > > Hi Andrew, >> > >> > >>> > > > > > > I agree with this sentiment, I asked on the PR if >> > there >> > >> > would >> > >> > >>> > be >> > >> > >>> > > > > another >> > >> > >>> > > > > > > pass and then I can merge it. >> > >> > >>> > > > > > > >> > >> > >>> > > > > > > Cheers, >> > >> > >>> > > > > > > Micah >> > >> > >>> > > > > > > >> > >> > >>> > > > > > > On Fri, Jun 14, 2024 at 3:20 AM Andrew Lamb < >> > >> > >>> > andrewlam...@gmail.com> >> > >> > >>> > > > > > > wrote: >> > >> > >>> > > > > > > >> > >> > >>> > > > > > > > Hello Parquet Devs, >> > >> > >>> > > > > > > > >> > >> > >>> > > > > > > > I propose we merge the first (admittedly bare >> bones) >> > >> > >>> > > > "Implementation >> > >> > >>> > > > > > > > Status" page PR [1] to the website soon. I think >> > this >> > >> > page >> > >> > >>> is >> > >> > >>> > vital >> > >> > >>> > > > > to >> > >> > >>> > > > > > > the >> > >> > >>> > > > > > > > Parquet community (and to any attempt to extend >> the >> > >> > >>> format) so >> > >> > >>> > the >> > >> > >>> > > > > > sooner >> > >> > >>> > > > > > > > the better. >> > >> > >>> > > > > > > > >> > >> > >>> > > > > > > > The reason to merge the PR now is to have a base >> > from >> > >> > which >> > >> > >>> > to >> > >> > >>> > > > build. >> > >> > >>> > > > > > > That >> > >> > >>> > > > > > > > PR is already over a year old and has so many >> > >> comments it >> > >> > >>> is >> > >> > >>> > hard >> > >> > >>> > > > to >> > >> > >>> > > > > > > follow >> > >> > >>> > > > > > > > or know what the path forward is. If we insist on >> > >> sorting >> > >> > >>> all >> > >> > >>> > the >> > >> > >>> > > > > > details >> > >> > >>> > > > > > > > out before we merge it I fear it will never >> merge. >> > >> > >>> > > > > > > > >> > >> > >>> > > > > > > > Once we have a page, I think the next steps are >> to >> > >> add a >> > >> > >>> > preamble >> > >> > >>> > > > > > > > explaining what it is for and to start trying to >> > fill >> > >> out >> > >> > >>> the >> > >> > >>> > chart >> > >> > >>> > > > > for >> > >> > >>> > > > > > > an >> > >> > >>> > > > > > > > implementation (I am happy to try for >> parquet-rs). I >> > >> > >>> suspect >> > >> > >>> > during >> > >> > >>> > > > > > that >> > >> > >>> > > > > > > > process we will have to adjust some of the charts >> > >> more. >> > >> > >>> > > > > > > > >> > >> > >>> > > > > > > > Thank you for your consideration (and thank you >> for >> > >> all >> > >> > the >> > >> > >>> > > > comments >> > >> > >>> > > > > so >> > >> > >>> > > > > > > > far) >> > >> > >>> > > > > > > > >> > >> > >>> > > > > > > > Andrew >> > >> > >>> > > > > > > > >> > >> > >>> > > > > > > > [1]: >> https://github.com/apache/parquet-site/pull/34 >> > >> > >>> > > > > > > > >> > >> > >>> > > > > > > >> > >> > >>> > > > > > >> > >> > >>> > > > > >> > >> > >>> > > > >> > >> > >>> > > >> > >> > >>> > >> > >> > >>> > >> > >> > >>> > >> > >> > >>> > >> > >> > >>> >> > >> > >> >> > >> > >> > >> >> > > >> > >> >
