After Jul 17th's Parquet Sync feedback I have updated the extensions proposal to remove the "reservation" mechanism. The updates are already reflected in the document <https://docs.google.com/document/d/1KkoR0DjzYnLQXO-d0oRBv2k157IZU0_injqd4eV4WiI/edit> and the PR <https://github.com/apache/parquet-format/pull/254>.
On Fri, Jun 28, 2024 at 10:02 AM Alkis Evlogimenos < alkis.evlogime...@databricks.com> wrote: > > I think we can at least have wording to encourage people doing > extensions to post them publicly and as part of the "reservation" mechanism > post a link the repo that they are being developed in, if anyone is curious. > > Good point. I will try to come up with something in the PR - unless you > beat me to it :) > > On Fri, Jun 28, 2024 at 7:15 AM Micah Kornfield <emkornfi...@gmail.com> > wrote: > >> > >> > 1. experimentation/prototyping is more often than not faster to iterate >> if >> > it is closed. Allowing this model of development was a primary goal of >> the >> > design. >> >> >> I agree there are advantages here. I think a large amount of speed comes >> from not having to gain consensus in the community. >> >> At the end of the day, I don't think there is any mechanism here to ensure >> everybody works in public, but I think we can at least have wording to >> encourage people doing extensions to post them publicly and as part of the >> "reservation" mechanism post a link the repo that they are being developed >> in, if anyone is curious. I think this would be particularly useful if >> there really is an intent for a number of organizations to experiment with >> new footer designs (but possibly also in others). >> >> Thanks, >> Micah >> >> >> >> >> On Wed, Jun 26, 2024 at 9:33 AM Alkis Evlogimenos >> <alkis.evlogime...@databricks.com.invalid> wrote: >> >> > Thank you for taking a look Micah. >> > >> > On the topic of openness there are various aspects that we have >> considered. >> > 1. experimentation/prototyping is more often than not faster to iterate >> if >> > it is closed. Allowing this model of development was a primary goal of >> the >> > design. >> > 2. when the design is final, keeping the design closed should have some >> > drawbacks. Duplicating content to support old readers puts some natural >> > incentive to make extensions official because at that point one can drop >> > the fat from the files and move on. Another aspect of the design is the >> > choice of a single extension field-id which makes the extension space >> tiny. >> > This in turn means that it is difficult to interop with others without >> > breaking their extensions. Ergo the easiest path to any interop is to >> open >> > the extension. >> > >> > The above, while not enforcing work to happen in the open, strike some >> > balance in between. >> > >> > I am open to suggestions on how to further incentivize opening >> extensions. >> > >> > On Wed, Jun 26, 2024 at 6:04 PM Micah Kornfield <emkornfi...@gmail.com> >> > wrote: >> > >> > > Hi Alkis, >> > > I'm generally in favor of this, my main concern/question is trying to >> > > encourage work to be in the open. I don't think in the long run it is >> > good >> > > for users to always have proprietary extensions inside of Parquet. >> > > >> > > IMO, I think the next steps would be to add implementations to write >> out >> > > the footer extension points. >> > > >> > > Thanks, >> > > Micah >> > > >> > > On Mon, Jun 24, 2024 at 1:24 PM Alkis Evlogimenos >> > > <alkis.evlogime...@databricks.com.invalid> wrote: >> > > >> > > > The snafus are fixed. The original should work now. >> > > > >> > > > On Sun, 23 Jun 2024, 17:58 Alkis Evlogimenos, < >> > > > alkis.evlogime...@databricks.com> wrote: >> > > > >> > > > > Due to some sharing snafus with automation, please request access >> to >> > > > > comment. If you are just reading I've published this here: >> > > > > >> > > > >> > > >> > >> https://docs.google.com/document/d/e/2PACX-1vThXkhHNozn_p1ZZWF-nCzOtoP1lKmkaV4Legq2FaRiIgwyY2XC9AmKpBtpeF8jbBB4wfjmQ6UTg03k/pub >> > > > > >> > > > > On Fri, Jun 21, 2024 at 10:29 AM Alkis Evlogimenos < >> > > > > alkis.evlogime...@databricks.com> wrote: >> > > > > >> > > > >> Hey folks. >> > > > >> >> > > > >> I want to move the extension PR >> > > > >> <https://github.com/apache/parquet-format/pull/254> forward. >> > > > >> Unfortunately the discussion was spread across the PR, other >> threads >> > > and >> > > > >> documents making it slow to progress. To avoid further >> > fragmentation I >> > > > have >> > > > >> put together a document >> > > > >> < >> > > > >> > > >> > >> https://docs.google.com/document/d/1KkoR0DjzYnLQXO-d0oRBv2k157IZU0_injqd4eV4WiI/edit >> > > > > >> > > > >> discussing the extensions mechanism in isolation. I believe the >> > > document >> > > > >> addresses all the concerns/comments from the PR and mailing list >> > > > >> discussions brought forward so far. >> > > > >> >> > > > >> I propose we continue the discussion in the document and once >> > > everything >> > > > >> is addressed, we finalize the PR. >> > > > >> >> > > > >> Thank you, >> > > > >> >> > > > > >> > > > >> > > >> > >> >
