Hi Alkis, Thanks for the revision. I'm OK with this as is, we can maybe wait a few more days to see if anybody else has comments and then discuss implementation of the extension mechanism?
Cheers, Micah On Thu, Jul 18, 2024 at 10:22 PM Alkis Evlogimenos <alkis.evlogime...@databricks.com.invalid> wrote: > After Jul 17th's Parquet Sync feedback I have updated the extensions > proposal to remove the "reservation" mechanism. The updates are already > reflected in the document > < > https://docs.google.com/document/d/1KkoR0DjzYnLQXO-d0oRBv2k157IZU0_injqd4eV4WiI/edit > > > and > the PR <https://github.com/apache/parquet-format/pull/254>. > > On Fri, Jun 28, 2024 at 10:02 AM Alkis Evlogimenos < > alkis.evlogime...@databricks.com> wrote: > > > > I think we can at least have wording to encourage people doing > > extensions to post them publicly and as part of the "reservation" > mechanism > > post a link the repo that they are being developed in, if anyone is > curious. > > > > Good point. I will try to come up with something in the PR - unless you > > beat me to it :) > > > > On Fri, Jun 28, 2024 at 7:15 AM Micah Kornfield <emkornfi...@gmail.com> > > wrote: > > > >> > > >> > 1. experimentation/prototyping is more often than not faster to > iterate > >> if > >> > it is closed. Allowing this model of development was a primary goal of > >> the > >> > design. > >> > >> > >> I agree there are advantages here. I think a large amount of speed > comes > >> from not having to gain consensus in the community. > >> > >> At the end of the day, I don't think there is any mechanism here to > ensure > >> everybody works in public, but I think we can at least have wording to > >> encourage people doing extensions to post them publicly and as part of > the > >> "reservation" mechanism post a link the repo that they are being > developed > >> in, if anyone is curious. I think this would be particularly useful if > >> there really is an intent for a number of organizations to experiment > with > >> new footer designs (but possibly also in others). > >> > >> Thanks, > >> Micah > >> > >> > >> > >> > >> On Wed, Jun 26, 2024 at 9:33 AM Alkis Evlogimenos > >> <alkis.evlogime...@databricks.com.invalid> wrote: > >> > >> > Thank you for taking a look Micah. > >> > > >> > On the topic of openness there are various aspects that we have > >> considered. > >> > 1. experimentation/prototyping is more often than not faster to > iterate > >> if > >> > it is closed. Allowing this model of development was a primary goal of > >> the > >> > design. > >> > 2. when the design is final, keeping the design closed should have > some > >> > drawbacks. Duplicating content to support old readers puts some > natural > >> > incentive to make extensions official because at that point one can > drop > >> > the fat from the files and move on. Another aspect of the design is > the > >> > choice of a single extension field-id which makes the extension space > >> tiny. > >> > This in turn means that it is difficult to interop with others without > >> > breaking their extensions. Ergo the easiest path to any interop is to > >> open > >> > the extension. > >> > > >> > The above, while not enforcing work to happen in the open, strike some > >> > balance in between. > >> > > >> > I am open to suggestions on how to further incentivize opening > >> extensions. > >> > > >> > On Wed, Jun 26, 2024 at 6:04 PM Micah Kornfield < > emkornfi...@gmail.com> > >> > wrote: > >> > > >> > > Hi Alkis, > >> > > I'm generally in favor of this, my main concern/question is trying > to > >> > > encourage work to be in the open. I don't think in the long run it > is > >> > good > >> > > for users to always have proprietary extensions inside of Parquet. > >> > > > >> > > IMO, I think the next steps would be to add implementations to write > >> out > >> > > the footer extension points. > >> > > > >> > > Thanks, > >> > > Micah > >> > > > >> > > On Mon, Jun 24, 2024 at 1:24 PM Alkis Evlogimenos > >> > > <alkis.evlogime...@databricks.com.invalid> wrote: > >> > > > >> > > > The snafus are fixed. The original should work now. > >> > > > > >> > > > On Sun, 23 Jun 2024, 17:58 Alkis Evlogimenos, < > >> > > > alkis.evlogime...@databricks.com> wrote: > >> > > > > >> > > > > Due to some sharing snafus with automation, please request > access > >> to > >> > > > > comment. If you are just reading I've published this here: > >> > > > > > >> > > > > >> > > > >> > > >> > https://docs.google.com/document/d/e/2PACX-1vThXkhHNozn_p1ZZWF-nCzOtoP1lKmkaV4Legq2FaRiIgwyY2XC9AmKpBtpeF8jbBB4wfjmQ6UTg03k/pub > >> > > > > > >> > > > > On Fri, Jun 21, 2024 at 10:29 AM Alkis Evlogimenos < > >> > > > > alkis.evlogime...@databricks.com> wrote: > >> > > > > > >> > > > >> Hey folks. > >> > > > >> > >> > > > >> I want to move the extension PR > >> > > > >> <https://github.com/apache/parquet-format/pull/254> forward. > >> > > > >> Unfortunately the discussion was spread across the PR, other > >> threads > >> > > and > >> > > > >> documents making it slow to progress. To avoid further > >> > fragmentation I > >> > > > have > >> > > > >> put together a document > >> > > > >> < > >> > > > > >> > > > >> > > >> > https://docs.google.com/document/d/1KkoR0DjzYnLQXO-d0oRBv2k157IZU0_injqd4eV4WiI/edit > >> > > > > > >> > > > >> discussing the extensions mechanism in isolation. I believe the > >> > > document > >> > > > >> addresses all the concerns/comments from the PR and mailing > list > >> > > > >> discussions brought forward so far. > >> > > > >> > >> > > > >> I propose we continue the discussion in the document and once > >> > > everything > >> > > > >> is addressed, we finalize the PR. > >> > > > >> > >> > > > >> Thank you, > >> > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > > >
