I'm supportive of most of the points in this thread. For 2), making encodings pluggable does not eliminate the work on implementation and interoperability. If people are worried about the lengthy process to promote a new encoding to the spec, perhaps we can preserve an encoding type for each new candidate in the spec at its early stage and then officially add or remove it once the idea gets mature.
Best, Gang On Wed, May 29, 2024 at 1:37 AM Micah Kornfield <emkornfi...@gmail.com> wrote: > As a follow-up to the "V3" Discussions [1][2] there were some open > questions around extensibility and how it might be handled, so that readers > could determine if they supported the necessary features. > > I think the areas discussed are: > 1. New encodings (In spec) > 2. Pluggable encodings > 3. Extensible logical types. > 4. New/additional metadata information in footer. > > For 1) these are already handled by existing mechanisms at the column level > (based on page encodings in column metadata). > For 2) the consensus I inferred from PMC members that commented on the doc > is that in general this was not a direction we wanted to take (I also > concur with this sentiment). But if people want to make a more public > argument on why it should be considered we can do it on the ML to make it > official > For 3) Antoine started a new thread on this [3] > For 4) I think any new footer will have a bitmap that will handle changes > and extensibility will likely be limited here. > > If this doesn't cover the use-cases people were thinking of this would be a > good place to bring it up. > > Thanks, > Micah > > > [1] https://lists.apache.org/thread/5jyhzkwyrjk9z52g0b49g31ygnz73gxo > [2] > > https://docs.google.com/document/d/19hQLYcU5_r5nJB7GtnjfODLlSDiNS24GXAtKg9b0_ls/edit > [3] https://lists.apache.org/thread/9xo3mp4n23p8psmrhso5t9q899vxwfjt >
