Sorry I meant 1.3.0-incubating and 1.4.0-incubating in terms of releases. Le sam. 22 nov. 2025 à 11:27, Jean-Baptiste Onofré <[email protected]> a écrit :
> Hi All, > > I understand Laurent’s points and agree with the sentiment regarding > the API’s future evolution. The current Generic Table API, despite its > limitations (like the Delta and CSV support demonstrates), is > functional. > > As a community, we need to achieve consensus on this topic. Consensus > does not require unanimous agreement on every detail but means the > project has reached a decision or compromise that everyone can accept. > Specifically, using lazy consensus means we can proceed unless there > is a legitimate objection accompanied by an alternative approach for > discussion. > > To move toward consensus and incorporate Laurent's concerns, I propose > the following compromise: > > 1. Keep the "beta" flag for the upcoming 1.2.0-incubating release. > 2. Re-evaluate and aim to remove the "beta" label for the > 1.3.0-incubating release. > > For the 1.3.0-incubating release to proceed without the "beta" label, > I suggest we clearly document the existing limitations of the Generic > Table API and reference the relevant discussions/issues concerning its > evolution (specifically the issues for schema support and credential > vending, as well as the Common Table API proposal). > > Does this compromise seem reasonable to everyone? > > If this approach does not gain approval, we can initiate a formal > vote, though I believe that may be unnecessary here. > > Regards, > JB > > On Fri, Nov 21, 2025 at 10:17 PM yun zou <[email protected]> > wrote: > > > > Hi Laurent, > > > > Thanks a lot for the input! I fully agree that there’s still plenty of > room > > for improvement before this feature reaches a perfect state. Meanwhile, > > since the feature is already attracting interest, I think it would be > > valuable to remove the “beta” label so we can draw wider attention and > > collect more feedback to help it evolve in the right direction. > > > > As JB pointed out, this won’t block the evolution of the API. If we need > to > > introduce a V2 in the future, I think that’s perfectly fine—it would show > > that the Polaris community is committed to continuously improving and > > supporting non-Iceberg standards. > > > > Since we’ve mostly aligned on removing the beta label, how about we > proceed > > with that? > > > > > > Best Regards, > > Yun > > > > On Fri, Nov 21, 2025 at 8:58 AM Laurent Goujon <[email protected]> > wrote: > > > > > Sorry didn't want to be difficult but I was just hoping we could > address > > > some of the possible issuers we flagged when we decided to add the beta > > > status to the api in the first place. I agree there's some great > interest > > > for it (we have been chatting about it also at the south bay meet up > this > > > week) but I don't know about adoption? But I also respect the > willingness > > > to move fast even if I think that keeping a beta flag for a long > period of > > > time is not necessarily a bad thing (maybe in the future we can > separate > > > experimental where things may be really removed or totally reworked > from > > > beta where the core structure is in place and we will try to minimize > > > breaking changes) > > > > > > Ultimately it was just a simple request, if the consensus is to remove > the > > > beta flag, then please ignore my previous email and proceed. > > > > > > Laurent > > > > > > On Fri, Nov 21, 2025 at 6:08 AM Jean-Baptiste Onofré <[email protected]> > > > wrote: > > > > > > > Hi All, > > > > > > > > I think it is important to distinguish between the "beta" status of > > > > the Generic Table API and its future evolution. > > > > > > > > Strictly speaking, I believe we should remove the "beta" label now, > as > > > > the API already has several implementations and is seeing real-world > > > > usage. > > > > > > > > While we all agree that the Generic Table API will continue to evolve > > > > (e.g., credential vending, common table API), I anticipate that these > > > > changes will primarily be "additions" rather than breaking changes to > > > > the existing specification (which is pretty thin honestly). For > > > > example, adding schema support should be backward compatible. > > > > > > > > Personally, I would rather avoid using the "beta" flag here. Features > > > > in open source projects are inherently evolving, often rapidly. > > > > > > > > Just my two cents. > > > > > > > > Regards, > > > > JB > > > > > > > > On Fri, Nov 21, 2025 at 4:46 AM Laurent Goujon <[email protected]> > > > wrote: > > > > > > > > > > If this is okay, I would like to keep the beta a bit longer until > we > > > see > > > > > some actual adoption cross-engines, and we address some of the > points > > > we > > > > > discussed previously like the lack of schema for example so we can > use > > > > the > > > > > feedback to improve on it and avoid having to release a v2 (or > hold off > > > > for > > > > > a long time to refactor). > > > > > > > > > > I'd also like to amend the common table API proposal we discussed > last > > > > > october and hopefully refactor it as an evolution of the existing > > > Generic > > > > > Table API since it focused amongst other things on the metadata > issue. > > > > > > > > > > Please let me know if you are agreeable to this. > > > > > > > > > > Laurent > > > > > > > > > > On Wed, Nov 19, 2025 at 11:03 AM yun zou < > [email protected]> > > > > wrote: > > > > > > > > > > > Hi Dmitri, > > > > > > > > > > > > Thanks! > > > > > > > > > > > > I have put up the PR: > https://github.com/apache/polaris/pull/3096, > > > > please > > > > > > help take a look! > > > > > > > > > > > > Will comment on the thread also! > > > > > > > > > > > > Best Regards, > > > > > > Yun > > > > > > > > > > > > On Wed, Nov 19, 2025 at 7:00 AM Dmitri Bourlatchkov < > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Hi Yun, > > > > > > > > > > > > > > I am personally ok with the approach you propose. If you would > like > > > > to > > > > > > > remove the beta label in 1.3.0, please open a PR to unblock the > > > > release > > > > > > as > > > > > > > discussed in [1]. If you prefer to remove the label later, > please > > > > comment > > > > > > > on [1]. > > > > > > > > > > > > > > [1] > > > https://lists.apache.org/thread/dhzop6tddl8f9ygbbgdoqywk0hwzsgb2 > > > > > > > > > > > > > > Thanks, > > > > > > > Dmitri. > > > > > > > > > > > > > > On Tue, Nov 18, 2025 at 9:47 PM yun zou < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Dmitri, > > > > > > > > > > > > > > > > Thanks for the clarification! > > > > > > > > > > > > > > > > -- if we happen to need a major Generic Tables API change > after > > > > > > removing > > > > > > > > the "beta" label, I believe we'd have to make a v2 of that > API > > > > > > > > > > > > > > > > Yes, if there are changes that require altering the > high-level > > > > > > semantics > > > > > > > of > > > > > > > > the API or modifying existing fields, we would need to move > to a > > > > V2. > > > > > > > > > > > > > > > > However, since the Generic Table API currently defines only > very > > > > basic > > > > > > > > fields, I believe the use cases described in the Table Source > > > > proposal > > > > > > > can > > > > > > > > be addressed by extending the existing APIs. If it eventually > > > > becomes > > > > > > > clear > > > > > > > > that a major change is required—such as a semantic-level > > > > shift—then we > > > > > > > can > > > > > > > > transition to V2. The goal is to evolve and build on the > current > > > > APIs > > > > > > > > wherever possible. > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > Yun > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 18, 2025 at 8:32 AM Dmitri Bourlatchkov < > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Yun, > > > > > > > > > > > > > > > > > > I personally do not think the "beta" label is related to > our > > > > > > commitment > > > > > > > > (or > > > > > > > > > lack of it) in maintaining the API. IMHO, it means that > the API > > > > is > > > > > > > likely > > > > > > > > > to undergo major changes. > > > > > > > > > > > > > > > > > > I did not and do not really oppose removing the "beta" > label > > > > from the > > > > > > > > > Generic Tables API :) > > > > > > > > > > > > > > > > > > My suggestion in keeping it a bit longer was purely to > allow > > > more > > > > > > time > > > > > > > > > for finding common use cases with the Table Sources > proposal > > > and > > > > > > > possibly > > > > > > > > > unifying some workflows. > > > > > > > > > > > > > > > > > > Having thought about that from that perspective some more, > I > > > > actually > > > > > > > > agree > > > > > > > > > that the Generic Tables API is _not_ likely to undergo > major > > > > changes. > > > > > > > If > > > > > > > > > synergies with Table Sources can be found, most of the > changes > > > > are > > > > > > > > probably > > > > > > > > > going to happen in clients that currently use the Generic > > > Tables > > > > API, > > > > > > > the > > > > > > > > > API itself is probably going to remain stable. > > > > > > > > > > > > > > > > > > That said, just for the sake of clarity, if we happen to > need a > > > > major > > > > > > > > > Generic Tables API change after removing the "beta" label, > I > > > > believe > > > > > > > we'd > > > > > > > > > have to make a v2 of that API (which is ok from my POV per > our > > > > > > > evolution > > > > > > > > > guidelines [1]). > > > > > > > > > > > > > > > > > > [1] > https://polaris.apache.org/in-dev/unreleased/evolution/ > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > On Mon, Nov 17, 2025 at 8:20 PM yun zou < > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Dmitri, > > > > > > > > > > > > > > > > > > > > Thanks a lot for the feedback! > > > > > > > > > > > > > > > > > > > > I definitely agree that the current generic table support > > > > still has > > > > > > > > > > limitations and that more work is needed. However, > removing > > > the > > > > > > > “beta” > > > > > > > > > > label doesn’t mean the feature is finalized. Instead, it > > > > signals to > > > > > > > > users > > > > > > > > > > that we are committed to maintaining and improving it > over > > > > time. > > > > > > > > > Therefore, > > > > > > > > > > I believe this will not block any future enhancements > > > including > > > > > > > > extending > > > > > > > > > > it to address the use cases described in the Table Source > > > > proposal. > > > > > > > > > > > > > > > > > > > > As Adam pointed out, we already have users trying it out > and > > > > > > > requesting > > > > > > > > > > improvements. I believe this is a good time to remove the > > > > “beta” > > > > > > > label > > > > > > > > to > > > > > > > > > > encourage broader adoption and welcome new contributors > who > > > can > > > > > > help > > > > > > > > > > advance the feature. > > > > > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > > > Yun > > > > > > > > > > > > > > > > > > > > On Mon, Nov 17, 2025 at 3:28 PM Adam Christian < > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > I'm in favor of removing the "beta" label because it is > > > > starting > > > > > > to > > > > > > > > be > > > > > > > > > > used > > > > > > > > > > > by users. From the Slack feedback we have seen, there > are > > > > users > > > > > > who > > > > > > > > > > > have started to try out this feature. While they are > still > > > > > > running > > > > > > > > into > > > > > > > > > > > issues such as not having a Spark 4 plugin [1] and not > > > having > > > > > > > > > credential > > > > > > > > > > > vending [2], there is real user usage indicating that > folks > > > > find > > > > > > > this > > > > > > > > > > > beneficial. > > > > > > > > > > > > > > > > > > > > > > Now, I do agree with Dmitri that there are known > > > limitations > > > > we > > > > > > > need > > > > > > > > to > > > > > > > > > > > handle that are impeding users. The two referenced > examples > > > > above > > > > > > > are > > > > > > > > > > > obvious ones that need fixes to get wider adoption. I > have > > > > been > > > > > > > > working > > > > > > > > > > > with Yun & others to be able to solve some of these > issues > > > > (like > > > > > > > this > > > > > > > > > PR > > > > > > > > > > > [3]), but I agree that there may need to be more > > > significant > > > > > > > changes > > > > > > > > > > > including a REST API change. > > > > > > > > > > > > > > > > > > > > > > From my understanding, during our last conversation in > the > > > > > > > community > > > > > > > > > > about > > > > > > > > > > > the "Table Sources" proposal [4], we decided we were > going > > > to > > > > > > > analyze > > > > > > > > > the > > > > > > > > > > > Parquet file use case and see if Generic Tables can > support > > > > this > > > > > > > use > > > > > > > > > case > > > > > > > > > > > or whether we need to bring some ideas from the Table > > > Sources > > > > > > > > proposal > > > > > > > > > > into > > > > > > > > > > > Generic Tables. I believe that this approach is still > valid > > > > and > > > > > > > will > > > > > > > > > > > benefit our adopting users by identifying any > enhancements > > > > with > > > > > > > these > > > > > > > > > new > > > > > > > > > > > use cases. > > > > > > > > > > > > > > > > > > > > > > What do y'all think? > > > > > > > > > > > > > > > > > > > > > > [1] - https://github.com/apache/polaris/issues/3021 > > > > > > > > > > > [2] - https://github.com/apache/polaris/issues/3020 > > > > > > > > > > > [3] - https://github.com/apache/polaris/pull/3026 > > > > > > > > > > > [4] - > > > > > > > > > https://lists.apache.org/thread/652z1f1n2pgf3g2ow5y382wlrtnoqth0 > > > > > > > > > > > > > > > > > > > > > > Go community, > > > > > > > > > > > > > > > > > > > > > > Adam > > > > > > > > > > > > > > > > > > > > > > On Mon, Nov 17, 2025 at 12:23 PM Dmitri Bourlatchkov < > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi Yun, > > > > > > > > > > > > > > > > > > > > > > > > We should indeed review the status of the Generic > Tables > > > > API, > > > > > > so > > > > > > > > > thanks > > > > > > > > > > > for > > > > > > > > > > > > starting this discussion! > > > > > > > > > > > > > > > > > > > > > > > > From my POV the key question is: how do we intend to > > > > proceed > > > > > > with > > > > > > > > > known > > > > > > > > > > > > limitations [1]? > > > > > > > > > > > > > > > > > > > > > > > > I believe the Table Sources proposal [2] addresses > some > > > > (if not > > > > > > > > all) > > > > > > > > > of > > > > > > > > > > > > those limitations. It is certainly suitable for the > same > > > > > > > > applications > > > > > > > > > > > that > > > > > > > > > > > > currently go through the Generic Tables API. > > > > > > > > > > > > > > > > > > > > > > > > I believe it would be wise to allow the Table Sources > > > > proposal > > > > > > to > > > > > > > > > > develop > > > > > > > > > > > > further to see if there are any synergies that can be > > > > leveraged > > > > > > > > with > > > > > > > > > > > > respect to Generic Tables. If we removed the "beta" > label > > > > from > > > > > > > the > > > > > > > > > > > Generic > > > > > > > > > > > > Tables API now, it would make it harder for users to > > > > benefit > > > > > > from > > > > > > > > > those > > > > > > > > > > > > synergies later (due to virtually freezing the > "spec"). > > > > > > > > > > > > > > > > > > > > > > > > At it stands now, the Generic Tables API is > supported by > > > > > > Polaris, > > > > > > > > so > > > > > > > > > > > > existing clients can continue operating normally. > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/polaris/blob/f056e22f7f3a7c53e233bef1b88d204d6a8e4d79/site/content/in-dev/unreleased/generic-table.md?plain=1#L162-L169 > > > > > > > > > > > > > > > > > > > > > > > > [2] > > > > > > > > > https://lists.apache.org/thread/9f5b75fy25l9yzrtnlzqg6yh1bqdyjbt > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Nov 14, 2025 at 12:33 PM yun zou < > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > > > > > Generic Table has been available since Polaris 1.0 > and > > > > has > > > > > > > > > attracted > > > > > > > > > > > > > interest from several users. We also have ongoing > > > > improvement > > > > > > > and > > > > > > > > > > > > extension > > > > > > > > > > > > > work in progress, including Hudi support, Parquet > > > > support, > > > > > > and > > > > > > > > > > > credential > > > > > > > > > > > > > vending. > > > > > > > > > > > > > > > > > > > > > > > > > > Given this progress, I believe it’s a good time to > > > > remove the > > > > > > > > > “beta” > > > > > > > > > > > > label. > > > > > > > > > > > > > If there are no objections, we will remove the > “beta” > > > > label > > > > > > > from > > > > > > > > > the > > > > > > > > > > > > > Generic Table in Polaris 1.3. > > > > > > > > > > > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > > > > > > > > > > > > > > > > > > > Yun > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
