Hi All, I understand Laurent’s points and agree with the sentiment regarding the API’s future evolution. The current Generic Table API, despite its limitations (like the Delta and CSV support demonstrates), is functional.
As a community, we need to achieve consensus on this topic. Consensus does not require unanimous agreement on every detail but means the project has reached a decision or compromise that everyone can accept. Specifically, using lazy consensus means we can proceed unless there is a legitimate objection accompanied by an alternative approach for discussion. To move toward consensus and incorporate Laurent's concerns, I propose the following compromise: 1. Keep the "beta" flag for the upcoming 1.2.0-incubating release. 2. Re-evaluate and aim to remove the "beta" label for the 1.3.0-incubating release. For the 1.3.0-incubating release to proceed without the "beta" label, I suggest we clearly document the existing limitations of the Generic Table API and reference the relevant discussions/issues concerning its evolution (specifically the issues for schema support and credential vending, as well as the Common Table API proposal). Does this compromise seem reasonable to everyone? If this approach does not gain approval, we can initiate a formal vote, though I believe that may be unnecessary here. Regards, JB On Fri, Nov 21, 2025 at 10:17 PM yun zou <[email protected]> wrote: > > Hi Laurent, > > Thanks a lot for the input! I fully agree that there’s still plenty of room > for improvement before this feature reaches a perfect state. Meanwhile, > since the feature is already attracting interest, I think it would be > valuable to remove the “beta” label so we can draw wider attention and > collect more feedback to help it evolve in the right direction. > > As JB pointed out, this won’t block the evolution of the API. If we need to > introduce a V2 in the future, I think that’s perfectly fine—it would show > that the Polaris community is committed to continuously improving and > supporting non-Iceberg standards. > > Since we’ve mostly aligned on removing the beta label, how about we proceed > with that? > > > Best Regards, > Yun > > On Fri, Nov 21, 2025 at 8:58 AM Laurent Goujon <[email protected]> wrote: > > > Sorry didn't want to be difficult but I was just hoping we could address > > some of the possible issuers we flagged when we decided to add the beta > > status to the api in the first place. I agree there's some great interest > > for it (we have been chatting about it also at the south bay meet up this > > week) but I don't know about adoption? But I also respect the willingness > > to move fast even if I think that keeping a beta flag for a long period of > > time is not necessarily a bad thing (maybe in the future we can separate > > experimental where things may be really removed or totally reworked from > > beta where the core structure is in place and we will try to minimize > > breaking changes) > > > > Ultimately it was just a simple request, if the consensus is to remove the > > beta flag, then please ignore my previous email and proceed. > > > > Laurent > > > > On Fri, Nov 21, 2025 at 6:08 AM Jean-Baptiste Onofré <[email protected]> > > wrote: > > > > > Hi All, > > > > > > I think it is important to distinguish between the "beta" status of > > > the Generic Table API and its future evolution. > > > > > > Strictly speaking, I believe we should remove the "beta" label now, as > > > the API already has several implementations and is seeing real-world > > > usage. > > > > > > While we all agree that the Generic Table API will continue to evolve > > > (e.g., credential vending, common table API), I anticipate that these > > > changes will primarily be "additions" rather than breaking changes to > > > the existing specification (which is pretty thin honestly). For > > > example, adding schema support should be backward compatible. > > > > > > Personally, I would rather avoid using the "beta" flag here. Features > > > in open source projects are inherently evolving, often rapidly. > > > > > > Just my two cents. > > > > > > Regards, > > > JB > > > > > > On Fri, Nov 21, 2025 at 4:46 AM Laurent Goujon <[email protected]> > > wrote: > > > > > > > > If this is okay, I would like to keep the beta a bit longer until we > > see > > > > some actual adoption cross-engines, and we address some of the points > > we > > > > discussed previously like the lack of schema for example so we can use > > > the > > > > feedback to improve on it and avoid having to release a v2 (or hold off > > > for > > > > a long time to refactor). > > > > > > > > I'd also like to amend the common table API proposal we discussed last > > > > october and hopefully refactor it as an evolution of the existing > > Generic > > > > Table API since it focused amongst other things on the metadata issue. > > > > > > > > Please let me know if you are agreeable to this. > > > > > > > > Laurent > > > > > > > > On Wed, Nov 19, 2025 at 11:03 AM yun zou <[email protected]> > > > wrote: > > > > > > > > > Hi Dmitri, > > > > > > > > > > Thanks! > > > > > > > > > > I have put up the PR: https://github.com/apache/polaris/pull/3096, > > > please > > > > > help take a look! > > > > > > > > > > Will comment on the thread also! > > > > > > > > > > Best Regards, > > > > > Yun > > > > > > > > > > On Wed, Nov 19, 2025 at 7:00 AM Dmitri Bourlatchkov < > > [email protected]> > > > > > wrote: > > > > > > > > > > > Hi Yun, > > > > > > > > > > > > I am personally ok with the approach you propose. If you would like > > > to > > > > > > remove the beta label in 1.3.0, please open a PR to unblock the > > > release > > > > > as > > > > > > discussed in [1]. If you prefer to remove the label later, please > > > comment > > > > > > on [1]. > > > > > > > > > > > > [1] > > https://lists.apache.org/thread/dhzop6tddl8f9ygbbgdoqywk0hwzsgb2 > > > > > > > > > > > > Thanks, > > > > > > Dmitri. > > > > > > > > > > > > On Tue, Nov 18, 2025 at 9:47 PM yun zou < > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Hi Dmitri, > > > > > > > > > > > > > > Thanks for the clarification! > > > > > > > > > > > > > > -- if we happen to need a major Generic Tables API change after > > > > > removing > > > > > > > the "beta" label, I believe we'd have to make a v2 of that API > > > > > > > > > > > > > > Yes, if there are changes that require altering the high-level > > > > > semantics > > > > > > of > > > > > > > the API or modifying existing fields, we would need to move to a > > > V2. > > > > > > > > > > > > > > However, since the Generic Table API currently defines only very > > > basic > > > > > > > fields, I believe the use cases described in the Table Source > > > proposal > > > > > > can > > > > > > > be addressed by extending the existing APIs. If it eventually > > > becomes > > > > > > clear > > > > > > > that a major change is required—such as a semantic-level > > > shift—then we > > > > > > can > > > > > > > transition to V2. The goal is to evolve and build on the current > > > APIs > > > > > > > wherever possible. > > > > > > > > > > > > > > Best Regards, > > > > > > > Yun > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 18, 2025 at 8:32 AM Dmitri Bourlatchkov < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Yun, > > > > > > > > > > > > > > > > I personally do not think the "beta" label is related to our > > > > > commitment > > > > > > > (or > > > > > > > > lack of it) in maintaining the API. IMHO, it means that the API > > > is > > > > > > likely > > > > > > > > to undergo major changes. > > > > > > > > > > > > > > > > I did not and do not really oppose removing the "beta" label > > > from the > > > > > > > > Generic Tables API :) > > > > > > > > > > > > > > > > My suggestion in keeping it a bit longer was purely to allow > > more > > > > > time > > > > > > > > for finding common use cases with the Table Sources proposal > > and > > > > > > possibly > > > > > > > > unifying some workflows. > > > > > > > > > > > > > > > > Having thought about that from that perspective some more, I > > > actually > > > > > > > agree > > > > > > > > that the Generic Tables API is _not_ likely to undergo major > > > changes. > > > > > > If > > > > > > > > synergies with Table Sources can be found, most of the changes > > > are > > > > > > > probably > > > > > > > > going to happen in clients that currently use the Generic > > Tables > > > API, > > > > > > the > > > > > > > > API itself is probably going to remain stable. > > > > > > > > > > > > > > > > That said, just for the sake of clarity, if we happen to need a > > > major > > > > > > > > Generic Tables API change after removing the "beta" label, I > > > believe > > > > > > we'd > > > > > > > > have to make a v2 of that API (which is ok from my POV per our > > > > > > evolution > > > > > > > > guidelines [1]). > > > > > > > > > > > > > > > > [1] https://polaris.apache.org/in-dev/unreleased/evolution/ > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > On Mon, Nov 17, 2025 at 8:20 PM yun zou < > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Dmitri, > > > > > > > > > > > > > > > > > > Thanks a lot for the feedback! > > > > > > > > > > > > > > > > > > I definitely agree that the current generic table support > > > still has > > > > > > > > > limitations and that more work is needed. However, removing > > the > > > > > > “beta” > > > > > > > > > label doesn’t mean the feature is finalized. Instead, it > > > signals to > > > > > > > users > > > > > > > > > that we are committed to maintaining and improving it over > > > time. > > > > > > > > Therefore, > > > > > > > > > I believe this will not block any future enhancements > > including > > > > > > > extending > > > > > > > > > it to address the use cases described in the Table Source > > > proposal. > > > > > > > > > > > > > > > > > > As Adam pointed out, we already have users trying it out and > > > > > > requesting > > > > > > > > > improvements. I believe this is a good time to remove the > > > “beta” > > > > > > label > > > > > > > to > > > > > > > > > encourage broader adoption and welcome new contributors who > > can > > > > > help > > > > > > > > > advance the feature. > > > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > > Yun > > > > > > > > > > > > > > > > > > On Mon, Nov 17, 2025 at 3:28 PM Adam Christian < > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > I'm in favor of removing the "beta" label because it is > > > starting > > > > > to > > > > > > > be > > > > > > > > > used > > > > > > > > > > by users. From the Slack feedback we have seen, there are > > > users > > > > > who > > > > > > > > > > have started to try out this feature. While they are still > > > > > running > > > > > > > into > > > > > > > > > > issues such as not having a Spark 4 plugin [1] and not > > having > > > > > > > > credential > > > > > > > > > > vending [2], there is real user usage indicating that folks > > > find > > > > > > this > > > > > > > > > > beneficial. > > > > > > > > > > > > > > > > > > > > Now, I do agree with Dmitri that there are known > > limitations > > > we > > > > > > need > > > > > > > to > > > > > > > > > > handle that are impeding users. The two referenced examples > > > above > > > > > > are > > > > > > > > > > obvious ones that need fixes to get wider adoption. I have > > > been > > > > > > > working > > > > > > > > > > with Yun & others to be able to solve some of these issues > > > (like > > > > > > this > > > > > > > > PR > > > > > > > > > > [3]), but I agree that there may need to be more > > significant > > > > > > changes > > > > > > > > > > including a REST API change. > > > > > > > > > > > > > > > > > > > > From my understanding, during our last conversation in the > > > > > > community > > > > > > > > > about > > > > > > > > > > the "Table Sources" proposal [4], we decided we were going > > to > > > > > > analyze > > > > > > > > the > > > > > > > > > > Parquet file use case and see if Generic Tables can support > > > this > > > > > > use > > > > > > > > case > > > > > > > > > > or whether we need to bring some ideas from the Table > > Sources > > > > > > > proposal > > > > > > > > > into > > > > > > > > > > Generic Tables. I believe that this approach is still valid > > > and > > > > > > will > > > > > > > > > > benefit our adopting users by identifying any enhancements > > > with > > > > > > these > > > > > > > > new > > > > > > > > > > use cases. > > > > > > > > > > > > > > > > > > > > What do y'all think? > > > > > > > > > > > > > > > > > > > > [1] - https://github.com/apache/polaris/issues/3021 > > > > > > > > > > [2] - https://github.com/apache/polaris/issues/3020 > > > > > > > > > > [3] - https://github.com/apache/polaris/pull/3026 > > > > > > > > > > [4] - > > > > > > > https://lists.apache.org/thread/652z1f1n2pgf3g2ow5y382wlrtnoqth0 > > > > > > > > > > > > > > > > > > > > Go community, > > > > > > > > > > > > > > > > > > > > Adam > > > > > > > > > > > > > > > > > > > > On Mon, Nov 17, 2025 at 12:23 PM Dmitri Bourlatchkov < > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi Yun, > > > > > > > > > > > > > > > > > > > > > > We should indeed review the status of the Generic Tables > > > API, > > > > > so > > > > > > > > thanks > > > > > > > > > > for > > > > > > > > > > > starting this discussion! > > > > > > > > > > > > > > > > > > > > > > From my POV the key question is: how do we intend to > > > proceed > > > > > with > > > > > > > > known > > > > > > > > > > > limitations [1]? > > > > > > > > > > > > > > > > > > > > > > I believe the Table Sources proposal [2] addresses some > > > (if not > > > > > > > all) > > > > > > > > of > > > > > > > > > > > those limitations. It is certainly suitable for the same > > > > > > > applications > > > > > > > > > > that > > > > > > > > > > > currently go through the Generic Tables API. > > > > > > > > > > > > > > > > > > > > > > I believe it would be wise to allow the Table Sources > > > proposal > > > > > to > > > > > > > > > develop > > > > > > > > > > > further to see if there are any synergies that can be > > > leveraged > > > > > > > with > > > > > > > > > > > respect to Generic Tables. If we removed the "beta" label > > > from > > > > > > the > > > > > > > > > > Generic > > > > > > > > > > > Tables API now, it would make it harder for users to > > > benefit > > > > > from > > > > > > > > those > > > > > > > > > > > synergies later (due to virtually freezing the "spec"). > > > > > > > > > > > > > > > > > > > > > > At it stands now, the Generic Tables API is supported by > > > > > Polaris, > > > > > > > so > > > > > > > > > > > existing clients can continue operating normally. > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/polaris/blob/f056e22f7f3a7c53e233bef1b88d204d6a8e4d79/site/content/in-dev/unreleased/generic-table.md?plain=1#L162-L169 > > > > > > > > > > > > > > > > > > > > > > [2] > > > > > > > https://lists.apache.org/thread/9f5b75fy25l9yzrtnlzqg6yh1bqdyjbt > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Nov 14, 2025 at 12:33 PM yun zou < > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > > > Generic Table has been available since Polaris 1.0 and > > > has > > > > > > > > attracted > > > > > > > > > > > > interest from several users. We also have ongoing > > > improvement > > > > > > and > > > > > > > > > > > extension > > > > > > > > > > > > work in progress, including Hudi support, Parquet > > > support, > > > > > and > > > > > > > > > > credential > > > > > > > > > > > > vending. > > > > > > > > > > > > > > > > > > > > > > > > Given this progress, I believe it’s a good time to > > > remove the > > > > > > > > “beta” > > > > > > > > > > > label. > > > > > > > > > > > > If there are no objections, we will remove the “beta” > > > label > > > > > > from > > > > > > > > the > > > > > > > > > > > > Generic Table in Polaris 1.3. > > > > > > > > > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > > > > > > > > > > > > > > > > > Yun > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
