Hi All, I noticed that PR #3096 was merged before this discussion reached a clear consensus. Moving forward, I think it would be beneficial to ensure we have community consensus before merging, especially without an update on the thread.
Regarding Laurent's concerns, I am fine with removing the "beta" flag for the Generic Table API, however, I suggest updating the Generic Table documentation to clearly state its current limitations and proposed future evolutions. Since documenting the limitations is not a blocker for the upcoming release, I propose we proceed with the 1.3.0-incubating release and address the documentation update immediately following the release. Regards, JB On Sat, Nov 22, 2025 at 3:17 PM Jean-Baptiste Onofré <[email protected]> wrote: > > Sorry I meant 1.3.0-incubating and 1.4.0-incubating in terms of releases. > > Le sam. 22 nov. 2025 à 11:27, Jean-Baptiste Onofré <[email protected]> a > écrit : >> >> Hi All, >> >> I understand Laurent’s points and agree with the sentiment regarding >> the API’s future evolution. The current Generic Table API, despite its >> limitations (like the Delta and CSV support demonstrates), is >> functional. >> >> As a community, we need to achieve consensus on this topic. Consensus >> does not require unanimous agreement on every detail but means the >> project has reached a decision or compromise that everyone can accept. >> Specifically, using lazy consensus means we can proceed unless there >> is a legitimate objection accompanied by an alternative approach for >> discussion. >> >> To move toward consensus and incorporate Laurent's concerns, I propose >> the following compromise: >> >> 1. Keep the "beta" flag for the upcoming 1.2.0-incubating release. >> 2. Re-evaluate and aim to remove the "beta" label for the >> 1.3.0-incubating release. >> >> For the 1.3.0-incubating release to proceed without the "beta" label, >> I suggest we clearly document the existing limitations of the Generic >> Table API and reference the relevant discussions/issues concerning its >> evolution (specifically the issues for schema support and credential >> vending, as well as the Common Table API proposal). >> >> Does this compromise seem reasonable to everyone? >> >> If this approach does not gain approval, we can initiate a formal >> vote, though I believe that may be unnecessary here. >> >> Regards, >> JB >> >> On Fri, Nov 21, 2025 at 10:17 PM yun zou <[email protected]> wrote: >> > >> > Hi Laurent, >> > >> > Thanks a lot for the input! I fully agree that there’s still plenty of room >> > for improvement before this feature reaches a perfect state. Meanwhile, >> > since the feature is already attracting interest, I think it would be >> > valuable to remove the “beta” label so we can draw wider attention and >> > collect more feedback to help it evolve in the right direction. >> > >> > As JB pointed out, this won’t block the evolution of the API. If we need to >> > introduce a V2 in the future, I think that’s perfectly fine—it would show >> > that the Polaris community is committed to continuously improving and >> > supporting non-Iceberg standards. >> > >> > Since we’ve mostly aligned on removing the beta label, how about we proceed >> > with that? >> > >> > >> > Best Regards, >> > Yun >> > >> > On Fri, Nov 21, 2025 at 8:58 AM Laurent Goujon <[email protected]> wrote: >> > >> > > Sorry didn't want to be difficult but I was just hoping we could address >> > > some of the possible issuers we flagged when we decided to add the beta >> > > status to the api in the first place. I agree there's some great interest >> > > for it (we have been chatting about it also at the south bay meet up this >> > > week) but I don't know about adoption? But I also respect the willingness >> > > to move fast even if I think that keeping a beta flag for a long period >> > > of >> > > time is not necessarily a bad thing (maybe in the future we can separate >> > > experimental where things may be really removed or totally reworked from >> > > beta where the core structure is in place and we will try to minimize >> > > breaking changes) >> > > >> > > Ultimately it was just a simple request, if the consensus is to remove >> > > the >> > > beta flag, then please ignore my previous email and proceed. >> > > >> > > Laurent >> > > >> > > On Fri, Nov 21, 2025 at 6:08 AM Jean-Baptiste Onofré <[email protected]> >> > > wrote: >> > > >> > > > Hi All, >> > > > >> > > > I think it is important to distinguish between the "beta" status of >> > > > the Generic Table API and its future evolution. >> > > > >> > > > Strictly speaking, I believe we should remove the "beta" label now, as >> > > > the API already has several implementations and is seeing real-world >> > > > usage. >> > > > >> > > > While we all agree that the Generic Table API will continue to evolve >> > > > (e.g., credential vending, common table API), I anticipate that these >> > > > changes will primarily be "additions" rather than breaking changes to >> > > > the existing specification (which is pretty thin honestly). For >> > > > example, adding schema support should be backward compatible. >> > > > >> > > > Personally, I would rather avoid using the "beta" flag here. Features >> > > > in open source projects are inherently evolving, often rapidly. >> > > > >> > > > Just my two cents. >> > > > >> > > > Regards, >> > > > JB >> > > > >> > > > On Fri, Nov 21, 2025 at 4:46 AM Laurent Goujon <[email protected]> >> > > wrote: >> > > > > >> > > > > If this is okay, I would like to keep the beta a bit longer until we >> > > see >> > > > > some actual adoption cross-engines, and we address some of the points >> > > we >> > > > > discussed previously like the lack of schema for example so we can >> > > > > use >> > > > the >> > > > > feedback to improve on it and avoid having to release a v2 (or hold >> > > > > off >> > > > for >> > > > > a long time to refactor). >> > > > > >> > > > > I'd also like to amend the common table API proposal we discussed >> > > > > last >> > > > > october and hopefully refactor it as an evolution of the existing >> > > Generic >> > > > > Table API since it focused amongst other things on the metadata >> > > > > issue. >> > > > > >> > > > > Please let me know if you are agreeable to this. >> > > > > >> > > > > Laurent >> > > > > >> > > > > On Wed, Nov 19, 2025 at 11:03 AM yun zou <[email protected]> >> > > > wrote: >> > > > > >> > > > > > Hi Dmitri, >> > > > > > >> > > > > > Thanks! >> > > > > > >> > > > > > I have put up the PR: https://github.com/apache/polaris/pull/3096, >> > > > please >> > > > > > help take a look! >> > > > > > >> > > > > > Will comment on the thread also! >> > > > > > >> > > > > > Best Regards, >> > > > > > Yun >> > > > > > >> > > > > > On Wed, Nov 19, 2025 at 7:00 AM Dmitri Bourlatchkov < >> > > [email protected]> >> > > > > > wrote: >> > > > > > >> > > > > > > Hi Yun, >> > > > > > > >> > > > > > > I am personally ok with the approach you propose. If you would >> > > > > > > like >> > > > to >> > > > > > > remove the beta label in 1.3.0, please open a PR to unblock the >> > > > release >> > > > > > as >> > > > > > > discussed in [1]. If you prefer to remove the label later, please >> > > > comment >> > > > > > > on [1]. >> > > > > > > >> > > > > > > [1] >> > > https://lists.apache.org/thread/dhzop6tddl8f9ygbbgdoqywk0hwzsgb2 >> > > > > > > >> > > > > > > Thanks, >> > > > > > > Dmitri. >> > > > > > > >> > > > > > > On Tue, Nov 18, 2025 at 9:47 PM yun zou < >> > > [email protected]> >> > > > > > > wrote: >> > > > > > > >> > > > > > > > Hi Dmitri, >> > > > > > > > >> > > > > > > > Thanks for the clarification! >> > > > > > > > >> > > > > > > > -- if we happen to need a major Generic Tables API change after >> > > > > > removing >> > > > > > > > the "beta" label, I believe we'd have to make a v2 of that API >> > > > > > > > >> > > > > > > > Yes, if there are changes that require altering the high-level >> > > > > > semantics >> > > > > > > of >> > > > > > > > the API or modifying existing fields, we would need to move to >> > > > > > > > a >> > > > V2. >> > > > > > > > >> > > > > > > > However, since the Generic Table API currently defines only >> > > > > > > > very >> > > > basic >> > > > > > > > fields, I believe the use cases described in the Table Source >> > > > proposal >> > > > > > > can >> > > > > > > > be addressed by extending the existing APIs. If it eventually >> > > > becomes >> > > > > > > clear >> > > > > > > > that a major change is required—such as a semantic-level >> > > > shift—then we >> > > > > > > can >> > > > > > > > transition to V2. The goal is to evolve and build on the >> > > > > > > > current >> > > > APIs >> > > > > > > > wherever possible. >> > > > > > > > >> > > > > > > > Best Regards, >> > > > > > > > Yun >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > On Tue, Nov 18, 2025 at 8:32 AM Dmitri Bourlatchkov < >> > > > [email protected]> >> > > > > > > > wrote: >> > > > > > > > >> > > > > > > > > Hi Yun, >> > > > > > > > > >> > > > > > > > > I personally do not think the "beta" label is related to our >> > > > > > commitment >> > > > > > > > (or >> > > > > > > > > lack of it) in maintaining the API. IMHO, it means that the >> > > > > > > > > API >> > > > is >> > > > > > > likely >> > > > > > > > > to undergo major changes. >> > > > > > > > > >> > > > > > > > > I did not and do not really oppose removing the "beta" label >> > > > from the >> > > > > > > > > Generic Tables API :) >> > > > > > > > > >> > > > > > > > > My suggestion in keeping it a bit longer was purely to allow >> > > more >> > > > > > time >> > > > > > > > > for finding common use cases with the Table Sources proposal >> > > and >> > > > > > > possibly >> > > > > > > > > unifying some workflows. >> > > > > > > > > >> > > > > > > > > Having thought about that from that perspective some more, I >> > > > actually >> > > > > > > > agree >> > > > > > > > > that the Generic Tables API is _not_ likely to undergo major >> > > > changes. >> > > > > > > If >> > > > > > > > > synergies with Table Sources can be found, most of the >> > > > > > > > > changes >> > > > are >> > > > > > > > probably >> > > > > > > > > going to happen in clients that currently use the Generic >> > > Tables >> > > > API, >> > > > > > > the >> > > > > > > > > API itself is probably going to remain stable. >> > > > > > > > > >> > > > > > > > > That said, just for the sake of clarity, if we happen to >> > > > > > > > > need a >> > > > major >> > > > > > > > > Generic Tables API change after removing the "beta" label, I >> > > > believe >> > > > > > > we'd >> > > > > > > > > have to make a v2 of that API (which is ok from my POV per >> > > > > > > > > our >> > > > > > > evolution >> > > > > > > > > guidelines [1]). >> > > > > > > > > >> > > > > > > > > [1] https://polaris.apache.org/in-dev/unreleased/evolution/ >> > > > > > > > > >> > > > > > > > > Cheers, >> > > > > > > > > Dmitri. >> > > > > > > > > >> > > > > > > > > On Mon, Nov 17, 2025 at 8:20 PM yun zou < >> > > > [email protected]> >> > > > > > > > > wrote: >> > > > > > > > > >> > > > > > > > > > Hi Dmitri, >> > > > > > > > > > >> > > > > > > > > > Thanks a lot for the feedback! >> > > > > > > > > > >> > > > > > > > > > I definitely agree that the current generic table support >> > > > still has >> > > > > > > > > > limitations and that more work is needed. However, removing >> > > the >> > > > > > > “beta” >> > > > > > > > > > label doesn’t mean the feature is finalized. Instead, it >> > > > signals to >> > > > > > > > users >> > > > > > > > > > that we are committed to maintaining and improving it over >> > > > time. >> > > > > > > > > Therefore, >> > > > > > > > > > I believe this will not block any future enhancements >> > > including >> > > > > > > > extending >> > > > > > > > > > it to address the use cases described in the Table Source >> > > > proposal. >> > > > > > > > > > >> > > > > > > > > > As Adam pointed out, we already have users trying it out >> > > > > > > > > > and >> > > > > > > requesting >> > > > > > > > > > improvements. I believe this is a good time to remove the >> > > > “beta” >> > > > > > > label >> > > > > > > > to >> > > > > > > > > > encourage broader adoption and welcome new contributors who >> > > can >> > > > > > help >> > > > > > > > > > advance the feature. >> > > > > > > > > > >> > > > > > > > > > Best Regards, >> > > > > > > > > > Yun >> > > > > > > > > > >> > > > > > > > > > On Mon, Nov 17, 2025 at 3:28 PM Adam Christian < >> > > > > > > > > > [email protected]> wrote: >> > > > > > > > > > >> > > > > > > > > > > I'm in favor of removing the "beta" label because it is >> > > > starting >> > > > > > to >> > > > > > > > be >> > > > > > > > > > used >> > > > > > > > > > > by users. From the Slack feedback we have seen, there are >> > > > users >> > > > > > who >> > > > > > > > > > > have started to try out this feature. While they are >> > > > > > > > > > > still >> > > > > > running >> > > > > > > > into >> > > > > > > > > > > issues such as not having a Spark 4 plugin [1] and not >> > > having >> > > > > > > > > credential >> > > > > > > > > > > vending [2], there is real user usage indicating that >> > > > > > > > > > > folks >> > > > find >> > > > > > > this >> > > > > > > > > > > beneficial. >> > > > > > > > > > > >> > > > > > > > > > > Now, I do agree with Dmitri that there are known >> > > limitations >> > > > we >> > > > > > > need >> > > > > > > > to >> > > > > > > > > > > handle that are impeding users. The two referenced >> > > > > > > > > > > examples >> > > > above >> > > > > > > are >> > > > > > > > > > > obvious ones that need fixes to get wider adoption. I >> > > > > > > > > > > have >> > > > been >> > > > > > > > working >> > > > > > > > > > > with Yun & others to be able to solve some of these >> > > > > > > > > > > issues >> > > > (like >> > > > > > > this >> > > > > > > > > PR >> > > > > > > > > > > [3]), but I agree that there may need to be more >> > > significant >> > > > > > > changes >> > > > > > > > > > > including a REST API change. >> > > > > > > > > > > >> > > > > > > > > > > From my understanding, during our last conversation in >> > > > > > > > > > > the >> > > > > > > community >> > > > > > > > > > about >> > > > > > > > > > > the "Table Sources" proposal [4], we decided we were >> > > > > > > > > > > going >> > > to >> > > > > > > analyze >> > > > > > > > > the >> > > > > > > > > > > Parquet file use case and see if Generic Tables can >> > > > > > > > > > > support >> > > > this >> > > > > > > use >> > > > > > > > > case >> > > > > > > > > > > or whether we need to bring some ideas from the Table >> > > Sources >> > > > > > > > proposal >> > > > > > > > > > into >> > > > > > > > > > > Generic Tables. I believe that this approach is still >> > > > > > > > > > > valid >> > > > and >> > > > > > > will >> > > > > > > > > > > benefit our adopting users by identifying any >> > > > > > > > > > > enhancements >> > > > with >> > > > > > > these >> > > > > > > > > new >> > > > > > > > > > > use cases. >> > > > > > > > > > > >> > > > > > > > > > > What do y'all think? >> > > > > > > > > > > >> > > > > > > > > > > [1] - https://github.com/apache/polaris/issues/3021 >> > > > > > > > > > > [2] - https://github.com/apache/polaris/issues/3020 >> > > > > > > > > > > [3] - https://github.com/apache/polaris/pull/3026 >> > > > > > > > > > > [4] - >> > > > > > > > https://lists.apache.org/thread/652z1f1n2pgf3g2ow5y382wlrtnoqth0 >> > > > > > > > > > > >> > > > > > > > > > > Go community, >> > > > > > > > > > > >> > > > > > > > > > > Adam >> > > > > > > > > > > >> > > > > > > > > > > On Mon, Nov 17, 2025 at 12:23 PM Dmitri Bourlatchkov < >> > > > > > > > [email protected] >> > > > > > > > > > >> > > > > > > > > > > wrote: >> > > > > > > > > > > >> > > > > > > > > > > > Hi Yun, >> > > > > > > > > > > > >> > > > > > > > > > > > We should indeed review the status of the Generic >> > > > > > > > > > > > Tables >> > > > API, >> > > > > > so >> > > > > > > > > thanks >> > > > > > > > > > > for >> > > > > > > > > > > > starting this discussion! >> > > > > > > > > > > > >> > > > > > > > > > > > From my POV the key question is: how do we intend to >> > > > proceed >> > > > > > with >> > > > > > > > > known >> > > > > > > > > > > > limitations [1]? >> > > > > > > > > > > > >> > > > > > > > > > > > I believe the Table Sources proposal [2] addresses some >> > > > (if not >> > > > > > > > all) >> > > > > > > > > of >> > > > > > > > > > > > those limitations. It is certainly suitable for the >> > > > > > > > > > > > same >> > > > > > > > applications >> > > > > > > > > > > that >> > > > > > > > > > > > currently go through the Generic Tables API. >> > > > > > > > > > > > >> > > > > > > > > > > > I believe it would be wise to allow the Table Sources >> > > > proposal >> > > > > > to >> > > > > > > > > > develop >> > > > > > > > > > > > further to see if there are any synergies that can be >> > > > leveraged >> > > > > > > > with >> > > > > > > > > > > > respect to Generic Tables. If we removed the "beta" >> > > > > > > > > > > > label >> > > > from >> > > > > > > the >> > > > > > > > > > > Generic >> > > > > > > > > > > > Tables API now, it would make it harder for users to >> > > > benefit >> > > > > > from >> > > > > > > > > those >> > > > > > > > > > > > synergies later (due to virtually freezing the "spec"). >> > > > > > > > > > > > >> > > > > > > > > > > > At it stands now, the Generic Tables API is supported >> > > > > > > > > > > > by >> > > > > > Polaris, >> > > > > > > > so >> > > > > > > > > > > > existing clients can continue operating normally. >> > > > > > > > > > > > >> > > > > > > > > > > > [1] >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > >> > > https://github.com/apache/polaris/blob/f056e22f7f3a7c53e233bef1b88d204d6a8e4d79/site/content/in-dev/unreleased/generic-table.md?plain=1#L162-L169 >> > > > > > > > > > > > >> > > > > > > > > > > > [2] >> > > > > > > > https://lists.apache.org/thread/9f5b75fy25l9yzrtnlzqg6yh1bqdyjbt >> > > > > > > > > > > > >> > > > > > > > > > > > Thanks, >> > > > > > > > > > > > Dmitri. >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > On Fri, Nov 14, 2025 at 12:33 PM yun zou < >> > > > > > > > [email protected] >> > > > > > > > > > >> > > > > > > > > > > > wrote: >> > > > > > > > > > > > >> > > > > > > > > > > > > Hi All, >> > > > > > > > > > > > > >> > > > > > > > > > > > > Generic Table has been available since Polaris 1.0 >> > > > > > > > > > > > > and >> > > > has >> > > > > > > > > attracted >> > > > > > > > > > > > > interest from several users. We also have ongoing >> > > > improvement >> > > > > > > and >> > > > > > > > > > > > extension >> > > > > > > > > > > > > work in progress, including Hudi support, Parquet >> > > > support, >> > > > > > and >> > > > > > > > > > > credential >> > > > > > > > > > > > > vending. >> > > > > > > > > > > > > >> > > > > > > > > > > > > Given this progress, I believe it’s a good time to >> > > > remove the >> > > > > > > > > “beta” >> > > > > > > > > > > > label. >> > > > > > > > > > > > > If there are no objections, we will remove the “beta” >> > > > label >> > > > > > > from >> > > > > > > > > the >> > > > > > > > > > > > > Generic Table in Polaris 1.3. >> > > > > > > > > > > > > >> > > > > > > > > > > > > Best Regards, >> > > > > > > > > > > > > >> > > > > > > > > > > > > Yun >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > >> > >
