Hi Yun, That's my mistake; I might have misinterpreted Laurent's last comment.
You made the right call on the merge. Thanks! Regards, JB Le lun. 24 nov. 2025 à 18:40, yun zou <[email protected]> a écrit : > Hi JB, > > Sorry about that! > > I went ahead with the merge since Laurent mentioned he was ok to move > forward as long as the rest of the community had reached consensus, and it > seemed that most people were aligned. I’m happy to revert it if you prefer. > > Regarding the limitations: the current documentation already includes a > clear section outlining the feature’s limitations [1]. If there are > additional points you think should be covered, just let me know and I can > definitely add them in. > > [1]: https://polaris.apache.org/releases/1.2.0/generic-table/ > > > Best Regards, > > Yun > > > On Mon, Nov 24, 2025 at 1:13 AM Jean-Baptiste Onofré <[email protected]> > wrote: > >> Hi All, >> >> I noticed that PR #3096 was merged before this discussion reached a >> clear consensus. Moving forward, I think it would be beneficial to >> ensure we have community consensus before merging, especially without >> an update on the thread. >> >> Regarding Laurent's concerns, I am fine with removing the "beta" flag >> for the Generic Table API, however, I suggest updating the Generic >> Table documentation to clearly state its current limitations and >> proposed future evolutions. >> >> Since documenting the limitations is not a blocker for the upcoming >> release, I propose we proceed with the 1.3.0-incubating release and >> address the documentation update immediately following the release. >> >> Regards, >> JB >> >> On Sat, Nov 22, 2025 at 3:17 PM Jean-Baptiste Onofré <[email protected]> >> wrote: >> > >> > Sorry I meant 1.3.0-incubating and 1.4.0-incubating in terms of >> releases. >> > >> > Le sam. 22 nov. 2025 à 11:27, Jean-Baptiste Onofré <[email protected]> a >> écrit : >> >> >> >> Hi All, >> >> >> >> I understand Laurent’s points and agree with the sentiment regarding >> >> the API’s future evolution. The current Generic Table API, despite its >> >> limitations (like the Delta and CSV support demonstrates), is >> >> functional. >> >> >> >> As a community, we need to achieve consensus on this topic. Consensus >> >> does not require unanimous agreement on every detail but means the >> >> project has reached a decision or compromise that everyone can accept. >> >> Specifically, using lazy consensus means we can proceed unless there >> >> is a legitimate objection accompanied by an alternative approach for >> >> discussion. >> >> >> >> To move toward consensus and incorporate Laurent's concerns, I propose >> >> the following compromise: >> >> >> >> 1. Keep the "beta" flag for the upcoming 1.2.0-incubating release. >> >> 2. Re-evaluate and aim to remove the "beta" label for the >> >> 1.3.0-incubating release. >> >> >> >> For the 1.3.0-incubating release to proceed without the "beta" label, >> >> I suggest we clearly document the existing limitations of the Generic >> >> Table API and reference the relevant discussions/issues concerning its >> >> evolution (specifically the issues for schema support and credential >> >> vending, as well as the Common Table API proposal). >> >> >> >> Does this compromise seem reasonable to everyone? >> >> >> >> If this approach does not gain approval, we can initiate a formal >> >> vote, though I believe that may be unnecessary here. >> >> >> >> Regards, >> >> JB >> >> >> >> On Fri, Nov 21, 2025 at 10:17 PM yun zou <[email protected]> >> wrote: >> >> > >> >> > Hi Laurent, >> >> > >> >> > Thanks a lot for the input! I fully agree that there’s still plenty >> of room >> >> > for improvement before this feature reaches a perfect state. >> Meanwhile, >> >> > since the feature is already attracting interest, I think it would be >> >> > valuable to remove the “beta” label so we can draw wider attention >> and >> >> > collect more feedback to help it evolve in the right direction. >> >> > >> >> > As JB pointed out, this won’t block the evolution of the API. If we >> need to >> >> > introduce a V2 in the future, I think that’s perfectly fine—it would >> show >> >> > that the Polaris community is committed to continuously improving and >> >> > supporting non-Iceberg standards. >> >> > >> >> > Since we’ve mostly aligned on removing the beta label, how about we >> proceed >> >> > with that? >> >> > >> >> > >> >> > Best Regards, >> >> > Yun >> >> > >> >> > On Fri, Nov 21, 2025 at 8:58 AM Laurent Goujon <[email protected]> >> wrote: >> >> > >> >> > > Sorry didn't want to be difficult but I was just hoping we could >> address >> >> > > some of the possible issuers we flagged when we decided to add the >> beta >> >> > > status to the api in the first place. I agree there's some great >> interest >> >> > > for it (we have been chatting about it also at the south bay meet >> up this >> >> > > week) but I don't know about adoption? But I also respect the >> willingness >> >> > > to move fast even if I think that keeping a beta flag for a long >> period of >> >> > > time is not necessarily a bad thing (maybe in the future we can >> separate >> >> > > experimental where things may be really removed or totally >> reworked from >> >> > > beta where the core structure is in place and we will try to >> minimize >> >> > > breaking changes) >> >> > > >> >> > > Ultimately it was just a simple request, if the consensus is to >> remove the >> >> > > beta flag, then please ignore my previous email and proceed. >> >> > > >> >> > > Laurent >> >> > > >> >> > > On Fri, Nov 21, 2025 at 6:08 AM Jean-Baptiste Onofré < >> [email protected]> >> >> > > wrote: >> >> > > >> >> > > > Hi All, >> >> > > > >> >> > > > I think it is important to distinguish between the "beta" status >> of >> >> > > > the Generic Table API and its future evolution. >> >> > > > >> >> > > > Strictly speaking, I believe we should remove the "beta" label >> now, as >> >> > > > the API already has several implementations and is seeing >> real-world >> >> > > > usage. >> >> > > > >> >> > > > While we all agree that the Generic Table API will continue to >> evolve >> >> > > > (e.g., credential vending, common table API), I anticipate that >> these >> >> > > > changes will primarily be "additions" rather than breaking >> changes to >> >> > > > the existing specification (which is pretty thin honestly). For >> >> > > > example, adding schema support should be backward compatible. >> >> > > > >> >> > > > Personally, I would rather avoid using the "beta" flag here. >> Features >> >> > > > in open source projects are inherently evolving, often rapidly. >> >> > > > >> >> > > > Just my two cents. >> >> > > > >> >> > > > Regards, >> >> > > > JB >> >> > > > >> >> > > > On Fri, Nov 21, 2025 at 4:46 AM Laurent Goujon < >> [email protected]> >> >> > > wrote: >> >> > > > > >> >> > > > > If this is okay, I would like to keep the beta a bit longer >> until we >> >> > > see >> >> > > > > some actual adoption cross-engines, and we address some of the >> points >> >> > > we >> >> > > > > discussed previously like the lack of schema for example so we >> can use >> >> > > > the >> >> > > > > feedback to improve on it and avoid having to release a v2 (or >> hold off >> >> > > > for >> >> > > > > a long time to refactor). >> >> > > > > >> >> > > > > I'd also like to amend the common table API proposal we >> discussed last >> >> > > > > october and hopefully refactor it as an evolution of the >> existing >> >> > > Generic >> >> > > > > Table API since it focused amongst other things on the >> metadata issue. >> >> > > > > >> >> > > > > Please let me know if you are agreeable to this. >> >> > > > > >> >> > > > > Laurent >> >> > > > > >> >> > > > > On Wed, Nov 19, 2025 at 11:03 AM yun zou < >> [email protected]> >> >> > > > wrote: >> >> > > > > >> >> > > > > > Hi Dmitri, >> >> > > > > > >> >> > > > > > Thanks! >> >> > > > > > >> >> > > > > > I have put up the PR: >> https://github.com/apache/polaris/pull/3096, >> >> > > > please >> >> > > > > > help take a look! >> >> > > > > > >> >> > > > > > Will comment on the thread also! >> >> > > > > > >> >> > > > > > Best Regards, >> >> > > > > > Yun >> >> > > > > > >> >> > > > > > On Wed, Nov 19, 2025 at 7:00 AM Dmitri Bourlatchkov < >> >> > > [email protected]> >> >> > > > > > wrote: >> >> > > > > > >> >> > > > > > > Hi Yun, >> >> > > > > > > >> >> > > > > > > I am personally ok with the approach you propose. If you >> would like >> >> > > > to >> >> > > > > > > remove the beta label in 1.3.0, please open a PR to >> unblock the >> >> > > > release >> >> > > > > > as >> >> > > > > > > discussed in [1]. If you prefer to remove the label later, >> please >> >> > > > comment >> >> > > > > > > on [1]. >> >> > > > > > > >> >> > > > > > > [1] >> >> > > https://lists.apache.org/thread/dhzop6tddl8f9ygbbgdoqywk0hwzsgb2 >> >> > > > > > > >> >> > > > > > > Thanks, >> >> > > > > > > Dmitri. >> >> > > > > > > >> >> > > > > > > On Tue, Nov 18, 2025 at 9:47 PM yun zou < >> >> > > [email protected]> >> >> > > > > > > wrote: >> >> > > > > > > >> >> > > > > > > > Hi Dmitri, >> >> > > > > > > > >> >> > > > > > > > Thanks for the clarification! >> >> > > > > > > > >> >> > > > > > > > -- if we happen to need a major Generic Tables API >> change after >> >> > > > > > removing >> >> > > > > > > > the "beta" label, I believe we'd have to make a v2 of >> that API >> >> > > > > > > > >> >> > > > > > > > Yes, if there are changes that require altering the >> high-level >> >> > > > > > semantics >> >> > > > > > > of >> >> > > > > > > > the API or modifying existing fields, we would need to >> move to a >> >> > > > V2. >> >> > > > > > > > >> >> > > > > > > > However, since the Generic Table API currently defines >> only very >> >> > > > basic >> >> > > > > > > > fields, I believe the use cases described in the Table >> Source >> >> > > > proposal >> >> > > > > > > can >> >> > > > > > > > be addressed by extending the existing APIs. If it >> eventually >> >> > > > becomes >> >> > > > > > > clear >> >> > > > > > > > that a major change is required—such as a semantic-level >> >> > > > shift—then we >> >> > > > > > > can >> >> > > > > > > > transition to V2. The goal is to evolve and build on the >> current >> >> > > > APIs >> >> > > > > > > > wherever possible. >> >> > > > > > > > >> >> > > > > > > > Best Regards, >> >> > > > > > > > Yun >> >> > > > > > > > >> >> > > > > > > > >> >> > > > > > > > >> >> > > > > > > > On Tue, Nov 18, 2025 at 8:32 AM Dmitri Bourlatchkov < >> >> > > > [email protected]> >> >> > > > > > > > wrote: >> >> > > > > > > > >> >> > > > > > > > > Hi Yun, >> >> > > > > > > > > >> >> > > > > > > > > I personally do not think the "beta" label is related >> to our >> >> > > > > > commitment >> >> > > > > > > > (or >> >> > > > > > > > > lack of it) in maintaining the API. IMHO, it means >> that the API >> >> > > > is >> >> > > > > > > likely >> >> > > > > > > > > to undergo major changes. >> >> > > > > > > > > >> >> > > > > > > > > I did not and do not really oppose removing the "beta" >> label >> >> > > > from the >> >> > > > > > > > > Generic Tables API :) >> >> > > > > > > > > >> >> > > > > > > > > My suggestion in keeping it a bit longer was purely to >> allow >> >> > > more >> >> > > > > > time >> >> > > > > > > > > for finding common use cases with the Table Sources >> proposal >> >> > > and >> >> > > > > > > possibly >> >> > > > > > > > > unifying some workflows. >> >> > > > > > > > > >> >> > > > > > > > > Having thought about that from that perspective some >> more, I >> >> > > > actually >> >> > > > > > > > agree >> >> > > > > > > > > that the Generic Tables API is _not_ likely to undergo >> major >> >> > > > changes. >> >> > > > > > > If >> >> > > > > > > > > synergies with Table Sources can be found, most of the >> changes >> >> > > > are >> >> > > > > > > > probably >> >> > > > > > > > > going to happen in clients that currently use the >> Generic >> >> > > Tables >> >> > > > API, >> >> > > > > > > the >> >> > > > > > > > > API itself is probably going to remain stable. >> >> > > > > > > > > >> >> > > > > > > > > That said, just for the sake of clarity, if we happen >> to need a >> >> > > > major >> >> > > > > > > > > Generic Tables API change after removing the "beta" >> label, I >> >> > > > believe >> >> > > > > > > we'd >> >> > > > > > > > > have to make a v2 of that API (which is ok from my POV >> per our >> >> > > > > > > evolution >> >> > > > > > > > > guidelines [1]). >> >> > > > > > > > > >> >> > > > > > > > > [1] >> https://polaris.apache.org/in-dev/unreleased/evolution/ >> >> > > > > > > > > >> >> > > > > > > > > Cheers, >> >> > > > > > > > > Dmitri. >> >> > > > > > > > > >> >> > > > > > > > > On Mon, Nov 17, 2025 at 8:20 PM yun zou < >> >> > > > [email protected]> >> >> > > > > > > > > wrote: >> >> > > > > > > > > >> >> > > > > > > > > > Hi Dmitri, >> >> > > > > > > > > > >> >> > > > > > > > > > Thanks a lot for the feedback! >> >> > > > > > > > > > >> >> > > > > > > > > > I definitely agree that the current generic table >> support >> >> > > > still has >> >> > > > > > > > > > limitations and that more work is needed. However, >> removing >> >> > > the >> >> > > > > > > “beta” >> >> > > > > > > > > > label doesn’t mean the feature is finalized. >> Instead, it >> >> > > > signals to >> >> > > > > > > > users >> >> > > > > > > > > > that we are committed to maintaining and improving >> it over >> >> > > > time. >> >> > > > > > > > > Therefore, >> >> > > > > > > > > > I believe this will not block any future enhancements >> >> > > including >> >> > > > > > > > extending >> >> > > > > > > > > > it to address the use cases described in the Table >> Source >> >> > > > proposal. >> >> > > > > > > > > > >> >> > > > > > > > > > As Adam pointed out, we already have users trying it >> out and >> >> > > > > > > requesting >> >> > > > > > > > > > improvements. I believe this is a good time to >> remove the >> >> > > > “beta” >> >> > > > > > > label >> >> > > > > > > > to >> >> > > > > > > > > > encourage broader adoption and welcome new >> contributors who >> >> > > can >> >> > > > > > help >> >> > > > > > > > > > advance the feature. >> >> > > > > > > > > > >> >> > > > > > > > > > Best Regards, >> >> > > > > > > > > > Yun >> >> > > > > > > > > > >> >> > > > > > > > > > On Mon, Nov 17, 2025 at 3:28 PM Adam Christian < >> >> > > > > > > > > > [email protected]> wrote: >> >> > > > > > > > > > >> >> > > > > > > > > > > I'm in favor of removing the "beta" label because >> it is >> >> > > > starting >> >> > > > > > to >> >> > > > > > > > be >> >> > > > > > > > > > used >> >> > > > > > > > > > > by users. From the Slack feedback we have seen, >> there are >> >> > > > users >> >> > > > > > who >> >> > > > > > > > > > > have started to try out this feature. While they >> are still >> >> > > > > > running >> >> > > > > > > > into >> >> > > > > > > > > > > issues such as not having a Spark 4 plugin [1] and >> not >> >> > > having >> >> > > > > > > > > credential >> >> > > > > > > > > > > vending [2], there is real user usage indicating >> that folks >> >> > > > find >> >> > > > > > > this >> >> > > > > > > > > > > beneficial. >> >> > > > > > > > > > > >> >> > > > > > > > > > > Now, I do agree with Dmitri that there are known >> >> > > limitations >> >> > > > we >> >> > > > > > > need >> >> > > > > > > > to >> >> > > > > > > > > > > handle that are impeding users. The two referenced >> examples >> >> > > > above >> >> > > > > > > are >> >> > > > > > > > > > > obvious ones that need fixes to get wider >> adoption. I have >> >> > > > been >> >> > > > > > > > working >> >> > > > > > > > > > > with Yun & others to be able to solve some of >> these issues >> >> > > > (like >> >> > > > > > > this >> >> > > > > > > > > PR >> >> > > > > > > > > > > [3]), but I agree that there may need to be more >> >> > > significant >> >> > > > > > > changes >> >> > > > > > > > > > > including a REST API change. >> >> > > > > > > > > > > >> >> > > > > > > > > > > From my understanding, during our last >> conversation in the >> >> > > > > > > community >> >> > > > > > > > > > about >> >> > > > > > > > > > > the "Table Sources" proposal [4], we decided we >> were going >> >> > > to >> >> > > > > > > analyze >> >> > > > > > > > > the >> >> > > > > > > > > > > Parquet file use case and see if Generic Tables >> can support >> >> > > > this >> >> > > > > > > use >> >> > > > > > > > > case >> >> > > > > > > > > > > or whether we need to bring some ideas from the >> Table >> >> > > Sources >> >> > > > > > > > proposal >> >> > > > > > > > > > into >> >> > > > > > > > > > > Generic Tables. I believe that this approach is >> still valid >> >> > > > and >> >> > > > > > > will >> >> > > > > > > > > > > benefit our adopting users by identifying any >> enhancements >> >> > > > with >> >> > > > > > > these >> >> > > > > > > > > new >> >> > > > > > > > > > > use cases. >> >> > > > > > > > > > > >> >> > > > > > > > > > > What do y'all think? >> >> > > > > > > > > > > >> >> > > > > > > > > > > [1] - >> https://github.com/apache/polaris/issues/3021 >> >> > > > > > > > > > > [2] - >> https://github.com/apache/polaris/issues/3020 >> >> > > > > > > > > > > [3] - https://github.com/apache/polaris/pull/3026 >> >> > > > > > > > > > > [4] - >> >> > > > > > > > >> https://lists.apache.org/thread/652z1f1n2pgf3g2ow5y382wlrtnoqth0 >> >> > > > > > > > > > > >> >> > > > > > > > > > > Go community, >> >> > > > > > > > > > > >> >> > > > > > > > > > > Adam >> >> > > > > > > > > > > >> >> > > > > > > > > > > On Mon, Nov 17, 2025 at 12:23 PM Dmitri >> Bourlatchkov < >> >> > > > > > > > [email protected] >> >> > > > > > > > > > >> >> > > > > > > > > > > wrote: >> >> > > > > > > > > > > >> >> > > > > > > > > > > > Hi Yun, >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > We should indeed review the status of the >> Generic Tables >> >> > > > API, >> >> > > > > > so >> >> > > > > > > > > thanks >> >> > > > > > > > > > > for >> >> > > > > > > > > > > > starting this discussion! >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > From my POV the key question is: how do we >> intend to >> >> > > > proceed >> >> > > > > > with >> >> > > > > > > > > known >> >> > > > > > > > > > > > limitations [1]? >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > I believe the Table Sources proposal [2] >> addresses some >> >> > > > (if not >> >> > > > > > > > all) >> >> > > > > > > > > of >> >> > > > > > > > > > > > those limitations. It is certainly suitable for >> the same >> >> > > > > > > > applications >> >> > > > > > > > > > > that >> >> > > > > > > > > > > > currently go through the Generic Tables API. >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > I believe it would be wise to allow the Table >> Sources >> >> > > > proposal >> >> > > > > > to >> >> > > > > > > > > > develop >> >> > > > > > > > > > > > further to see if there are any synergies that >> can be >> >> > > > leveraged >> >> > > > > > > > with >> >> > > > > > > > > > > > respect to Generic Tables. If we removed the >> "beta" label >> >> > > > from >> >> > > > > > > the >> >> > > > > > > > > > > Generic >> >> > > > > > > > > > > > Tables API now, it would make it harder for >> users to >> >> > > > benefit >> >> > > > > > from >> >> > > > > > > > > those >> >> > > > > > > > > > > > synergies later (due to virtually freezing the >> "spec"). >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > At it stands now, the Generic Tables API is >> supported by >> >> > > > > > Polaris, >> >> > > > > > > > so >> >> > > > > > > > > > > > existing clients can continue operating normally. >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > [1] >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > >> >> > > >> https://github.com/apache/polaris/blob/f056e22f7f3a7c53e233bef1b88d204d6a8e4d79/site/content/in-dev/unreleased/generic-table.md?plain=1#L162-L169 >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > [2] >> >> > > > > > > > >> https://lists.apache.org/thread/9f5b75fy25l9yzrtnlzqg6yh1bqdyjbt >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > Thanks, >> >> > > > > > > > > > > > Dmitri. >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > On Fri, Nov 14, 2025 at 12:33 PM yun zou < >> >> > > > > > > > [email protected] >> >> > > > > > > > > > >> >> > > > > > > > > > > > wrote: >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > > Hi All, >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > Generic Table has been available since Polaris >> 1.0 and >> >> > > > has >> >> > > > > > > > > attracted >> >> > > > > > > > > > > > > interest from several users. We also have >> ongoing >> >> > > > improvement >> >> > > > > > > and >> >> > > > > > > > > > > > extension >> >> > > > > > > > > > > > > work in progress, including Hudi support, >> Parquet >> >> > > > support, >> >> > > > > > and >> >> > > > > > > > > > > credential >> >> > > > > > > > > > > > > vending. >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > Given this progress, I believe it’s a good >> time to >> >> > > > remove the >> >> > > > > > > > > “beta” >> >> > > > > > > > > > > > label. >> >> > > > > > > > > > > > > If there are no objections, we will remove the >> “beta” >> >> > > > label >> >> > > > > > > from >> >> > > > > > > > > the >> >> > > > > > > > > > > > > Generic Table in Polaris 1.3. >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > Best Regards, >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > Yun >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > >> >> > > >> >
