Hi Yun, I am personally ok with the approach you propose. If you would like to remove the beta label in 1.3.0, please open a PR to unblock the release as discussed in [1]. If you prefer to remove the label later, please comment on [1].
[1] https://lists.apache.org/thread/dhzop6tddl8f9ygbbgdoqywk0hwzsgb2 Thanks, Dmitri. On Tue, Nov 18, 2025 at 9:47 PM yun zou <[email protected]> wrote: > Hi Dmitri, > > Thanks for the clarification! > > -- if we happen to need a major Generic Tables API change after removing > the "beta" label, I believe we'd have to make a v2 of that API > > Yes, if there are changes that require altering the high-level semantics of > the API or modifying existing fields, we would need to move to a V2. > > However, since the Generic Table API currently defines only very basic > fields, I believe the use cases described in the Table Source proposal can > be addressed by extending the existing APIs. If it eventually becomes clear > that a major change is required—such as a semantic-level shift—then we can > transition to V2. The goal is to evolve and build on the current APIs > wherever possible. > > Best Regards, > Yun > > > > On Tue, Nov 18, 2025 at 8:32 AM Dmitri Bourlatchkov <[email protected]> > wrote: > > > Hi Yun, > > > > I personally do not think the "beta" label is related to our commitment > (or > > lack of it) in maintaining the API. IMHO, it means that the API is likely > > to undergo major changes. > > > > I did not and do not really oppose removing the "beta" label from the > > Generic Tables API :) > > > > My suggestion in keeping it a bit longer was purely to allow more time > > for finding common use cases with the Table Sources proposal and possibly > > unifying some workflows. > > > > Having thought about that from that perspective some more, I actually > agree > > that the Generic Tables API is _not_ likely to undergo major changes. If > > synergies with Table Sources can be found, most of the changes are > probably > > going to happen in clients that currently use the Generic Tables API, the > > API itself is probably going to remain stable. > > > > That said, just for the sake of clarity, if we happen to need a major > > Generic Tables API change after removing the "beta" label, I believe we'd > > have to make a v2 of that API (which is ok from my POV per our evolution > > guidelines [1]). > > > > [1] https://polaris.apache.org/in-dev/unreleased/evolution/ > > > > Cheers, > > Dmitri. > > > > On Mon, Nov 17, 2025 at 8:20 PM yun zou <[email protected]> > > wrote: > > > > > Hi Dmitri, > > > > > > Thanks a lot for the feedback! > > > > > > I definitely agree that the current generic table support still has > > > limitations and that more work is needed. However, removing the “beta” > > > label doesn’t mean the feature is finalized. Instead, it signals to > users > > > that we are committed to maintaining and improving it over time. > > Therefore, > > > I believe this will not block any future enhancements including > extending > > > it to address the use cases described in the Table Source proposal. > > > > > > As Adam pointed out, we already have users trying it out and requesting > > > improvements. I believe this is a good time to remove the “beta” label > to > > > encourage broader adoption and welcome new contributors who can help > > > advance the feature. > > > > > > Best Regards, > > > Yun > > > > > > On Mon, Nov 17, 2025 at 3:28 PM Adam Christian < > > > [email protected]> wrote: > > > > > > > I'm in favor of removing the "beta" label because it is starting to > be > > > used > > > > by users. From the Slack feedback we have seen, there are users who > > > > have started to try out this feature. While they are still running > into > > > > issues such as not having a Spark 4 plugin [1] and not having > > credential > > > > vending [2], there is real user usage indicating that folks find this > > > > beneficial. > > > > > > > > Now, I do agree with Dmitri that there are known limitations we need > to > > > > handle that are impeding users. The two referenced examples above are > > > > obvious ones that need fixes to get wider adoption. I have been > working > > > > with Yun & others to be able to solve some of these issues (like this > > PR > > > > [3]), but I agree that there may need to be more significant changes > > > > including a REST API change. > > > > > > > > From my understanding, during our last conversation in the community > > > about > > > > the "Table Sources" proposal [4], we decided we were going to analyze > > the > > > > Parquet file use case and see if Generic Tables can support this use > > case > > > > or whether we need to bring some ideas from the Table Sources > proposal > > > into > > > > Generic Tables. I believe that this approach is still valid and will > > > > benefit our adopting users by identifying any enhancements with these > > new > > > > use cases. > > > > > > > > What do y'all think? > > > > > > > > [1] - https://github.com/apache/polaris/issues/3021 > > > > [2] - https://github.com/apache/polaris/issues/3020 > > > > [3] - https://github.com/apache/polaris/pull/3026 > > > > [4] - > https://lists.apache.org/thread/652z1f1n2pgf3g2ow5y382wlrtnoqth0 > > > > > > > > Go community, > > > > > > > > Adam > > > > > > > > On Mon, Nov 17, 2025 at 12:23 PM Dmitri Bourlatchkov < > [email protected] > > > > > > > wrote: > > > > > > > > > Hi Yun, > > > > > > > > > > We should indeed review the status of the Generic Tables API, so > > thanks > > > > for > > > > > starting this discussion! > > > > > > > > > > From my POV the key question is: how do we intend to proceed with > > known > > > > > limitations [1]? > > > > > > > > > > I believe the Table Sources proposal [2] addresses some (if not > all) > > of > > > > > those limitations. It is certainly suitable for the same > applications > > > > that > > > > > currently go through the Generic Tables API. > > > > > > > > > > I believe it would be wise to allow the Table Sources proposal to > > > develop > > > > > further to see if there are any synergies that can be leveraged > with > > > > > respect to Generic Tables. If we removed the "beta" label from the > > > > Generic > > > > > Tables API now, it would make it harder for users to benefit from > > those > > > > > synergies later (due to virtually freezing the "spec"). > > > > > > > > > > At it stands now, the Generic Tables API is supported by Polaris, > so > > > > > existing clients can continue operating normally. > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > https://github.com/apache/polaris/blob/f056e22f7f3a7c53e233bef1b88d204d6a8e4d79/site/content/in-dev/unreleased/generic-table.md?plain=1#L162-L169 > > > > > > > > > > [2] > https://lists.apache.org/thread/9f5b75fy25l9yzrtnlzqg6yh1bqdyjbt > > > > > > > > > > Thanks, > > > > > Dmitri. > > > > > > > > > > > > > > > On Fri, Nov 14, 2025 at 12:33 PM yun zou < > [email protected] > > > > > > > > wrote: > > > > > > > > > > > Hi All, > > > > > > > > > > > > Generic Table has been available since Polaris 1.0 and has > > attracted > > > > > > interest from several users. We also have ongoing improvement and > > > > > extension > > > > > > work in progress, including Hudi support, Parquet support, and > > > > credential > > > > > > vending. > > > > > > > > > > > > Given this progress, I believe it’s a good time to remove the > > “beta” > > > > > label. > > > > > > If there are no objections, we will remove the “beta” label from > > the > > > > > > Generic Table in Polaris 1.3. > > > > > > > > > > > > Best Regards, > > > > > > > > > > > > Yun > > > > > > > > > > > > > > > > > > > > >
