Hi Dmitri, Thanks for the clarification!
-- if we happen to need a major Generic Tables API change after removing the "beta" label, I believe we'd have to make a v2 of that API Yes, if there are changes that require altering the high-level semantics of the API or modifying existing fields, we would need to move to a V2. However, since the Generic Table API currently defines only very basic fields, I believe the use cases described in the Table Source proposal can be addressed by extending the existing APIs. If it eventually becomes clear that a major change is required—such as a semantic-level shift—then we can transition to V2. The goal is to evolve and build on the current APIs wherever possible. Best Regards, Yun On Tue, Nov 18, 2025 at 8:32 AM Dmitri Bourlatchkov <[email protected]> wrote: > Hi Yun, > > I personally do not think the "beta" label is related to our commitment (or > lack of it) in maintaining the API. IMHO, it means that the API is likely > to undergo major changes. > > I did not and do not really oppose removing the "beta" label from the > Generic Tables API :) > > My suggestion in keeping it a bit longer was purely to allow more time > for finding common use cases with the Table Sources proposal and possibly > unifying some workflows. > > Having thought about that from that perspective some more, I actually agree > that the Generic Tables API is _not_ likely to undergo major changes. If > synergies with Table Sources can be found, most of the changes are probably > going to happen in clients that currently use the Generic Tables API, the > API itself is probably going to remain stable. > > That said, just for the sake of clarity, if we happen to need a major > Generic Tables API change after removing the "beta" label, I believe we'd > have to make a v2 of that API (which is ok from my POV per our evolution > guidelines [1]). > > [1] https://polaris.apache.org/in-dev/unreleased/evolution/ > > Cheers, > Dmitri. > > On Mon, Nov 17, 2025 at 8:20 PM yun zou <[email protected]> > wrote: > > > Hi Dmitri, > > > > Thanks a lot for the feedback! > > > > I definitely agree that the current generic table support still has > > limitations and that more work is needed. However, removing the “beta” > > label doesn’t mean the feature is finalized. Instead, it signals to users > > that we are committed to maintaining and improving it over time. > Therefore, > > I believe this will not block any future enhancements including extending > > it to address the use cases described in the Table Source proposal. > > > > As Adam pointed out, we already have users trying it out and requesting > > improvements. I believe this is a good time to remove the “beta” label to > > encourage broader adoption and welcome new contributors who can help > > advance the feature. > > > > Best Regards, > > Yun > > > > On Mon, Nov 17, 2025 at 3:28 PM Adam Christian < > > [email protected]> wrote: > > > > > I'm in favor of removing the "beta" label because it is starting to be > > used > > > by users. From the Slack feedback we have seen, there are users who > > > have started to try out this feature. While they are still running into > > > issues such as not having a Spark 4 plugin [1] and not having > credential > > > vending [2], there is real user usage indicating that folks find this > > > beneficial. > > > > > > Now, I do agree with Dmitri that there are known limitations we need to > > > handle that are impeding users. The two referenced examples above are > > > obvious ones that need fixes to get wider adoption. I have been working > > > with Yun & others to be able to solve some of these issues (like this > PR > > > [3]), but I agree that there may need to be more significant changes > > > including a REST API change. > > > > > > From my understanding, during our last conversation in the community > > about > > > the "Table Sources" proposal [4], we decided we were going to analyze > the > > > Parquet file use case and see if Generic Tables can support this use > case > > > or whether we need to bring some ideas from the Table Sources proposal > > into > > > Generic Tables. I believe that this approach is still valid and will > > > benefit our adopting users by identifying any enhancements with these > new > > > use cases. > > > > > > What do y'all think? > > > > > > [1] - https://github.com/apache/polaris/issues/3021 > > > [2] - https://github.com/apache/polaris/issues/3020 > > > [3] - https://github.com/apache/polaris/pull/3026 > > > [4] - https://lists.apache.org/thread/652z1f1n2pgf3g2ow5y382wlrtnoqth0 > > > > > > Go community, > > > > > > Adam > > > > > > On Mon, Nov 17, 2025 at 12:23 PM Dmitri Bourlatchkov <[email protected] > > > > > wrote: > > > > > > > Hi Yun, > > > > > > > > We should indeed review the status of the Generic Tables API, so > thanks > > > for > > > > starting this discussion! > > > > > > > > From my POV the key question is: how do we intend to proceed with > known > > > > limitations [1]? > > > > > > > > I believe the Table Sources proposal [2] addresses some (if not all) > of > > > > those limitations. It is certainly suitable for the same applications > > > that > > > > currently go through the Generic Tables API. > > > > > > > > I believe it would be wise to allow the Table Sources proposal to > > develop > > > > further to see if there are any synergies that can be leveraged with > > > > respect to Generic Tables. If we removed the "beta" label from the > > > Generic > > > > Tables API now, it would make it harder for users to benefit from > those > > > > synergies later (due to virtually freezing the "spec"). > > > > > > > > At it stands now, the Generic Tables API is supported by Polaris, so > > > > existing clients can continue operating normally. > > > > > > > > [1] > > > > > > > > > > > > > > https://github.com/apache/polaris/blob/f056e22f7f3a7c53e233bef1b88d204d6a8e4d79/site/content/in-dev/unreleased/generic-table.md?plain=1#L162-L169 > > > > > > > > [2] https://lists.apache.org/thread/9f5b75fy25l9yzrtnlzqg6yh1bqdyjbt > > > > > > > > Thanks, > > > > Dmitri. > > > > > > > > > > > > On Fri, Nov 14, 2025 at 12:33 PM yun zou <[email protected] > > > > > > wrote: > > > > > > > > > Hi All, > > > > > > > > > > Generic Table has been available since Polaris 1.0 and has > attracted > > > > > interest from several users. We also have ongoing improvement and > > > > extension > > > > > work in progress, including Hudi support, Parquet support, and > > > credential > > > > > vending. > > > > > > > > > > Given this progress, I believe it’s a good time to remove the > “beta” > > > > label. > > > > > If there are no objections, we will remove the “beta” label from > the > > > > > Generic Table in Polaris 1.3. > > > > > > > > > > Best Regards, > > > > > > > > > > Yun > > > > > > > > > > > > > > >
