Hi folks, Following the Polaris Table Source Proposal Community Meeting on October 2nd, we conducted extensive investigations based on the valuable feedback provided.
We are now working on an updated proposal that consolidates existing features, current proposals, and the feedback received. This update will include: - Clarification of the proposal's purpose and specific use cases. - Identification of which Polaris values this proposal focuses on (interoperability at core). - An analysis of leveraging a Generic Table approach, including pros and cons. - A more detailed section for the Object Table proposal, with more comparison with similar proposals (like the Volume proposal). We also identified some missing components, particularly regarding general credential vending. To move forward, we propose the following next steps: 1. Update the proposal and share the revised version here for review (in the coming days). 2. Start developing a concrete prototype to better illustrate the proposal's concepts. 3. Schedule a new Table Source meeting to discuss the latest version of the proposal in depth. Thank you for your input! Regards, JB On Fri, Oct 3, 2025 at 12:11 AM Jean-Baptiste Onofré <[email protected]> wrote: > > Hi folks, > > Thanks to everyone who joined the community meeting about Table Source > Proposal. > > Here's the main takeaways: > 1. I think we have a consensus about the use case about reading > existing Parquet files to easily create Iceberg metadata and so > leverage Polaris features (especially about governance). > 2. We don't have yet a clear consensus for "existing Iceberg tables" > (that can be addressed with Catalog Federation) and unstructured data > (PDF files, video, image, ...) needs more discussion. > 3. In order to move forward, I propose to focus on the "existing > Parquet files" use case. > 4. Then, I'm proposing the following action plan: > 4.1. I propose to split the Table Source proposal document, with a > focus on the "Parquet file" use case. > 4.2. We discussed leveraging Generic Table and server side scan API > for that. I propose to work with Yun and I will start a PoC to verify > it's a viable option and identify the changes eventually required on > Generic Table. > 4.3. Depending of 4.2, I will update the proposal document about > "existing Parquet files" and open a PR with change. > > Thoughts ? > > Here's the record: > https://drive.google.com/file/d/1x4XjZCop7WaA8L0m81UrepE2nvRsfUU1/view?usp=sharing > > I will submit a PR to update the website too and I will update the > corresponding GitHub Issue and proposal document. > > Thanks again! > > Regards > JB
