Recording and AI Notes I don't pay for the enhanced notes... maybe I should
https://fathom.video/share/xDtaN_YsssoxcYZRGenrWzzCsj6J_NSr Apache Polaris - Catalog Federation Meeting - January 16 *VIEW RECORDING - 73 mins (No highlights)* <https://fathom.video/share/xDtaN_YsssoxcYZRGenrWzzCsj6J_NSr> Catalog Federation Overview @ 0:00 <https://fathom.video/share/xDtaN_YsssoxcYZRGenrWzzCsj6J_NSr?tab=summary×tamp=0> Dennis provided an overview of the broader scope and goals of catalog federation in Polaris. The key concepts introduced include: - Identity and roles resolution - Securables and access control - Execution of operations - The distinction between source of truth, facade, and implicit federation - Static vs. pass-through facades MVP Proposal for Iceberg REST Federation @ 2:10 <https://fathom.video/share/xDtaN_YsssoxcYZRGenrWzzCsj6J_NSr?tab=summary×tamp=130> For the initial MVP, Dennis proposed focusing on federating Iceberg REST catalogs, with the following key points: - Polaris will create static facade entities to represent remote Iceberg REST catalogs - These facades will be pass-through for namespaces, tables, and API operations (especially writes) - Polaris will handle identity and access control, using its own RBAC system - No caching will be implemented in the MVP, to keep the initial scope simple Concerns and Considerations @ 10:55 <https://fathom.video/share/xDtaN_YsssoxcYZRGenrWzzCsj6J_NSr?tab=summary×tamp=655> The group discussed several concerns and considerations around the federation proposal: - How to handle authentication and authorization, including potential security risks of token passing - The need to support a variety of catalog types beyond just Iceberg REST - Potential challenges with name collisions and cross-catalog views - The tradeoffs between a clean, extensible design and supporting legacy catalog integrations Next Steps @ 56:47 <https://fathom.video/share/xDtaN_YsssoxcYZRGenrWzzCsj6J_NSr?tab=summary×tamp=3407> The group agreed to continue the discussion on the mailing list, with a focus on: - Clarifying the scope and goals of the initial MVP - Addressing the key concerns raised around authentication, authorization, and supporting diverse catalog types - Exploring options for a more unified "virtual catalog" concept in Polaris On Wed, Jan 15, 2025 at 3:12 PM Dennis Huo <huoi...@gmail.com> wrote: > In case anyone wasn't on the googlegroups list but wanted to join the > meeting tomorrow, here's the meeting info for tomorrow: > > Apache Polaris - Catalog Federation Meeting > Thursday, January 16 · 9:00 – 10:00am > Time zone: America/Los_Angeles > Google Meet joining info > Video call link: https://meet.google.com/ryj-mueq-csf > > Thanks JB for setting it up! > > > On Mon, Jan 13, 2025 at 11:15 AM Russell Spitzer < > russell.spit...@gmail.com> > wrote: > > > +1 Works for me (meeting on thursday) > > > > On Sun, Jan 12, 2025 at 8:56 PM Dennis Huo <huoi...@gmail.com> wrote: > > > > > Ah so sorry, I meant to include the proposed date of Thursday, January > > 16th > > > for the meeting. > > > > > > I think it's definitely worth including the relationship to > > > notifications/events in the discussion, though tentatively my thinking > so > > > far is that while notifications can be supplemental to the federation > > > implementation, it should be possible to initially proceed > independently, > > > and there should still be a reasonable federation experience without > > > notifications as well. > > > > > > On Sat, Jan 11, 2025 at 5:14 AM Alex Dutra > <alex.du...@dremio.com.invalid > > > > > > wrote: > > > > > > > Hi Dennis, > > > > > > > > Thanks for starting this discussion! > > > > > > > > The proposed time slot is generally OK for me, but what weekday do > you > > > > have in mind? I wouldn't be able to attend on some weekdays. > > > > > > > > Other than that, I was wondering how much of the Catalog Federation > > > > work would have to be built on top of a yet-to-be-created > notification > > > > / event system. This topic has been mentioned already in 2 Polaris > dev > > > > mailing list threads: > > > > > > > > https://lists.apache.org/thread/48fczg6okw9mos94o90tr347fbz9qc3b > > > > https://lists.apache.org/thread/03yz5wolkvy8l7rbcwjnqdq1bl8p065v > > > > > > > > Besides, you also sparked a very interesting discussion on the > Iceberg > > > > dev mailing list a while ago where these 2 topics were intertwined: > > > > > > > > https://lists.apache.org/thread/zcv6qm9ysknrhfpg093qgnrkrolptcht > > > > > > > > So, maybe one of the first items that we need to discuss here is the > > > > notification mechanism to use to synchronize between catalogs. Wdyt? > > > > > > > > Thanks, > > > > > > > > Alex > > > > > > > > > > > > On Fri, Jan 10, 2025 at 3:36 AM Dennis Huo <huoi...@gmail.com> > wrote: > > > > > > > > > > Hello! > > > > > > > > > > I wanted to kick off next steps to drive forward the design > > discussion > > > > for > > > > > Polaris Catalog Federation: > > > > > > > > > > > > > > > https://docs.google.com/document/d/1Q6eEytxb0btpOPcL8RtkULskOlYUCo_3FLvFRnHkzBY/edit?tab=t.0 > > > > > > > > > > In some of the prior community syncs I think folks expressed a > desire > > > to > > > > > have some dedicated discussion time on the linked proposal and > other > > > > > related considerations. Tentatively, would 9AM Pacific (GMT-8) (6PM > > CET > > > > > GMT+1) work for any folks who are interested in participating? > > > > > > > > > > I'd also like to use this thread to preemptively gather key areas > > folks > > > > > want to cover in the discussion so that we can make the best use of > > > > > everyone's time. Please feel free to respond with any suggestions > on > > > open > > > > > design questions, missing areas of consideration, etc., you'd like > to > > > > cover. > > > > > > > > > > Otherwise, any input regarding timing of this design meeting, a > > simple > > > +1 > > > > > to indicate interest in participating, or even suggestions for a > > > > different > > > > > time would be much appreciated so we can ensure an inclusive quorum > > of > > > > > interested parties. > > > > > > > > > > Cheers, > > > > > Dennis > > > > > > > > > >
