Hi Folks, We’ve made some adjustments to the design, moving from *Option 1* to *Option 2*:
1. *Option 1:* Store maintenance policies in catalog/namespace/table properties. 2. *Option 2:* Store maintenance policies as separate entities. The key concern with Option 1 is that the access control model isn't scalable. On the other hand, Option 2 provides greater flexibility, improved privilege enforcement, and better overall performance. I’ve updated the design document with the latest changes, which you can find here: Updated Design Document <https://docs.google.com/document/d/1Pd_mzZcfvnUvcH98IbwsIYf4eryet1lQDfclKYx-t-M/edit?usp=sharing> . To discuss this design change in detail, I’ll be hosting a session on Thursday. Please find the meeting details below: Table maintenance in Polaris @ Thu, Nov 7, 2024 9:00am – 10:00am (GMT-08) Thursday, December 5 · 9:00 – 10:00am Time zone: America/Los_Angeles Google Meet joining info Video call link: https://meet.google.com/dix-cdfm-pve Feel free to review the updated document ahead of the session. Looking forward to your thoughts and feedback during the meeting! Yufei On Mon, Nov 18, 2024 at 9:43 PM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > Hi Yufei > > Not sure we got consensus in all details but the overall picture is ok for > me. > > Let’s continue the details definition in the PR. > > Thanks ! > Regards > JB > > Le jeu. 14 nov. 2024 à 02:39, Yufei Gu <flyrain...@gmail.com> a écrit : > > > Hi everyone, > > > > > > Thank you for joining the table maintenance discussion today! We made > > significant progress, and here are the key takeaways: > > > > 1. Clarified furthermore and reached consensus on introducing table > > maintenance properties in Polaris to support for different TMS and > > promote > > interoperability. > > 2. Agreed to proceed with Option 1, which stores metadata as > > catalog/namespace/table properties. > > 3. Confirmed the new privileges to ensure that maintenance properties > > are safeguarded from being altered by clients with existing write > > access. > > 4. Briefly discussed the support for customized maintenance policies . > > > > Next step: > > > > 1. Will file maintenance properties related PRs per design > > 2. Will add more details for customized policy support. > > > > *Note*: Unfortunately, I wasn’t able to record the meeting due to the > need > > for a Google premium account. > > > > > > Yufei > > > > > > On Tue, Nov 12, 2024 at 10:10 AM Omar Al-Safi <o...@oalsafi.com> wrote: > > > > > Thank you! Will try to be there > > > > > > On Tue, 12 Nov 2024, 18:55 Yufei Gu, <flyrain...@gmail.com> wrote: > > > > > > > Hi Omar, I sent the invitation to dev@polaris.apache.org, as well as > > > your > > > > email address. > > > > > > > > Yufei > > > > > > > > > > > > On Tue, Nov 12, 2024 at 9:51 AM Omar Al-Safi <o...@oalsafi.com> > wrote: > > > > > > > > > Thanks Yufei, is it possible to send the invitation to the > > > > > Polaris google group? > > > > > > > > > > Regards, > > > > > Omar > > > > > > > > > > On Tue, Nov 12, 2024 at 6:48 PM Yufei Gu <flyrain...@gmail.com> > > wrote: > > > > > > > > > > > Hi folks, > > > > > > > > > > > > We are going to have another sync for table maintenance in > Polaris > > > per > > > > > > discussion with JB. Here are meeting details: > > > > > > > > > > > > Polaris Table maintenance sync > > > > > > Wednesday, November 13 · 10:00 – 11:00am > > > > > > Time zone: America/Los_Angeles > > > > > > Google Meet joining info > > > > > > Video call link: https://meet.google.com/nyy-ahmn-jqd > > > > > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > On Fri, Nov 8, 2024 at 5:23 PM Yufei Gu <flyrain...@gmail.com> > > > wrote: > > > > > > > > > > > > > Thanks everyone for joining the discussion. Sorry I couldn't > > record > > > > the > > > > > > > session due to a tech issue. Here are meeting notes: > > > > > > > > > > > > > > 1. We discussed the boundary between Polaris and the Table > > > > > Maintenance > > > > > > > System(TMS). We agreed that they should be separated > systems. > > > > > > > 2. A general agreement on the minimal metadata added to > > Polaris > > > to > > > > > > > support TMS, focusing on essential data needed for > > > > interoperability. > > > > > > > 3. A general consensus on option 1 to store metadata as > > > > > > > catalog/namespace/table properties. We could introduce > policy > > > > > > entities in > > > > > > > the future for other use cases, like column masking. Will > > > address > > > > > two > > > > > > > feedbacks: > > > > > > > 1. Caching the table properties in the catalog to reduce > IO > > > > cost. > > > > > > > 2. Introducing new permissions for table maintenance > > related > > > > > > > metadata to prevent any clients with the write permission > > to > > > > mess > > > > > > up with > > > > > > > them. > > > > > > > 4. Briefly touched on the communication module between TMS > and > > > > > > > Polaris, as a long-term plan, an event system from Polaris > is > > > > > > necessary, > > > > > > > not only benefits TMS, but also benefits other systems which > > > > consume > > > > > > change > > > > > > > from Polaris. > > > > > > > > > > > > > > Next Steps: > > > > > > > > > > > > > > 1. Implement metadata storage as properties > > > > > > > 1. Design detailed schema for properties > > > > > > > 2. Figure out a way to be extensible for future > maintenance > > > > > policy > > > > > > > or customized policies. > > > > > > > 3. Add new permissions for new properties > > > > > > > 2. Begin planning for event system > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 5, 2024 at 12:25 AM Jean-Baptiste Onofré < > > > > j...@nanthrax.net> > > > > > > > wrote: > > > > > > > > > > > > > >> Hi Yufei > > > > > > >> > > > > > > >> Thanks for scheduling this ! > > > > > > >> > > > > > > >> I should be able to join. > > > > > > >> > > > > > > >> For the community, will you be able to record ? > > > > > > >> > > > > > > >> Regards > > > > > > >> JB > > > > > > >> > > > > > > >> On Mon, Nov 4, 2024 at 10:40 PM Yufei Gu < > flyrain...@gmail.com> > > > > > wrote: > > > > > > >> > > > > > > > >> > Hi Folks, > > > > > > >> > > > > > > > >> > I've scheduled a community sync to discuss table maintenance > > in > > > > > > Polaris > > > > > > >> > this Thursday at 9 AM PST. Since we didn’t have a chance to > > dive > > > > > into > > > > > > >> this > > > > > > >> > topic during our last sync, this will be a dedicated session > > to > > > > > cover > > > > > > >> it in > > > > > > >> > detail. > > > > > > >> > > > > > > > >> > *Updates to Note:* I've made some updates to the design > > > document, > > > > > > with a > > > > > > >> > particular focus on the approach for maintenance metadata. > The > > > > > > document > > > > > > >> now > > > > > > >> > favors *Option 1*, which involves leveraging table, > namespace, > > > and > > > > > > >> catalog > > > > > > >> > properties for maintenance metadata. > > > > > > >> > > > > > > > >> > Please review the latest version of the design doc before > the > > > > > meeting, > > > > > > >> as > > > > > > >> > it will help us streamline the discussion. > > > > > > >> > > > > > > > >> > Looking forward to everyone’s insights! > > > > > > >> > Video call link: https://meet.google.com/opc-vath-mgb > > > > > > >> > Design doc: > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1Pd_mzZcfvnUvcH98IbwsIYf4eryet1lQDfclKYx-t-M/edit?usp=sharing > > > > > > >> > < > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://www.google.com/url?q=https://docs.google.com/document/d/1Pd_mzZcfvnUvcH98IbwsIYf4eryet1lQDfclKYx-t-M/edit?usp%3Dsharing&sa=D&source=calendar&usd=2&usg=AOvVaw2V3IjIcadea8miDcKKSG9I > > > > > > >> > > > > > > > >> > > > > > > > >> > Yufei > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
