Thanks Yufei, I think you meant Thursday, December 5th 9:00am – 10:00am (GMT-08).
Anurag Mantripragada > On Dec 3, 2024, at 11:33 AM, Yufei Gu <flyrain...@gmail.com> wrote: > > Hi Folks, > > We’ve made some adjustments to the design, moving from *Option 1* to *Option > 2*: > > 1. *Option 1:* Store maintenance policies in catalog/namespace/table > properties. > 2. *Option 2:* Store maintenance policies as separate entities. > > The key concern with Option 1 is that the access control model isn't > scalable. On the other hand, Option 2 provides greater flexibility, > improved privilege enforcement, and better overall performance. > > I’ve updated the design document with the latest changes, which you can > find here: Updated Design Document > <https://docs.google.com/document/d/1Pd_mzZcfvnUvcH98IbwsIYf4eryet1lQDfclKYx-t-M/edit?usp=sharing> > . > > To discuss this design change in detail, I’ll be hosting a session on > Thursday. Please find the meeting details below: > Table maintenance in Polaris @ Thu, Nov 7, 2024 9:00am – 10:00am (GMT-08) > Thursday, December 5 · 9:00 – 10:00am > Time zone: America/Los_Angeles > Google Meet joining info > Video call link: https://meet.google.com/dix-cdfm-pve > > Feel free to review the updated document ahead of the session. Looking > forward to your thoughts and feedback during the meeting! > > Yufei > > > On Mon, Nov 18, 2024 at 9:43 PM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > >> Hi Yufei >> >> Not sure we got consensus in all details but the overall picture is ok for >> me. >> >> Let’s continue the details definition in the PR. >> >> Thanks ! >> Regards >> JB >> >> Le jeu. 14 nov. 2024 à 02:39, Yufei Gu <flyrain...@gmail.com> a écrit : >> >>> Hi everyone, >>> >>> >>> Thank you for joining the table maintenance discussion today! We made >>> significant progress, and here are the key takeaways: >>> >>> 1. Clarified furthermore and reached consensus on introducing table >>> maintenance properties in Polaris to support for different TMS and >>> promote >>> interoperability. >>> 2. Agreed to proceed with Option 1, which stores metadata as >>> catalog/namespace/table properties. >>> 3. Confirmed the new privileges to ensure that maintenance properties >>> are safeguarded from being altered by clients with existing write >>> access. >>> 4. Briefly discussed the support for customized maintenance policies . >>> >>> Next step: >>> >>> 1. Will file maintenance properties related PRs per design >>> 2. Will add more details for customized policy support. >>> >>> *Note*: Unfortunately, I wasn’t able to record the meeting due to the >> need >>> for a Google premium account. >>> >>> >>> Yufei >>> >>> >>> On Tue, Nov 12, 2024 at 10:10 AM Omar Al-Safi <o...@oalsafi.com> wrote: >>> >>>> Thank you! Will try to be there >>>> >>>> On Tue, 12 Nov 2024, 18:55 Yufei Gu, <flyrain...@gmail.com> wrote: >>>> >>>>> Hi Omar, I sent the invitation to dev@polaris.apache.org, as well as >>>> your >>>>> email address. >>>>> >>>>> Yufei >>>>> >>>>> >>>>> On Tue, Nov 12, 2024 at 9:51 AM Omar Al-Safi <o...@oalsafi.com> >> wrote: >>>>> >>>>>> Thanks Yufei, is it possible to send the invitation to the >>>>>> Polaris google group? >>>>>> >>>>>> Regards, >>>>>> Omar >>>>>> >>>>>> On Tue, Nov 12, 2024 at 6:48 PM Yufei Gu <flyrain...@gmail.com> >>> wrote: >>>>>> >>>>>>> Hi folks, >>>>>>> >>>>>>> We are going to have another sync for table maintenance in >> Polaris >>>> per >>>>>>> discussion with JB. Here are meeting details: >>>>>>> >>>>>>> Polaris Table maintenance sync >>>>>>> Wednesday, November 13 · 10:00 – 11:00am >>>>>>> Time zone: America/Los_Angeles >>>>>>> Google Meet joining info >>>>>>> Video call link: https://meet.google.com/nyy-ahmn-jqd >>>>>>> >>>>>>> >>>>>>> Yufei >>>>>>> >>>>>>> >>>>>>> On Fri, Nov 8, 2024 at 5:23 PM Yufei Gu <flyrain...@gmail.com> >>>> wrote: >>>>>>> >>>>>>>> Thanks everyone for joining the discussion. Sorry I couldn't >>> record >>>>> the >>>>>>>> session due to a tech issue. Here are meeting notes: >>>>>>>> >>>>>>>> 1. We discussed the boundary between Polaris and the Table >>>>>> Maintenance >>>>>>>> System(TMS). We agreed that they should be separated >> systems. >>>>>>>> 2. A general agreement on the minimal metadata added to >>> Polaris >>>> to >>>>>>>> support TMS, focusing on essential data needed for >>>>> interoperability. >>>>>>>> 3. A general consensus on option 1 to store metadata as >>>>>>>> catalog/namespace/table properties. We could introduce >> policy >>>>>>> entities in >>>>>>>> the future for other use cases, like column masking. Will >>>> address >>>>>> two >>>>>>>> feedbacks: >>>>>>>> 1. Caching the table properties in the catalog to reduce >> IO >>>>> cost. >>>>>>>> 2. Introducing new permissions for table maintenance >>> related >>>>>>>> metadata to prevent any clients with the write permission >>> to >>>>> mess >>>>>>> up with >>>>>>>> them. >>>>>>>> 4. Briefly touched on the communication module between TMS >> and >>>>>>>> Polaris, as a long-term plan, an event system from Polaris >> is >>>>>>> necessary, >>>>>>>> not only benefits TMS, but also benefits other systems which >>>>> consume >>>>>>> change >>>>>>>> from Polaris. >>>>>>>> >>>>>>>> Next Steps: >>>>>>>> >>>>>>>> 1. Implement metadata storage as properties >>>>>>>> 1. Design detailed schema for properties >>>>>>>> 2. Figure out a way to be extensible for future >> maintenance >>>>>> policy >>>>>>>> or customized policies. >>>>>>>> 3. Add new permissions for new properties >>>>>>>> 2. Begin planning for event system >>>>>>>> >>>>>>>> Yufei >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Nov 5, 2024 at 12:25 AM Jean-Baptiste Onofré < >>>>> j...@nanthrax.net> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Yufei >>>>>>>>> >>>>>>>>> Thanks for scheduling this ! >>>>>>>>> >>>>>>>>> I should be able to join. >>>>>>>>> >>>>>>>>> For the community, will you be able to record ? >>>>>>>>> >>>>>>>>> Regards >>>>>>>>> JB >>>>>>>>> >>>>>>>>> On Mon, Nov 4, 2024 at 10:40 PM Yufei Gu < >> flyrain...@gmail.com> >>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Hi Folks, >>>>>>>>>> >>>>>>>>>> I've scheduled a community sync to discuss table maintenance >>> in >>>>>>> Polaris >>>>>>>>>> this Thursday at 9 AM PST. Since we didn’t have a chance to >>> dive >>>>>> into >>>>>>>>> this >>>>>>>>>> topic during our last sync, this will be a dedicated session >>> to >>>>>> cover >>>>>>>>> it in >>>>>>>>>> detail. >>>>>>>>>> >>>>>>>>>> *Updates to Note:* I've made some updates to the design >>>> document, >>>>>>> with a >>>>>>>>>> particular focus on the approach for maintenance metadata. >> The >>>>>>> document >>>>>>>>> now >>>>>>>>>> favors *Option 1*, which involves leveraging table, >> namespace, >>>> and >>>>>>>>> catalog >>>>>>>>>> properties for maintenance metadata. >>>>>>>>>> >>>>>>>>>> Please review the latest version of the design doc before >> the >>>>>> meeting, >>>>>>>>> as >>>>>>>>>> it will help us streamline the discussion. >>>>>>>>>> >>>>>>>>>> Looking forward to everyone’s insights! >>>>>>>>>> Video call link: https://meet.google.com/opc-vath-mgb >>>>>>>>>> Design doc: >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> https://docs.google.com/document/d/1Pd_mzZcfvnUvcH98IbwsIYf4eryet1lQDfclKYx-t-M/edit?usp=sharing >>>>>>>>>> < >>>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> https://www.google.com/url?q=https://docs.google.com/document/d/1Pd_mzZcfvnUvcH98IbwsIYf4eryet1lQDfclKYx-t-M/edit?usp%3Dsharing&sa=D&source=calendar&usd=2&usg=AOvVaw2V3IjIcadea8miDcKKSG9I >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Yufei >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>
