Sorry the meeting title is misleading, the meeting itself is scheduled on Dec. 5th. Thanks Anurag for pointing that out.
Table maintenance in Polaris Thursday, December 5 · 9:00 – 10:00am Time zone: America/Los_Angeles Google Meet joining info Video call link: https://meet.google.com/dix-cdfm-pve Yufei On Tue, Dec 3, 2024 at 12:32 PM Anurag Mantripragada <amantriprag...@apple.com.invalid> wrote: > Thanks Yufei, I think you meant Thursday, December 5th 9:00am – 10:00am > (GMT-08). > > > Anurag Mantripragada > > > > On Dec 3, 2024, at 11:33 AM, Yufei Gu <flyrain...@gmail.com> wrote: > > > > Hi Folks, > > > > We’ve made some adjustments to the design, moving from *Option 1* to > *Option > > 2*: > > > > 1. *Option 1:* Store maintenance policies in catalog/namespace/table > > properties. > > 2. *Option 2:* Store maintenance policies as separate entities. > > > > The key concern with Option 1 is that the access control model isn't > > scalable. On the other hand, Option 2 provides greater flexibility, > > improved privilege enforcement, and better overall performance. > > > > I’ve updated the design document with the latest changes, which you can > > find here: Updated Design Document > > < > https://docs.google.com/document/d/1Pd_mzZcfvnUvcH98IbwsIYf4eryet1lQDfclKYx-t-M/edit?usp=sharing > > > > . > > > > To discuss this design change in detail, I’ll be hosting a session on > > Thursday. Please find the meeting details below: > > Table maintenance in Polaris @ Thu, Nov 7, 2024 9:00am – 10:00am (GMT-08) > > Thursday, December 5 · 9:00 – 10:00am > > Time zone: America/Los_Angeles > > Google Meet joining info > > Video call link: https://meet.google.com/dix-cdfm-pve > > > > Feel free to review the updated document ahead of the session. Looking > > forward to your thoughts and feedback during the meeting! > > > > Yufei > > > > > > On Mon, Nov 18, 2024 at 9:43 PM Jean-Baptiste Onofré <j...@nanthrax.net> > > wrote: > > > >> Hi Yufei > >> > >> Not sure we got consensus in all details but the overall picture is ok > for > >> me. > >> > >> Let’s continue the details definition in the PR. > >> > >> Thanks ! > >> Regards > >> JB > >> > >> Le jeu. 14 nov. 2024 à 02:39, Yufei Gu <flyrain...@gmail.com> a écrit : > >> > >>> Hi everyone, > >>> > >>> > >>> Thank you for joining the table maintenance discussion today! We made > >>> significant progress, and here are the key takeaways: > >>> > >>> 1. Clarified furthermore and reached consensus on introducing table > >>> maintenance properties in Polaris to support for different TMS and > >>> promote > >>> interoperability. > >>> 2. Agreed to proceed with Option 1, which stores metadata as > >>> catalog/namespace/table properties. > >>> 3. Confirmed the new privileges to ensure that maintenance properties > >>> are safeguarded from being altered by clients with existing write > >>> access. > >>> 4. Briefly discussed the support for customized maintenance policies > . > >>> > >>> Next step: > >>> > >>> 1. Will file maintenance properties related PRs per design > >>> 2. Will add more details for customized policy support. > >>> > >>> *Note*: Unfortunately, I wasn’t able to record the meeting due to the > >> need > >>> for a Google premium account. > >>> > >>> > >>> Yufei > >>> > >>> > >>> On Tue, Nov 12, 2024 at 10:10 AM Omar Al-Safi <o...@oalsafi.com> > wrote: > >>> > >>>> Thank you! Will try to be there > >>>> > >>>> On Tue, 12 Nov 2024, 18:55 Yufei Gu, <flyrain...@gmail.com> wrote: > >>>> > >>>>> Hi Omar, I sent the invitation to dev@polaris.apache.org, as well as > >>>> your > >>>>> email address. > >>>>> > >>>>> Yufei > >>>>> > >>>>> > >>>>> On Tue, Nov 12, 2024 at 9:51 AM Omar Al-Safi <o...@oalsafi.com> > >> wrote: > >>>>> > >>>>>> Thanks Yufei, is it possible to send the invitation to the > >>>>>> Polaris google group? > >>>>>> > >>>>>> Regards, > >>>>>> Omar > >>>>>> > >>>>>> On Tue, Nov 12, 2024 at 6:48 PM Yufei Gu <flyrain...@gmail.com> > >>> wrote: > >>>>>> > >>>>>>> Hi folks, > >>>>>>> > >>>>>>> We are going to have another sync for table maintenance in > >> Polaris > >>>> per > >>>>>>> discussion with JB. Here are meeting details: > >>>>>>> > >>>>>>> Polaris Table maintenance sync > >>>>>>> Wednesday, November 13 · 10:00 – 11:00am > >>>>>>> Time zone: America/Los_Angeles > >>>>>>> Google Meet joining info > >>>>>>> Video call link: https://meet.google.com/nyy-ahmn-jqd > >>>>>>> > >>>>>>> > >>>>>>> Yufei > >>>>>>> > >>>>>>> > >>>>>>> On Fri, Nov 8, 2024 at 5:23 PM Yufei Gu <flyrain...@gmail.com> > >>>> wrote: > >>>>>>> > >>>>>>>> Thanks everyone for joining the discussion. Sorry I couldn't > >>> record > >>>>> the > >>>>>>>> session due to a tech issue. Here are meeting notes: > >>>>>>>> > >>>>>>>> 1. We discussed the boundary between Polaris and the Table > >>>>>> Maintenance > >>>>>>>> System(TMS). We agreed that they should be separated > >> systems. > >>>>>>>> 2. A general agreement on the minimal metadata added to > >>> Polaris > >>>> to > >>>>>>>> support TMS, focusing on essential data needed for > >>>>> interoperability. > >>>>>>>> 3. A general consensus on option 1 to store metadata as > >>>>>>>> catalog/namespace/table properties. We could introduce > >> policy > >>>>>>> entities in > >>>>>>>> the future for other use cases, like column masking. Will > >>>> address > >>>>>> two > >>>>>>>> feedbacks: > >>>>>>>> 1. Caching the table properties in the catalog to reduce > >> IO > >>>>> cost. > >>>>>>>> 2. Introducing new permissions for table maintenance > >>> related > >>>>>>>> metadata to prevent any clients with the write permission > >>> to > >>>>> mess > >>>>>>> up with > >>>>>>>> them. > >>>>>>>> 4. Briefly touched on the communication module between TMS > >> and > >>>>>>>> Polaris, as a long-term plan, an event system from Polaris > >> is > >>>>>>> necessary, > >>>>>>>> not only benefits TMS, but also benefits other systems which > >>>>> consume > >>>>>>> change > >>>>>>>> from Polaris. > >>>>>>>> > >>>>>>>> Next Steps: > >>>>>>>> > >>>>>>>> 1. Implement metadata storage as properties > >>>>>>>> 1. Design detailed schema for properties > >>>>>>>> 2. Figure out a way to be extensible for future > >> maintenance > >>>>>> policy > >>>>>>>> or customized policies. > >>>>>>>> 3. Add new permissions for new properties > >>>>>>>> 2. Begin planning for event system > >>>>>>>> > >>>>>>>> Yufei > >>>>>>>> > >>>>>>>> > >>>>>>>> On Tue, Nov 5, 2024 at 12:25 AM Jean-Baptiste Onofré < > >>>>> j...@nanthrax.net> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> Hi Yufei > >>>>>>>>> > >>>>>>>>> Thanks for scheduling this ! > >>>>>>>>> > >>>>>>>>> I should be able to join. > >>>>>>>>> > >>>>>>>>> For the community, will you be able to record ? > >>>>>>>>> > >>>>>>>>> Regards > >>>>>>>>> JB > >>>>>>>>> > >>>>>>>>> On Mon, Nov 4, 2024 at 10:40 PM Yufei Gu < > >> flyrain...@gmail.com> > >>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>> Hi Folks, > >>>>>>>>>> > >>>>>>>>>> I've scheduled a community sync to discuss table maintenance > >>> in > >>>>>>> Polaris > >>>>>>>>>> this Thursday at 9 AM PST. Since we didn’t have a chance to > >>> dive > >>>>>> into > >>>>>>>>> this > >>>>>>>>>> topic during our last sync, this will be a dedicated session > >>> to > >>>>>> cover > >>>>>>>>> it in > >>>>>>>>>> detail. > >>>>>>>>>> > >>>>>>>>>> *Updates to Note:* I've made some updates to the design > >>>> document, > >>>>>>> with a > >>>>>>>>>> particular focus on the approach for maintenance metadata. > >> The > >>>>>>> document > >>>>>>>>> now > >>>>>>>>>> favors *Option 1*, which involves leveraging table, > >> namespace, > >>>> and > >>>>>>>>> catalog > >>>>>>>>>> properties for maintenance metadata. > >>>>>>>>>> > >>>>>>>>>> Please review the latest version of the design doc before > >> the > >>>>>> meeting, > >>>>>>>>> as > >>>>>>>>>> it will help us streamline the discussion. > >>>>>>>>>> > >>>>>>>>>> Looking forward to everyone’s insights! > >>>>>>>>>> Video call link: https://meet.google.com/opc-vath-mgb > >>>>>>>>>> Design doc: > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > https://docs.google.com/document/d/1Pd_mzZcfvnUvcH98IbwsIYf4eryet1lQDfclKYx-t-M/edit?usp=sharing > >>>>>>>>>> < > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > https://www.google.com/url?q=https://docs.google.com/document/d/1Pd_mzZcfvnUvcH98IbwsIYf4eryet1lQDfclKYx-t-M/edit?usp%3Dsharing&sa=D&source=calendar&usd=2&usg=AOvVaw2V3IjIcadea8miDcKKSG9I > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Yufei > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > >
