Thank you Yufei for the flexibility! Regards, Omar
On Wed, 4 Dec 2024, 23:12 Yufei Gu, <flyrain...@gmail.com> wrote: > I've rescheduled it to next Monday due to the availability. Sorry for > any inconvenience. FYI, I will not record it as I don't have a > premium google account yet. > > Table maintenance in Polaris > Monday, December 9 · 9:00 – 10:00am > Time zone: America/Los_Angeles > Google Meet joining info > Video call link: https://meet.google.com/dix-cdfm-pve > > Yufei > > > On Wed, Dec 4, 2024 at 1:15 AM Omar Al-Safi <o...@oalsafi.com> wrote: > > > Thank you Yufei for getting this moving. > > > > Unfortunately tomorrow I won't be able to make it plus I think a couple > of > > guys are at reinvent (JB for example), would it make sense to reschedule > to > > early next week? Or maybe have it recorded. > > As I highlighted in the document, I am feeling embedding the policies > into > > Polaris feels more of TMS concern rather than Polaris concern. Unless, we > > provide a way to have pluggable policies where you can either rely on > > Polaris to store the polices or the pluggable implementation would handle > > how that can be stored, which I think fits well in both worlds. > > > > Regards, > > Omar > > > > On Tue, Dec 3, 2024 at 10:26 PM Yufei Gu <flyrain...@gmail.com> wrote: > > > > > Sorry the meeting title is misleading, the meeting itself is scheduled > on > > > Dec. 5th. Thanks Anurag for pointing that out. > > > > > > Table maintenance in Polaris > > > Thursday, December 5 · 9:00 – 10:00am > > > Time zone: America/Los_Angeles > > > Google Meet joining info > > > Video call link: https://meet.google.com/dix-cdfm-pve > > > > > > Yufei > > > > > > > > > On Tue, Dec 3, 2024 at 12:32 PM Anurag Mantripragada > > > <amantriprag...@apple.com.invalid> wrote: > > > > > > > Thanks Yufei, I think you meant Thursday, December 5th 9:00am – > 10:00am > > > > (GMT-08). > > > > > > > > > > > > Anurag Mantripragada > > > > > > > > > > > > > On Dec 3, 2024, at 11:33 AM, Yufei Gu <flyrain...@gmail.com> > wrote: > > > > > > > > > > Hi Folks, > > > > > > > > > > We’ve made some adjustments to the design, moving from *Option 1* > to > > > > *Option > > > > > 2*: > > > > > > > > > > 1. *Option 1:* Store maintenance policies in > > catalog/namespace/table > > > > > properties. > > > > > 2. *Option 2:* Store maintenance policies as separate entities. > > > > > > > > > > The key concern with Option 1 is that the access control model > isn't > > > > > scalable. On the other hand, Option 2 provides greater flexibility, > > > > > improved privilege enforcement, and better overall performance. > > > > > > > > > > I’ve updated the design document with the latest changes, which you > > can > > > > > find here: Updated Design Document > > > > > < > > > > > > > > > > https://docs.google.com/document/d/1Pd_mzZcfvnUvcH98IbwsIYf4eryet1lQDfclKYx-t-M/edit?usp=sharing > > > > > > > > > > . > > > > > > > > > > To discuss this design change in detail, I’ll be hosting a session > on > > > > > Thursday. Please find the meeting details below: > > > > > Table maintenance in Polaris @ Thu, Nov 7, 2024 9:00am – 10:00am > > > (GMT-08) > > > > > Thursday, December 5 · 9:00 – 10:00am > > > > > Time zone: America/Los_Angeles > > > > > Google Meet joining info > > > > > Video call link: https://meet.google.com/dix-cdfm-pve > > > > > > > > > > Feel free to review the updated document ahead of the session. > > Looking > > > > > forward to your thoughts and feedback during the meeting! > > > > > > > > > > Yufei > > > > > > > > > > > > > > > On Mon, Nov 18, 2024 at 9:43 PM Jean-Baptiste Onofré < > > j...@nanthrax.net> > > > > > wrote: > > > > > > > > > >> Hi Yufei > > > > >> > > > > >> Not sure we got consensus in all details but the overall picture > is > > ok > > > > for > > > > >> me. > > > > >> > > > > >> Let’s continue the details definition in the PR. > > > > >> > > > > >> Thanks ! > > > > >> Regards > > > > >> JB > > > > >> > > > > >> Le jeu. 14 nov. 2024 à 02:39, Yufei Gu <flyrain...@gmail.com> a > > > écrit : > > > > >> > > > > >>> Hi everyone, > > > > >>> > > > > >>> > > > > >>> Thank you for joining the table maintenance discussion today! We > > made > > > > >>> significant progress, and here are the key takeaways: > > > > >>> > > > > >>> 1. Clarified furthermore and reached consensus on introducing > > table > > > > >>> maintenance properties in Polaris to support for different TMS > > and > > > > >>> promote > > > > >>> interoperability. > > > > >>> 2. Agreed to proceed with Option 1, which stores metadata as > > > > >>> catalog/namespace/table properties. > > > > >>> 3. Confirmed the new privileges to ensure that maintenance > > > properties > > > > >>> are safeguarded from being altered by clients with existing > write > > > > >>> access. > > > > >>> 4. Briefly discussed the support for customized maintenance > > > policies > > > > . > > > > >>> > > > > >>> Next step: > > > > >>> > > > > >>> 1. Will file maintenance properties related PRs per design > > > > >>> 2. Will add more details for customized policy support. > > > > >>> > > > > >>> *Note*: Unfortunately, I wasn’t able to record the meeting due to > > the > > > > >> need > > > > >>> for a Google premium account. > > > > >>> > > > > >>> > > > > >>> Yufei > > > > >>> > > > > >>> > > > > >>> On Tue, Nov 12, 2024 at 10:10 AM Omar Al-Safi <o...@oalsafi.com> > > > > wrote: > > > > >>> > > > > >>>> Thank you! Will try to be there > > > > >>>> > > > > >>>> On Tue, 12 Nov 2024, 18:55 Yufei Gu, <flyrain...@gmail.com> > > wrote: > > > > >>>> > > > > >>>>> Hi Omar, I sent the invitation to dev@polaris.apache.org, as > > well > > > as > > > > >>>> your > > > > >>>>> email address. > > > > >>>>> > > > > >>>>> Yufei > > > > >>>>> > > > > >>>>> > > > > >>>>> On Tue, Nov 12, 2024 at 9:51 AM Omar Al-Safi <o...@oalsafi.com > > > > > > >> wrote: > > > > >>>>> > > > > >>>>>> Thanks Yufei, is it possible to send the invitation to the > > > > >>>>>> Polaris google group? > > > > >>>>>> > > > > >>>>>> Regards, > > > > >>>>>> Omar > > > > >>>>>> > > > > >>>>>> On Tue, Nov 12, 2024 at 6:48 PM Yufei Gu < > flyrain...@gmail.com> > > > > >>> wrote: > > > > >>>>>> > > > > >>>>>>> Hi folks, > > > > >>>>>>> > > > > >>>>>>> We are going to have another sync for table maintenance in > > > > >> Polaris > > > > >>>> per > > > > >>>>>>> discussion with JB. Here are meeting details: > > > > >>>>>>> > > > > >>>>>>> Polaris Table maintenance sync > > > > >>>>>>> Wednesday, November 13 · 10:00 – 11:00am > > > > >>>>>>> Time zone: America/Los_Angeles > > > > >>>>>>> Google Meet joining info > > > > >>>>>>> Video call link: https://meet.google.com/nyy-ahmn-jqd > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>> Yufei > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>> On Fri, Nov 8, 2024 at 5:23 PM Yufei Gu < > flyrain...@gmail.com> > > > > >>>> wrote: > > > > >>>>>>> > > > > >>>>>>>> Thanks everyone for joining the discussion. Sorry I couldn't > > > > >>> record > > > > >>>>> the > > > > >>>>>>>> session due to a tech issue. Here are meeting notes: > > > > >>>>>>>> > > > > >>>>>>>> 1. We discussed the boundary between Polaris and the Table > > > > >>>>>> Maintenance > > > > >>>>>>>> System(TMS). We agreed that they should be separated > > > > >> systems. > > > > >>>>>>>> 2. A general agreement on the minimal metadata added to > > > > >>> Polaris > > > > >>>> to > > > > >>>>>>>> support TMS, focusing on essential data needed for > > > > >>>>> interoperability. > > > > >>>>>>>> 3. A general consensus on option 1 to store metadata as > > > > >>>>>>>> catalog/namespace/table properties. We could introduce > > > > >> policy > > > > >>>>>>> entities in > > > > >>>>>>>> the future for other use cases, like column masking. Will > > > > >>>> address > > > > >>>>>> two > > > > >>>>>>>> feedbacks: > > > > >>>>>>>> 1. Caching the table properties in the catalog to > reduce > > > > >> IO > > > > >>>>> cost. > > > > >>>>>>>> 2. Introducing new permissions for table maintenance > > > > >>> related > > > > >>>>>>>> metadata to prevent any clients with the write > permission > > > > >>> to > > > > >>>>> mess > > > > >>>>>>> up with > > > > >>>>>>>> them. > > > > >>>>>>>> 4. Briefly touched on the communication module between TMS > > > > >> and > > > > >>>>>>>> Polaris, as a long-term plan, an event system from Polaris > > > > >> is > > > > >>>>>>> necessary, > > > > >>>>>>>> not only benefits TMS, but also benefits other systems > which > > > > >>>>> consume > > > > >>>>>>> change > > > > >>>>>>>> from Polaris. > > > > >>>>>>>> > > > > >>>>>>>> Next Steps: > > > > >>>>>>>> > > > > >>>>>>>> 1. Implement metadata storage as properties > > > > >>>>>>>> 1. Design detailed schema for properties > > > > >>>>>>>> 2. Figure out a way to be extensible for future > > > > >> maintenance > > > > >>>>>> policy > > > > >>>>>>>> or customized policies. > > > > >>>>>>>> 3. Add new permissions for new properties > > > > >>>>>>>> 2. Begin planning for event system > > > > >>>>>>>> > > > > >>>>>>>> Yufei > > > > >>>>>>>> > > > > >>>>>>>> > > > > >>>>>>>> On Tue, Nov 5, 2024 at 12:25 AM Jean-Baptiste Onofré < > > > > >>>>> j...@nanthrax.net> > > > > >>>>>>>> wrote: > > > > >>>>>>>> > > > > >>>>>>>>> Hi Yufei > > > > >>>>>>>>> > > > > >>>>>>>>> Thanks for scheduling this ! > > > > >>>>>>>>> > > > > >>>>>>>>> I should be able to join. > > > > >>>>>>>>> > > > > >>>>>>>>> For the community, will you be able to record ? > > > > >>>>>>>>> > > > > >>>>>>>>> Regards > > > > >>>>>>>>> JB > > > > >>>>>>>>> > > > > >>>>>>>>> On Mon, Nov 4, 2024 at 10:40 PM Yufei Gu < > > > > >> flyrain...@gmail.com> > > > > >>>>>> wrote: > > > > >>>>>>>>>> > > > > >>>>>>>>>> Hi Folks, > > > > >>>>>>>>>> > > > > >>>>>>>>>> I've scheduled a community sync to discuss table > maintenance > > > > >>> in > > > > >>>>>>> Polaris > > > > >>>>>>>>>> this Thursday at 9 AM PST. Since we didn’t have a chance > to > > > > >>> dive > > > > >>>>>> into > > > > >>>>>>>>> this > > > > >>>>>>>>>> topic during our last sync, this will be a dedicated > session > > > > >>> to > > > > >>>>>> cover > > > > >>>>>>>>> it in > > > > >>>>>>>>>> detail. > > > > >>>>>>>>>> > > > > >>>>>>>>>> *Updates to Note:* I've made some updates to the design > > > > >>>> document, > > > > >>>>>>> with a > > > > >>>>>>>>>> particular focus on the approach for maintenance metadata. > > > > >> The > > > > >>>>>>> document > > > > >>>>>>>>> now > > > > >>>>>>>>>> favors *Option 1*, which involves leveraging table, > > > > >> namespace, > > > > >>>> and > > > > >>>>>>>>> catalog > > > > >>>>>>>>>> properties for maintenance metadata. > > > > >>>>>>>>>> > > > > >>>>>>>>>> Please review the latest version of the design doc before > > > > >> the > > > > >>>>>> meeting, > > > > >>>>>>>>> as > > > > >>>>>>>>>> it will help us streamline the discussion. > > > > >>>>>>>>>> > > > > >>>>>>>>>> Looking forward to everyone’s insights! > > > > >>>>>>>>>> Video call link: https://meet.google.com/opc-vath-mgb > > > > >>>>>>>>>> Design doc: > > > > >>>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>> > > > > >>>>>> > > > > >>>>> > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > https://docs.google.com/document/d/1Pd_mzZcfvnUvcH98IbwsIYf4eryet1lQDfclKYx-t-M/edit?usp=sharing > > > > >>>>>>>>>> < > > > > >>>>>>>>> > > > > >>>>>>> > > > > >>>>>> > > > > >>>>> > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > https://www.google.com/url?q=https://docs.google.com/document/d/1Pd_mzZcfvnUvcH98IbwsIYf4eryet1lQDfclKYx-t-M/edit?usp%3Dsharing&sa=D&source=calendar&usd=2&usg=AOvVaw2V3IjIcadea8miDcKKSG9I > > > > >>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>>> Yufei > > > > >>>>>>>>> > > > > >>>>>>>> > > > > >>>>>>> > > > > >>>>>> > > > > >>>>> > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > > > > >
