Hi Folks, Happy New Year! I hope you all had a wonderful and refreshing break.
Following our previous discussions, we have decided to use separated policy entities(option 2) for table maintenance. I've outlined the detailed design here, https://docs.google.com/document/d/1kIiVkFFg9tPa5SH70b9WwzbmclrzH3qWHKfCKXw5lbs/edit?usp=sharing. It is based on the previous design with a wider scope for policy management. I’d love to hear your thoughts, feedback, or suggestions, so feel free to review and share your input. Yufei On Tue, Dec 10, 2024 at 5:43 PM Yufei Gu <flyrain...@gmail.com> wrote: > Hi everyone, > > > Thank you all for taking the time to meet! Here’s a summary of our > discussion: > > 1. *Challenges with Storing Policies as Properties (Option 1):* > - We identified scalability limitations for access control in this > approach. > 2. *Benefits of Using Separate Policy Entities (Option 2):* > - This approach offers a more generic solution with improved access > control and better performance. > - This approach could apply to a variety of use cases, like column > masking. > - There are certain agreements on this approach. > 3. *Other Options Considered:* > - Storing policies as Polaris entity properties or using a 1:1 > mapping of policy entities with catalog/namespace/table entities. > - While slightly different from Option 1, these approaches still > present notable drawbacks similar to option 1. > 4. *Option to Delegate Policy Storage to TMS:* > - We discussed the possibility of not storing any policies in > Polaris, allowing TMS to manage all policies. > - However, the proposed approach aims to promote interoperability > across engines and systems like TMS, without preventing them from having > their own rules or policies. > > > Please let me know if I missed anything or if further clarifications are > needed. > > > > Yufei > > > On Wed, Dec 4, 2024 at 2:37 PM Omar Al-Safi <o...@oalsafi.com> wrote: > >> Thank you Yufei for the flexibility! >> >> Regards, >> Omar >> >> On Wed, 4 Dec 2024, 23:12 Yufei Gu, <flyrain...@gmail.com> wrote: >> >> > I've rescheduled it to next Monday due to the availability. Sorry for >> > any inconvenience. FYI, I will not record it as I don't have a >> > premium google account yet. >> > >> > Table maintenance in Polaris >> > Monday, December 9 · 9:00 – 10:00am >> > Time zone: America/Los_Angeles >> > Google Meet joining info >> > Video call link: https://meet.google.com/dix-cdfm-pve >> > >> > Yufei >> > >> > >> > On Wed, Dec 4, 2024 at 1:15 AM Omar Al-Safi <o...@oalsafi.com> wrote: >> > >> > > Thank you Yufei for getting this moving. >> > > >> > > Unfortunately tomorrow I won't be able to make it plus I think a >> couple >> > of >> > > guys are at reinvent (JB for example), would it make sense to >> reschedule >> > to >> > > early next week? Or maybe have it recorded. >> > > As I highlighted in the document, I am feeling embedding the policies >> > into >> > > Polaris feels more of TMS concern rather than Polaris concern. >> Unless, we >> > > provide a way to have pluggable policies where you can either rely on >> > > Polaris to store the polices or the pluggable implementation would >> handle >> > > how that can be stored, which I think fits well in both worlds. >> > > >> > > Regards, >> > > Omar >> > > >> > > On Tue, Dec 3, 2024 at 10:26 PM Yufei Gu <flyrain...@gmail.com> >> wrote: >> > > >> > > > Sorry the meeting title is misleading, the meeting itself is >> scheduled >> > on >> > > > Dec. 5th. Thanks Anurag for pointing that out. >> > > > >> > > > Table maintenance in Polaris >> > > > Thursday, December 5 · 9:00 – 10:00am >> > > > Time zone: America/Los_Angeles >> > > > Google Meet joining info >> > > > Video call link: https://meet.google.com/dix-cdfm-pve >> > > > >> > > > Yufei >> > > > >> > > > >> > > > On Tue, Dec 3, 2024 at 12:32 PM Anurag Mantripragada >> > > > <amantriprag...@apple.com.invalid> wrote: >> > > > >> > > > > Thanks Yufei, I think you meant Thursday, December 5th 9:00am – >> > 10:00am >> > > > > (GMT-08). >> > > > > >> > > > > >> > > > > Anurag Mantripragada >> > > > > >> > > > > >> > > > > > On Dec 3, 2024, at 11:33 AM, Yufei Gu <flyrain...@gmail.com> >> > wrote: >> > > > > > >> > > > > > Hi Folks, >> > > > > > >> > > > > > We’ve made some adjustments to the design, moving from *Option >> 1* >> > to >> > > > > *Option >> > > > > > 2*: >> > > > > > >> > > > > > 1. *Option 1:* Store maintenance policies in >> > > catalog/namespace/table >> > > > > > properties. >> > > > > > 2. *Option 2:* Store maintenance policies as separate >> entities. >> > > > > > >> > > > > > The key concern with Option 1 is that the access control model >> > isn't >> > > > > > scalable. On the other hand, Option 2 provides greater >> flexibility, >> > > > > > improved privilege enforcement, and better overall performance. >> > > > > > >> > > > > > I’ve updated the design document with the latest changes, which >> you >> > > can >> > > > > > find here: Updated Design Document >> > > > > > < >> > > > > >> > > > >> > > >> > >> https://docs.google.com/document/d/1Pd_mzZcfvnUvcH98IbwsIYf4eryet1lQDfclKYx-t-M/edit?usp=sharing >> > > > > > >> > > > > > . >> > > > > > >> > > > > > To discuss this design change in detail, I’ll be hosting a >> session >> > on >> > > > > > Thursday. Please find the meeting details below: >> > > > > > Table maintenance in Polaris @ Thu, Nov 7, 2024 9:00am – 10:00am >> > > > (GMT-08) >> > > > > > Thursday, December 5 · 9:00 – 10:00am >> > > > > > Time zone: America/Los_Angeles >> > > > > > Google Meet joining info >> > > > > > Video call link: https://meet.google.com/dix-cdfm-pve >> > > > > > >> > > > > > Feel free to review the updated document ahead of the session. >> > > Looking >> > > > > > forward to your thoughts and feedback during the meeting! >> > > > > > >> > > > > > Yufei >> > > > > > >> > > > > > >> > > > > > On Mon, Nov 18, 2024 at 9:43 PM Jean-Baptiste Onofré < >> > > j...@nanthrax.net> >> > > > > > wrote: >> > > > > > >> > > > > >> Hi Yufei >> > > > > >> >> > > > > >> Not sure we got consensus in all details but the overall >> picture >> > is >> > > ok >> > > > > for >> > > > > >> me. >> > > > > >> >> > > > > >> Let’s continue the details definition in the PR. >> > > > > >> >> > > > > >> Thanks ! >> > > > > >> Regards >> > > > > >> JB >> > > > > >> >> > > > > >> Le jeu. 14 nov. 2024 à 02:39, Yufei Gu <flyrain...@gmail.com> >> a >> > > > écrit : >> > > > > >> >> > > > > >>> Hi everyone, >> > > > > >>> >> > > > > >>> >> > > > > >>> Thank you for joining the table maintenance discussion today! >> We >> > > made >> > > > > >>> significant progress, and here are the key takeaways: >> > > > > >>> >> > > > > >>> 1. Clarified furthermore and reached consensus on >> introducing >> > > table >> > > > > >>> maintenance properties in Polaris to support for different >> TMS >> > > and >> > > > > >>> promote >> > > > > >>> interoperability. >> > > > > >>> 2. Agreed to proceed with Option 1, which stores metadata as >> > > > > >>> catalog/namespace/table properties. >> > > > > >>> 3. Confirmed the new privileges to ensure that maintenance >> > > > properties >> > > > > >>> are safeguarded from being altered by clients with existing >> > write >> > > > > >>> access. >> > > > > >>> 4. Briefly discussed the support for customized maintenance >> > > > policies >> > > > > . >> > > > > >>> >> > > > > >>> Next step: >> > > > > >>> >> > > > > >>> 1. Will file maintenance properties related PRs per design >> > > > > >>> 2. Will add more details for customized policy support. >> > > > > >>> >> > > > > >>> *Note*: Unfortunately, I wasn’t able to record the meeting >> due to >> > > the >> > > > > >> need >> > > > > >>> for a Google premium account. >> > > > > >>> >> > > > > >>> >> > > > > >>> Yufei >> > > > > >>> >> > > > > >>> >> > > > > >>> On Tue, Nov 12, 2024 at 10:10 AM Omar Al-Safi < >> o...@oalsafi.com> >> > > > > wrote: >> > > > > >>> >> > > > > >>>> Thank you! Will try to be there >> > > > > >>>> >> > > > > >>>> On Tue, 12 Nov 2024, 18:55 Yufei Gu, <flyrain...@gmail.com> >> > > wrote: >> > > > > >>>> >> > > > > >>>>> Hi Omar, I sent the invitation to dev@polaris.apache.org, >> as >> > > well >> > > > as >> > > > > >>>> your >> > > > > >>>>> email address. >> > > > > >>>>> >> > > > > >>>>> Yufei >> > > > > >>>>> >> > > > > >>>>> >> > > > > >>>>> On Tue, Nov 12, 2024 at 9:51 AM Omar Al-Safi < >> o...@oalsafi.com >> > > >> > > > > >> wrote: >> > > > > >>>>> >> > > > > >>>>>> Thanks Yufei, is it possible to send the invitation to the >> > > > > >>>>>> Polaris google group? >> > > > > >>>>>> >> > > > > >>>>>> Regards, >> > > > > >>>>>> Omar >> > > > > >>>>>> >> > > > > >>>>>> On Tue, Nov 12, 2024 at 6:48 PM Yufei Gu < >> > flyrain...@gmail.com> >> > > > > >>> wrote: >> > > > > >>>>>> >> > > > > >>>>>>> Hi folks, >> > > > > >>>>>>> >> > > > > >>>>>>> We are going to have another sync for table maintenance in >> > > > > >> Polaris >> > > > > >>>> per >> > > > > >>>>>>> discussion with JB. Here are meeting details: >> > > > > >>>>>>> >> > > > > >>>>>>> Polaris Table maintenance sync >> > > > > >>>>>>> Wednesday, November 13 · 10:00 – 11:00am >> > > > > >>>>>>> Time zone: America/Los_Angeles >> > > > > >>>>>>> Google Meet joining info >> > > > > >>>>>>> Video call link: https://meet.google.com/nyy-ahmn-jqd >> > > > > >>>>>>> >> > > > > >>>>>>> >> > > > > >>>>>>> Yufei >> > > > > >>>>>>> >> > > > > >>>>>>> >> > > > > >>>>>>> On Fri, Nov 8, 2024 at 5:23 PM Yufei Gu < >> > flyrain...@gmail.com> >> > > > > >>>> wrote: >> > > > > >>>>>>> >> > > > > >>>>>>>> Thanks everyone for joining the discussion. Sorry I >> couldn't >> > > > > >>> record >> > > > > >>>>> the >> > > > > >>>>>>>> session due to a tech issue. Here are meeting notes: >> > > > > >>>>>>>> >> > > > > >>>>>>>> 1. We discussed the boundary between Polaris and the >> Table >> > > > > >>>>>> Maintenance >> > > > > >>>>>>>> System(TMS). We agreed that they should be separated >> > > > > >> systems. >> > > > > >>>>>>>> 2. A general agreement on the minimal metadata added to >> > > > > >>> Polaris >> > > > > >>>> to >> > > > > >>>>>>>> support TMS, focusing on essential data needed for >> > > > > >>>>> interoperability. >> > > > > >>>>>>>> 3. A general consensus on option 1 to store metadata as >> > > > > >>>>>>>> catalog/namespace/table properties. We could introduce >> > > > > >> policy >> > > > > >>>>>>> entities in >> > > > > >>>>>>>> the future for other use cases, like column masking. >> Will >> > > > > >>>> address >> > > > > >>>>>> two >> > > > > >>>>>>>> feedbacks: >> > > > > >>>>>>>> 1. Caching the table properties in the catalog to >> > reduce >> > > > > >> IO >> > > > > >>>>> cost. >> > > > > >>>>>>>> 2. Introducing new permissions for table maintenance >> > > > > >>> related >> > > > > >>>>>>>> metadata to prevent any clients with the write >> > permission >> > > > > >>> to >> > > > > >>>>> mess >> > > > > >>>>>>> up with >> > > > > >>>>>>>> them. >> > > > > >>>>>>>> 4. Briefly touched on the communication module between >> TMS >> > > > > >> and >> > > > > >>>>>>>> Polaris, as a long-term plan, an event system from >> Polaris >> > > > > >> is >> > > > > >>>>>>> necessary, >> > > > > >>>>>>>> not only benefits TMS, but also benefits other systems >> > which >> > > > > >>>>> consume >> > > > > >>>>>>> change >> > > > > >>>>>>>> from Polaris. >> > > > > >>>>>>>> >> > > > > >>>>>>>> Next Steps: >> > > > > >>>>>>>> >> > > > > >>>>>>>> 1. Implement metadata storage as properties >> > > > > >>>>>>>> 1. Design detailed schema for properties >> > > > > >>>>>>>> 2. Figure out a way to be extensible for future >> > > > > >> maintenance >> > > > > >>>>>> policy >> > > > > >>>>>>>> or customized policies. >> > > > > >>>>>>>> 3. Add new permissions for new properties >> > > > > >>>>>>>> 2. Begin planning for event system >> > > > > >>>>>>>> >> > > > > >>>>>>>> Yufei >> > > > > >>>>>>>> >> > > > > >>>>>>>> >> > > > > >>>>>>>> On Tue, Nov 5, 2024 at 12:25 AM Jean-Baptiste Onofré < >> > > > > >>>>> j...@nanthrax.net> >> > > > > >>>>>>>> wrote: >> > > > > >>>>>>>> >> > > > > >>>>>>>>> Hi Yufei >> > > > > >>>>>>>>> >> > > > > >>>>>>>>> Thanks for scheduling this ! >> > > > > >>>>>>>>> >> > > > > >>>>>>>>> I should be able to join. >> > > > > >>>>>>>>> >> > > > > >>>>>>>>> For the community, will you be able to record ? >> > > > > >>>>>>>>> >> > > > > >>>>>>>>> Regards >> > > > > >>>>>>>>> JB >> > > > > >>>>>>>>> >> > > > > >>>>>>>>> On Mon, Nov 4, 2024 at 10:40 PM Yufei Gu < >> > > > > >> flyrain...@gmail.com> >> > > > > >>>>>> wrote: >> > > > > >>>>>>>>>> >> > > > > >>>>>>>>>> Hi Folks, >> > > > > >>>>>>>>>> >> > > > > >>>>>>>>>> I've scheduled a community sync to discuss table >> > maintenance >> > > > > >>> in >> > > > > >>>>>>> Polaris >> > > > > >>>>>>>>>> this Thursday at 9 AM PST. Since we didn’t have a >> chance >> > to >> > > > > >>> dive >> > > > > >>>>>> into >> > > > > >>>>>>>>> this >> > > > > >>>>>>>>>> topic during our last sync, this will be a dedicated >> > session >> > > > > >>> to >> > > > > >>>>>> cover >> > > > > >>>>>>>>> it in >> > > > > >>>>>>>>>> detail. >> > > > > >>>>>>>>>> >> > > > > >>>>>>>>>> *Updates to Note:* I've made some updates to the design >> > > > > >>>> document, >> > > > > >>>>>>> with a >> > > > > >>>>>>>>>> particular focus on the approach for maintenance >> metadata. >> > > > > >> The >> > > > > >>>>>>> document >> > > > > >>>>>>>>> now >> > > > > >>>>>>>>>> favors *Option 1*, which involves leveraging table, >> > > > > >> namespace, >> > > > > >>>> and >> > > > > >>>>>>>>> catalog >> > > > > >>>>>>>>>> properties for maintenance metadata. >> > > > > >>>>>>>>>> >> > > > > >>>>>>>>>> Please review the latest version of the design doc >> before >> > > > > >> the >> > > > > >>>>>> meeting, >> > > > > >>>>>>>>> as >> > > > > >>>>>>>>>> it will help us streamline the discussion. >> > > > > >>>>>>>>>> >> > > > > >>>>>>>>>> Looking forward to everyone’s insights! >> > > > > >>>>>>>>>> Video call link: https://meet.google.com/opc-vath-mgb >> > > > > >>>>>>>>>> Design doc: >> > > > > >>>>>>>>>> >> > > > > >>>>>>>>> >> > > > > >>>>>>> >> > > > > >>>>>> >> > > > > >>>>> >> > > > > >>>> >> > > > > >>> >> > > > > >> >> > > > > >> > > > >> > > >> > >> https://docs.google.com/document/d/1Pd_mzZcfvnUvcH98IbwsIYf4eryet1lQDfclKYx-t-M/edit?usp=sharing >> > > > > >>>>>>>>>> < >> > > > > >>>>>>>>> >> > > > > >>>>>>> >> > > > > >>>>>> >> > > > > >>>>> >> > > > > >>>> >> > > > > >>> >> > > > > >> >> > > > > >> > > > >> > > >> > >> https://www.google.com/url?q=https://docs.google.com/document/d/1Pd_mzZcfvnUvcH98IbwsIYf4eryet1lQDfclKYx-t-M/edit?usp%3Dsharing&sa=D&source=calendar&usd=2&usg=AOvVaw2V3IjIcadea8miDcKKSG9I >> > > > > >>>>>>>>>> >> > > > > >>>>>>>>>> >> > > > > >>>>>>>>>> Yufei >> > > > > >>>>>>>>> >> > > > > >>>>>>>> >> > > > > >>>>>>> >> > > > > >>>>>> >> > > > > >>>>> >> > > > > >>>> >> > > > > >>> >> > > > > >> >> > > > > >> > > > > >> > > > >> > > >> > >> >
