Hi everyone,
Thank you all for taking the time to meet! Here’s a summary of our
discussion:
1. *Challenges with Storing Policies as Properties (Option 1):*
- We identified scalability limitations for access control in this
approach.
2. *Benefits of Using Separate Policy Entities (Option 2):*
- This approach offers a more generic solution with improved access
control and better performance.
- This approach could apply to a variety of use cases, like column
masking.
- There are certain agreements on this approach.
3. *Other Options Considered:*
- Storing policies as Polaris entity properties or using a 1:1
mapping of policy entities with catalog/namespace/table entities.
- While slightly different from Option 1, these approaches still
present notable drawbacks similar to option 1.
4. *Option to Delegate Policy Storage to TMS:*
- We discussed the possibility of not storing any policies in
Polaris, allowing TMS to manage all policies.
- However, the proposed approach aims to promote interoperability
across engines and systems like TMS, without preventing them from having
their own rules or policies.
Please let me know if I missed anything or if further clarifications are
needed.
Yufei
On Wed, Dec 4, 2024 at 2:37 PM Omar Al-Safi <o...@oalsafi.com> wrote:
> Thank you Yufei for the flexibility!
>
> Regards,
> Omar
>
> On Wed, 4 Dec 2024, 23:12 Yufei Gu, <flyrain...@gmail.com> wrote:
>
> > I've rescheduled it to next Monday due to the availability. Sorry for
> > any inconvenience. FYI, I will not record it as I don't have a
> > premium google account yet.
> >
> > Table maintenance in Polaris
> > Monday, December 9 · 9:00 – 10:00am
> > Time zone: America/Los_Angeles
> > Google Meet joining info
> > Video call link: https://meet.google.com/dix-cdfm-pve
> >
> > Yufei
> >
> >
> > On Wed, Dec 4, 2024 at 1:15 AM Omar Al-Safi <o...@oalsafi.com> wrote:
> >
> > > Thank you Yufei for getting this moving.
> > >
> > > Unfortunately tomorrow I won't be able to make it plus I think a couple
> > of
> > > guys are at reinvent (JB for example), would it make sense to
> reschedule
> > to
> > > early next week? Or maybe have it recorded.
> > > As I highlighted in the document, I am feeling embedding the policies
> > into
> > > Polaris feels more of TMS concern rather than Polaris concern. Unless,
> we
> > > provide a way to have pluggable policies where you can either rely on
> > > Polaris to store the polices or the pluggable implementation would
> handle
> > > how that can be stored, which I think fits well in both worlds.
> > >
> > > Regards,
> > > Omar
> > >
> > > On Tue, Dec 3, 2024 at 10:26 PM Yufei Gu <flyrain...@gmail.com> wrote:
> > >
> > > > Sorry the meeting title is misleading, the meeting itself is
> scheduled
> > on
> > > > Dec. 5th. Thanks Anurag for pointing that out.
> > > >
> > > > Table maintenance in Polaris
> > > > Thursday, December 5 · 9:00 – 10:00am
> > > > Time zone: America/Los_Angeles
> > > > Google Meet joining info
> > > > Video call link: https://meet.google.com/dix-cdfm-pve
> > > >
> > > > Yufei
> > > >
> > > >
> > > > On Tue, Dec 3, 2024 at 12:32 PM Anurag Mantripragada
> > > > <amantriprag...@apple.com.invalid> wrote:
> > > >
> > > > > Thanks Yufei, I think you meant Thursday, December 5th 9:00am –
> > 10:00am
> > > > > (GMT-08).
> > > > >
> > > > >
> > > > > Anurag Mantripragada
> > > > >
> > > > >
> > > > > > On Dec 3, 2024, at 11:33 AM, Yufei Gu <flyrain...@gmail.com>
> > wrote:
> > > > > >
> > > > > > Hi Folks,
> > > > > >
> > > > > > We’ve made some adjustments to the design, moving from *Option 1*
> > to
> > > > > *Option
> > > > > > 2*:
> > > > > >
> > > > > > 1. *Option 1:* Store maintenance policies in
> > > catalog/namespace/table
> > > > > > properties.
> > > > > > 2. *Option 2:* Store maintenance policies as separate entities.
> > > > > >
> > > > > > The key concern with Option 1 is that the access control model
> > isn't
> > > > > > scalable. On the other hand, Option 2 provides greater
> flexibility,
> > > > > > improved privilege enforcement, and better overall performance.
> > > > > >
> > > > > > I’ve updated the design document with the latest changes, which
> you
> > > can
> > > > > > find here: Updated Design Document
> > > > > > <
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1Pd_mzZcfvnUvcH98IbwsIYf4eryet1lQDfclKYx-t-M/edit?usp=sharing
> > > > > >
> > > > > > .
> > > > > >
> > > > > > To discuss this design change in detail, I’ll be hosting a
> session
> > on
> > > > > > Thursday. Please find the meeting details below:
> > > > > > Table maintenance in Polaris @ Thu, Nov 7, 2024 9:00am – 10:00am
> > > > (GMT-08)
> > > > > > Thursday, December 5 · 9:00 – 10:00am
> > > > > > Time zone: America/Los_Angeles
> > > > > > Google Meet joining info
> > > > > > Video call link: https://meet.google.com/dix-cdfm-pve
> > > > > >
> > > > > > Feel free to review the updated document ahead of the session.
> > > Looking
> > > > > > forward to your thoughts and feedback during the meeting!
> > > > > >
> > > > > > Yufei
> > > > > >
> > > > > >
> > > > > > On Mon, Nov 18, 2024 at 9:43 PM Jean-Baptiste Onofré <
> > > j...@nanthrax.net>
> > > > > > wrote:
> > > > > >
> > > > > >> Hi Yufei
> > > > > >>
> > > > > >> Not sure we got consensus in all details but the overall picture
> > is
> > > ok
> > > > > for
> > > > > >> me.
> > > > > >>
> > > > > >> Let’s continue the details definition in the PR.
> > > > > >>
> > > > > >> Thanks !
> > > > > >> Regards
> > > > > >> JB
> > > > > >>
> > > > > >> Le jeu. 14 nov. 2024 à 02:39, Yufei Gu <flyrain...@gmail.com> a
> > > > écrit :
> > > > > >>
> > > > > >>> Hi everyone,
> > > > > >>>
> > > > > >>>
> > > > > >>> Thank you for joining the table maintenance discussion today!
> We
> > > made
> > > > > >>> significant progress, and here are the key takeaways:
> > > > > >>>
> > > > > >>> 1. Clarified furthermore and reached consensus on introducing
> > > table
> > > > > >>> maintenance properties in Polaris to support for different
> TMS
> > > and
> > > > > >>> promote
> > > > > >>> interoperability.
> > > > > >>> 2. Agreed to proceed with Option 1, which stores metadata as
> > > > > >>> catalog/namespace/table properties.
> > > > > >>> 3. Confirmed the new privileges to ensure that maintenance
> > > > properties
> > > > > >>> are safeguarded from being altered by clients with existing
> > write
> > > > > >>> access.
> > > > > >>> 4. Briefly discussed the support for customized maintenance
> > > > policies
> > > > > .
> > > > > >>>
> > > > > >>> Next step:
> > > > > >>>
> > > > > >>> 1. Will file maintenance properties related PRs per design
> > > > > >>> 2. Will add more details for customized policy support.
> > > > > >>>
> > > > > >>> *Note*: Unfortunately, I wasn’t able to record the meeting due
> to
> > > the
> > > > > >> need
> > > > > >>> for a Google premium account.
> > > > > >>>
> > > > > >>>
> > > > > >>> Yufei
> > > > > >>>
> > > > > >>>
> > > > > >>> On Tue, Nov 12, 2024 at 10:10 AM Omar Al-Safi <
> o...@oalsafi.com>
> > > > > wrote:
> > > > > >>>
> > > > > >>>> Thank you! Will try to be there
> > > > > >>>>
> > > > > >>>> On Tue, 12 Nov 2024, 18:55 Yufei Gu, <flyrain...@gmail.com>
> > > wrote:
> > > > > >>>>
> > > > > >>>>> Hi Omar, I sent the invitation to dev@polaris.apache.org, as
> > > well
> > > > as
> > > > > >>>> your
> > > > > >>>>> email address.
> > > > > >>>>>
> > > > > >>>>> Yufei
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> On Tue, Nov 12, 2024 at 9:51 AM Omar Al-Safi <
> o...@oalsafi.com
> > >
> > > > > >> wrote:
> > > > > >>>>>
> > > > > >>>>>> Thanks Yufei, is it possible to send the invitation to the
> > > > > >>>>>> Polaris google group?
> > > > > >>>>>>
> > > > > >>>>>> Regards,
> > > > > >>>>>> Omar
> > > > > >>>>>>
> > > > > >>>>>> On Tue, Nov 12, 2024 at 6:48 PM Yufei Gu <
> > flyrain...@gmail.com>
> > > > > >>> wrote:
> > > > > >>>>>>
> > > > > >>>>>>> Hi folks,
> > > > > >>>>>>>
> > > > > >>>>>>> We are going to have another sync for table maintenance in
> > > > > >> Polaris
> > > > > >>>> per
> > > > > >>>>>>> discussion with JB. Here are meeting details:
> > > > > >>>>>>>
> > > > > >>>>>>> Polaris Table maintenance sync
> > > > > >>>>>>> Wednesday, November 13 · 10:00 – 11:00am
> > > > > >>>>>>> Time zone: America/Los_Angeles
> > > > > >>>>>>> Google Meet joining info
> > > > > >>>>>>> Video call link: https://meet.google.com/nyy-ahmn-jqd
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> Yufei
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> On Fri, Nov 8, 2024 at 5:23 PM Yufei Gu <
> > flyrain...@gmail.com>
> > > > > >>>> wrote:
> > > > > >>>>>>>
> > > > > >>>>>>>> Thanks everyone for joining the discussion. Sorry I
> couldn't
> > > > > >>> record
> > > > > >>>>> the
> > > > > >>>>>>>> session due to a tech issue. Here are meeting notes:
> > > > > >>>>>>>>
> > > > > >>>>>>>> 1. We discussed the boundary between Polaris and the
> Table
> > > > > >>>>>> Maintenance
> > > > > >>>>>>>> System(TMS). We agreed that they should be separated
> > > > > >> systems.
> > > > > >>>>>>>> 2. A general agreement on the minimal metadata added to
> > > > > >>> Polaris
> > > > > >>>> to
> > > > > >>>>>>>> support TMS, focusing on essential data needed for
> > > > > >>>>> interoperability.
> > > > > >>>>>>>> 3. A general consensus on option 1 to store metadata as
> > > > > >>>>>>>> catalog/namespace/table properties. We could introduce
> > > > > >> policy
> > > > > >>>>>>> entities in
> > > > > >>>>>>>> the future for other use cases, like column masking.
> Will
> > > > > >>>> address
> > > > > >>>>>> two
> > > > > >>>>>>>> feedbacks:
> > > > > >>>>>>>> 1. Caching the table properties in the catalog to
> > reduce
> > > > > >> IO
> > > > > >>>>> cost.
> > > > > >>>>>>>> 2. Introducing new permissions for table maintenance
> > > > > >>> related
> > > > > >>>>>>>> metadata to prevent any clients with the write
> > permission
> > > > > >>> to
> > > > > >>>>> mess
> > > > > >>>>>>> up with
> > > > > >>>>>>>> them.
> > > > > >>>>>>>> 4. Briefly touched on the communication module between
> TMS
> > > > > >> and
> > > > > >>>>>>>> Polaris, as a long-term plan, an event system from
> Polaris
> > > > > >> is
> > > > > >>>>>>> necessary,
> > > > > >>>>>>>> not only benefits TMS, but also benefits other systems
> > which
> > > > > >>>>> consume
> > > > > >>>>>>> change
> > > > > >>>>>>>> from Polaris.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Next Steps:
> > > > > >>>>>>>>
> > > > > >>>>>>>> 1. Implement metadata storage as properties
> > > > > >>>>>>>> 1. Design detailed schema for properties
> > > > > >>>>>>>> 2. Figure out a way to be extensible for future
> > > > > >> maintenance
> > > > > >>>>>> policy
> > > > > >>>>>>>> or customized policies.
> > > > > >>>>>>>> 3. Add new permissions for new properties
> > > > > >>>>>>>> 2. Begin planning for event system
> > > > > >>>>>>>>
> > > > > >>>>>>>> Yufei
> > > > > >>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>> On Tue, Nov 5, 2024 at 12:25 AM Jean-Baptiste Onofré <
> > > > > >>>>> j...@nanthrax.net>
> > > > > >>>>>>>> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>>> Hi Yufei
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Thanks for scheduling this !
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> I should be able to join.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> For the community, will you be able to record ?
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Regards
> > > > > >>>>>>>>> JB
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> On Mon, Nov 4, 2024 at 10:40 PM Yufei Gu <
> > > > > >> flyrain...@gmail.com>
> > > > > >>>>>> wrote:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Hi Folks,
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> I've scheduled a community sync to discuss table
> > maintenance
> > > > > >>> in
> > > > > >>>>>>> Polaris
> > > > > >>>>>>>>>> this Thursday at 9 AM PST. Since we didn’t have a chance
> > to
> > > > > >>> dive
> > > > > >>>>>> into
> > > > > >>>>>>>>> this
> > > > > >>>>>>>>>> topic during our last sync, this will be a dedicated
> > session
> > > > > >>> to
> > > > > >>>>>> cover
> > > > > >>>>>>>>> it in
> > > > > >>>>>>>>>> detail.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> *Updates to Note:* I've made some updates to the design
> > > > > >>>> document,
> > > > > >>>>>>> with a
> > > > > >>>>>>>>>> particular focus on the approach for maintenance
> metadata.
> > > > > >> The
> > > > > >>>>>>> document
> > > > > >>>>>>>>> now
> > > > > >>>>>>>>>> favors *Option 1*, which involves leveraging table,
> > > > > >> namespace,
> > > > > >>>> and
> > > > > >>>>>>>>> catalog
> > > > > >>>>>>>>>> properties for maintenance metadata.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Please review the latest version of the design doc
> before
> > > > > >> the
> > > > > >>>>>> meeting,
> > > > > >>>>>>>>> as
> > > > > >>>>>>>>>> it will help us streamline the discussion.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Looking forward to everyone’s insights!
> > > > > >>>>>>>>>> Video call link: https://meet.google.com/opc-vath-mgb
> > > > > >>>>>>>>>> Design doc:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1Pd_mzZcfvnUvcH98IbwsIYf4eryet1lQDfclKYx-t-M/edit?usp=sharing
> > > > > >>>>>>>>>> <
> > > > > >>>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://www.google.com/url?q=https://docs.google.com/document/d/1Pd_mzZcfvnUvcH98IbwsIYf4eryet1lQDfclKYx-t-M/edit?usp%3Dsharing&sa=D&source=calendar&usd=2&usg=AOvVaw2V3IjIcadea8miDcKKSG9I
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Yufei
> > > > > >>>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
