Thanks everyone for joining the discussion. Sorry I couldn't record the
session due to a tech issue. Here are meeting notes:
1. We discussed the boundary between Polaris and the Table Maintenance
System(TMS). We agreed that they should be separated systems.
2. A general agreement on the minimal metadata added to Polaris to
support TMS, focusing on essential data needed for interoperability.
3. A general consensus on option 1 to store metadata as
catalog/namespace/table properties. We could introduce policy entities in
the future for other use cases, like column masking. Will address two
feedbacks:
1. Caching the table properties in the catalog to reduce IO cost.
2. Introducing new permissions for table maintenance related metadata
to prevent any clients with the write permission to mess up with them.
4. Briefly touched on the communication module between TMS and Polaris,
as a long-term plan, an event system from Polaris is necessary, not only
benefits TMS, but also benefits other systems which consume change from
Polaris.
Next Steps:
1. Implement metadata storage as properties
1. Design detailed schema for properties
2. Figure out a way to be extensible for future maintenance policy or
customized policies.
3. Add new permissions for new properties
2. Begin planning for event system
Yufei
On Tue, Nov 5, 2024 at 12:25 AM Jean-Baptiste Onofré <j...@nanthrax.net>
wrote:
> Hi Yufei
>
> Thanks for scheduling this !
>
> I should be able to join.
>
> For the community, will you be able to record ?
>
> Regards
> JB
>
> On Mon, Nov 4, 2024 at 10:40 PM Yufei Gu <flyrain...@gmail.com> wrote:
> >
> > Hi Folks,
> >
> > I've scheduled a community sync to discuss table maintenance in Polaris
> > this Thursday at 9 AM PST. Since we didn’t have a chance to dive into
> this
> > topic during our last sync, this will be a dedicated session to cover it
> in
> > detail.
> >
> > *Updates to Note:* I've made some updates to the design document, with a
> > particular focus on the approach for maintenance metadata. The document
> now
> > favors *Option 1*, which involves leveraging table, namespace, and
> catalog
> > properties for maintenance metadata.
> >
> > Please review the latest version of the design doc before the meeting, as
> > it will help us streamline the discussion.
> >
> > Looking forward to everyone’s insights!
> > Video call link: https://meet.google.com/opc-vath-mgb
> > Design doc:
> >
> https://docs.google.com/document/d/1Pd_mzZcfvnUvcH98IbwsIYf4eryet1lQDfclKYx-t-M/edit?usp=sharing
> > <
> https://www.google.com/url?q=https://docs.google.com/document/d/1Pd_mzZcfvnUvcH98IbwsIYf4eryet1lQDfclKYx-t-M/edit?usp%3Dsharing&sa=D&source=calendar&usd=2&usg=AOvVaw2V3IjIcadea8miDcKKSG9I
> >
> >
> > Yufei
>
