Hi folks, We are going to have another sync for table maintenance in Polaris per discussion with JB. Here are meeting details:
Polaris Table maintenance sync Wednesday, November 13 · 10:00 – 11:00am Time zone: America/Los_Angeles Google Meet joining info Video call link: https://meet.google.com/nyy-ahmn-jqd Yufei On Fri, Nov 8, 2024 at 5:23 PM Yufei Gu <flyrain...@gmail.com> wrote: > Thanks everyone for joining the discussion. Sorry I couldn't record the > session due to a tech issue. Here are meeting notes: > > 1. We discussed the boundary between Polaris and the Table Maintenance > System(TMS). We agreed that they should be separated systems. > 2. A general agreement on the minimal metadata added to Polaris to > support TMS, focusing on essential data needed for interoperability. > 3. A general consensus on option 1 to store metadata as > catalog/namespace/table properties. We could introduce policy entities in > the future for other use cases, like column masking. Will address two > feedbacks: > 1. Caching the table properties in the catalog to reduce IO cost. > 2. Introducing new permissions for table maintenance related > metadata to prevent any clients with the write permission to mess up > with > them. > 4. Briefly touched on the communication module between TMS and > Polaris, as a long-term plan, an event system from Polaris is necessary, > not only benefits TMS, but also benefits other systems which consume change > from Polaris. > > Next Steps: > > 1. Implement metadata storage as properties > 1. Design detailed schema for properties > 2. Figure out a way to be extensible for future maintenance policy > or customized policies. > 3. Add new permissions for new properties > 2. Begin planning for event system > > Yufei > > > On Tue, Nov 5, 2024 at 12:25 AM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > >> Hi Yufei >> >> Thanks for scheduling this ! >> >> I should be able to join. >> >> For the community, will you be able to record ? >> >> Regards >> JB >> >> On Mon, Nov 4, 2024 at 10:40 PM Yufei Gu <flyrain...@gmail.com> wrote: >> > >> > Hi Folks, >> > >> > I've scheduled a community sync to discuss table maintenance in Polaris >> > this Thursday at 9 AM PST. Since we didn’t have a chance to dive into >> this >> > topic during our last sync, this will be a dedicated session to cover >> it in >> > detail. >> > >> > *Updates to Note:* I've made some updates to the design document, with a >> > particular focus on the approach for maintenance metadata. The document >> now >> > favors *Option 1*, which involves leveraging table, namespace, and >> catalog >> > properties for maintenance metadata. >> > >> > Please review the latest version of the design doc before the meeting, >> as >> > it will help us streamline the discussion. >> > >> > Looking forward to everyone’s insights! >> > Video call link: https://meet.google.com/opc-vath-mgb >> > Design doc: >> > >> https://docs.google.com/document/d/1Pd_mzZcfvnUvcH98IbwsIYf4eryet1lQDfclKYx-t-M/edit?usp=sharing >> > < >> https://www.google.com/url?q=https://docs.google.com/document/d/1Pd_mzZcfvnUvcH98IbwsIYf4eryet1lQDfclKYx-t-M/edit?usp%3Dsharing&sa=D&source=calendar&usd=2&usg=AOvVaw2V3IjIcadea8miDcKKSG9I >> > >> > >> > Yufei >> >
