Hi everyone,
Thank you for joining the table maintenance discussion today! We made significant progress, and here are the key takeaways: 1. Clarified furthermore and reached consensus on introducing table maintenance properties in Polaris to support for different TMS and promote interoperability. 2. Agreed to proceed with Option 1, which stores metadata as catalog/namespace/table properties. 3. Confirmed the new privileges to ensure that maintenance properties are safeguarded from being altered by clients with existing write access. 4. Briefly discussed the support for customized maintenance policies . Next step: 1. Will file maintenance properties related PRs per design 2. Will add more details for customized policy support. *Note*: Unfortunately, I wasn’t able to record the meeting due to the need for a Google premium account. Yufei On Tue, Nov 12, 2024 at 10:10 AM Omar Al-Safi <o...@oalsafi.com> wrote: > Thank you! Will try to be there > > On Tue, 12 Nov 2024, 18:55 Yufei Gu, <flyrain...@gmail.com> wrote: > > > Hi Omar, I sent the invitation to dev@polaris.apache.org, as well as > your > > email address. > > > > Yufei > > > > > > On Tue, Nov 12, 2024 at 9:51 AM Omar Al-Safi <o...@oalsafi.com> wrote: > > > > > Thanks Yufei, is it possible to send the invitation to the > > > Polaris google group? > > > > > > Regards, > > > Omar > > > > > > On Tue, Nov 12, 2024 at 6:48 PM Yufei Gu <flyrain...@gmail.com> wrote: > > > > > > > Hi folks, > > > > > > > > We are going to have another sync for table maintenance in Polaris > per > > > > discussion with JB. Here are meeting details: > > > > > > > > Polaris Table maintenance sync > > > > Wednesday, November 13 · 10:00 – 11:00am > > > > Time zone: America/Los_Angeles > > > > Google Meet joining info > > > > Video call link: https://meet.google.com/nyy-ahmn-jqd > > > > > > > > > > > > Yufei > > > > > > > > > > > > On Fri, Nov 8, 2024 at 5:23 PM Yufei Gu <flyrain...@gmail.com> > wrote: > > > > > > > > > Thanks everyone for joining the discussion. Sorry I couldn't record > > the > > > > > session due to a tech issue. Here are meeting notes: > > > > > > > > > > 1. We discussed the boundary between Polaris and the Table > > > Maintenance > > > > > System(TMS). We agreed that they should be separated systems. > > > > > 2. A general agreement on the minimal metadata added to Polaris > to > > > > > support TMS, focusing on essential data needed for > > interoperability. > > > > > 3. A general consensus on option 1 to store metadata as > > > > > catalog/namespace/table properties. We could introduce policy > > > > entities in > > > > > the future for other use cases, like column masking. Will > address > > > two > > > > > feedbacks: > > > > > 1. Caching the table properties in the catalog to reduce IO > > cost. > > > > > 2. Introducing new permissions for table maintenance related > > > > > metadata to prevent any clients with the write permission to > > mess > > > > up with > > > > > them. > > > > > 4. Briefly touched on the communication module between TMS and > > > > > Polaris, as a long-term plan, an event system from Polaris is > > > > necessary, > > > > > not only benefits TMS, but also benefits other systems which > > consume > > > > change > > > > > from Polaris. > > > > > > > > > > Next Steps: > > > > > > > > > > 1. Implement metadata storage as properties > > > > > 1. Design detailed schema for properties > > > > > 2. Figure out a way to be extensible for future maintenance > > > policy > > > > > or customized policies. > > > > > 3. Add new permissions for new properties > > > > > 2. Begin planning for event system > > > > > > > > > > Yufei > > > > > > > > > > > > > > > On Tue, Nov 5, 2024 at 12:25 AM Jean-Baptiste Onofré < > > j...@nanthrax.net> > > > > > wrote: > > > > > > > > > >> Hi Yufei > > > > >> > > > > >> Thanks for scheduling this ! > > > > >> > > > > >> I should be able to join. > > > > >> > > > > >> For the community, will you be able to record ? > > > > >> > > > > >> Regards > > > > >> JB > > > > >> > > > > >> On Mon, Nov 4, 2024 at 10:40 PM Yufei Gu <flyrain...@gmail.com> > > > wrote: > > > > >> > > > > > >> > Hi Folks, > > > > >> > > > > > >> > I've scheduled a community sync to discuss table maintenance in > > > > Polaris > > > > >> > this Thursday at 9 AM PST. Since we didn’t have a chance to dive > > > into > > > > >> this > > > > >> > topic during our last sync, this will be a dedicated session to > > > cover > > > > >> it in > > > > >> > detail. > > > > >> > > > > > >> > *Updates to Note:* I've made some updates to the design > document, > > > > with a > > > > >> > particular focus on the approach for maintenance metadata. The > > > > document > > > > >> now > > > > >> > favors *Option 1*, which involves leveraging table, namespace, > and > > > > >> catalog > > > > >> > properties for maintenance metadata. > > > > >> > > > > > >> > Please review the latest version of the design doc before the > > > meeting, > > > > >> as > > > > >> > it will help us streamline the discussion. > > > > >> > > > > > >> > Looking forward to everyone’s insights! > > > > >> > Video call link: https://meet.google.com/opc-vath-mgb > > > > >> > Design doc: > > > > >> > > > > > >> > > > > > > > > > > https://docs.google.com/document/d/1Pd_mzZcfvnUvcH98IbwsIYf4eryet1lQDfclKYx-t-M/edit?usp=sharing > > > > >> > < > > > > >> > > > > > > > > > > https://www.google.com/url?q=https://docs.google.com/document/d/1Pd_mzZcfvnUvcH98IbwsIYf4eryet1lQDfclKYx-t-M/edit?usp%3Dsharing&sa=D&source=calendar&usd=2&usg=AOvVaw2V3IjIcadea8miDcKKSG9I > > > > >> > > > > > >> > > > > > >> > Yufei > > > > >> > > > > > > > > > > > > > > >
