Hi Folks, First PR is out. Please take a look. https://github.com/apache/polaris/pull/457/files
Yufei On Wed, Nov 13, 2024 at 5:39 PM Yufei Gu <flyrain...@gmail.com> wrote: > Hi everyone, > > > Thank you for joining the table maintenance discussion today! We made > significant progress, and here are the key takeaways: > > 1. Clarified furthermore and reached consensus on introducing table > maintenance properties in Polaris to support for different TMS and promote > interoperability. > 2. Agreed to proceed with Option 1, which stores metadata as > catalog/namespace/table properties. > 3. Confirmed the new privileges to ensure that maintenance properties > are safeguarded from being altered by clients with existing write access. > 4. Briefly discussed the support for customized maintenance policies . > > Next step: > > 1. Will file maintenance properties related PRs per design > 2. Will add more details for customized policy support. > > *Note*: Unfortunately, I wasn’t able to record the meeting due to the > need for a Google premium account. > > > Yufei > > > On Tue, Nov 12, 2024 at 10:10 AM Omar Al-Safi <o...@oalsafi.com> wrote: > >> Thank you! Will try to be there >> >> On Tue, 12 Nov 2024, 18:55 Yufei Gu, <flyrain...@gmail.com> wrote: >> >> > Hi Omar, I sent the invitation to dev@polaris.apache.org, as well as >> your >> > email address. >> > >> > Yufei >> > >> > >> > On Tue, Nov 12, 2024 at 9:51 AM Omar Al-Safi <o...@oalsafi.com> wrote: >> > >> > > Thanks Yufei, is it possible to send the invitation to the >> > > Polaris google group? >> > > >> > > Regards, >> > > Omar >> > > >> > > On Tue, Nov 12, 2024 at 6:48 PM Yufei Gu <flyrain...@gmail.com> >> wrote: >> > > >> > > > Hi folks, >> > > > >> > > > We are going to have another sync for table maintenance in Polaris >> per >> > > > discussion with JB. Here are meeting details: >> > > > >> > > > Polaris Table maintenance sync >> > > > Wednesday, November 13 · 10:00 – 11:00am >> > > > Time zone: America/Los_Angeles >> > > > Google Meet joining info >> > > > Video call link: https://meet.google.com/nyy-ahmn-jqd >> > > > >> > > > >> > > > Yufei >> > > > >> > > > >> > > > On Fri, Nov 8, 2024 at 5:23 PM Yufei Gu <flyrain...@gmail.com> >> wrote: >> > > > >> > > > > Thanks everyone for joining the discussion. Sorry I couldn't >> record >> > the >> > > > > session due to a tech issue. Here are meeting notes: >> > > > > >> > > > > 1. We discussed the boundary between Polaris and the Table >> > > Maintenance >> > > > > System(TMS). We agreed that they should be separated systems. >> > > > > 2. A general agreement on the minimal metadata added to >> Polaris to >> > > > > support TMS, focusing on essential data needed for >> > interoperability. >> > > > > 3. A general consensus on option 1 to store metadata as >> > > > > catalog/namespace/table properties. We could introduce policy >> > > > entities in >> > > > > the future for other use cases, like column masking. Will >> address >> > > two >> > > > > feedbacks: >> > > > > 1. Caching the table properties in the catalog to reduce IO >> > cost. >> > > > > 2. Introducing new permissions for table maintenance related >> > > > > metadata to prevent any clients with the write permission to >> > mess >> > > > up with >> > > > > them. >> > > > > 4. Briefly touched on the communication module between TMS and >> > > > > Polaris, as a long-term plan, an event system from Polaris is >> > > > necessary, >> > > > > not only benefits TMS, but also benefits other systems which >> > consume >> > > > change >> > > > > from Polaris. >> > > > > >> > > > > Next Steps: >> > > > > >> > > > > 1. Implement metadata storage as properties >> > > > > 1. Design detailed schema for properties >> > > > > 2. Figure out a way to be extensible for future maintenance >> > > policy >> > > > > or customized policies. >> > > > > 3. Add new permissions for new properties >> > > > > 2. Begin planning for event system >> > > > > >> > > > > Yufei >> > > > > >> > > > > >> > > > > On Tue, Nov 5, 2024 at 12:25 AM Jean-Baptiste Onofré < >> > j...@nanthrax.net> >> > > > > wrote: >> > > > > >> > > > >> Hi Yufei >> > > > >> >> > > > >> Thanks for scheduling this ! >> > > > >> >> > > > >> I should be able to join. >> > > > >> >> > > > >> For the community, will you be able to record ? >> > > > >> >> > > > >> Regards >> > > > >> JB >> > > > >> >> > > > >> On Mon, Nov 4, 2024 at 10:40 PM Yufei Gu <flyrain...@gmail.com> >> > > wrote: >> > > > >> > >> > > > >> > Hi Folks, >> > > > >> > >> > > > >> > I've scheduled a community sync to discuss table maintenance in >> > > > Polaris >> > > > >> > this Thursday at 9 AM PST. Since we didn’t have a chance to >> dive >> > > into >> > > > >> this >> > > > >> > topic during our last sync, this will be a dedicated session to >> > > cover >> > > > >> it in >> > > > >> > detail. >> > > > >> > >> > > > >> > *Updates to Note:* I've made some updates to the design >> document, >> > > > with a >> > > > >> > particular focus on the approach for maintenance metadata. The >> > > > document >> > > > >> now >> > > > >> > favors *Option 1*, which involves leveraging table, namespace, >> and >> > > > >> catalog >> > > > >> > properties for maintenance metadata. >> > > > >> > >> > > > >> > Please review the latest version of the design doc before the >> > > meeting, >> > > > >> as >> > > > >> > it will help us streamline the discussion. >> > > > >> > >> > > > >> > Looking forward to everyone’s insights! >> > > > >> > Video call link: https://meet.google.com/opc-vath-mgb >> > > > >> > Design doc: >> > > > >> > >> > > > >> >> > > > >> > > >> > >> https://docs.google.com/document/d/1Pd_mzZcfvnUvcH98IbwsIYf4eryet1lQDfclKYx-t-M/edit?usp=sharing >> > > > >> > < >> > > > >> >> > > > >> > > >> > >> https://www.google.com/url?q=https://docs.google.com/document/d/1Pd_mzZcfvnUvcH98IbwsIYf4eryet1lQDfclKYx-t-M/edit?usp%3Dsharing&sa=D&source=calendar&usd=2&usg=AOvVaw2V3IjIcadea8miDcKKSG9I >> > > > >> > >> > > > >> > >> > > > >> > Yufei >> > > > >> >> > > > > >> > > > >> > > >> > >> >
