Hi Yufei Thanks for bringing this!
I have a few comments: 1. I agree with the challenges you mentioned ... but I would push further :) Imho, I don't see a huge value to have PolarisBaseEntity, including for clarity on the use and code. I would rather prefer that each entity is atomic (not inheriting from a base class), removing PolarisBaseEntity. The overhead is not painful and it would allow us to treat each entity independently. We would not need typeCode with "descriptive & atomic" entities. 2. I would propose to wait for the ongoing persistence effort to rebase on that (I think we can step on each toes :)). Regards JB On Thu, Mar 6, 2025 at 9:31 AM Yufei Gu <flyrain...@gmail.com> wrote: > > Hi Folks, > > As a part of persistence refactor effort, I proposed to separate entities > in Polaris. Currently, PolarisBaseEntity serves as a parent class for 8 > sub-entities(e.g., catalog, namespace, table, principal, etc). While this > design has been effective so far, it introduces several challenges as the > Polaris data model evolves: > > 1. *Adding New Entities:* Entities such as policyEntity and > genericTableEntity may not naturally align with PolarisBaseEntity, > complicating their integration. > 2. *Modifying Existing Entities:* Introducing new fields to a specific > entity (e.g., table) currently necessitates modifying the shared data model > for all 8 sub-entities, increasing the risk of changes. > 3. *Implementing New Access Patterns: *Creating secondary indexes or new > index tables/slices currently impacts all 8 sub-entities due to their > shared structure. > 4. *Separation of Concerns and Modularity: *The current structure makes > it difficult to separate concerns effectively, limiting modularity and > making future maintenance and enhancements more challenging. > > Here is the detail design doc: > https://docs.google.com/document/d/1Jxir9JHZ9g0QNNieRj3y9bHOgNmNuXOvJujI0bZ11mQ/edit?usp=sharing > > I also did a *POC* here https://github.com/apache/polaris/pull/1128, which > is based on a WIP PR 1112. I believe it can help to archive the separation > without a huge refactor. > > Yufei
