Hi Yufei, I'm not proposing to generalize the type system upfront. Let's start from concrete use cases and introduce/facilitate the extension point where it makes sense.
To clarify the problem: the documentation PR (#5776) tells users to first try modeling a new concept on an existing entity, for example a generic table with a dedicated format and properties. That works and doesn't require any core change. It falls short when users need: - their own privileges, instead of reusing the TABLE_* ones, - their own REST API and path resolution - their own "entity" to be store and manage by Polaris Today the only option is to patch polaris-core. Semantic models show what that costs: besides the extension module itself, adding SEMANTIC_MODEL touched PolarisEntityType, PolarisPrivilege, PolarisAuthorizableOperation, RbacOperationSemantics, PolarisAuthorizerImpl, ResolvePathKey, PolarisResourcePaths, PolarisAdminService, PolarisSecurableMapper and the NoSQL EntityObjMappings. I think we can say it's pretty "down dive" and painful for user extension :) A downstream user has to carry that patch and rebase it on every release. They also have to choose persisted entity type and privilege codes and hope that upstream never allocates them. I looked at what the community has proposed/discussed during the last 18 months. Almost every new catalog concept needed changes to the closed enums in polaris-core: - Policies: new POLICY entity type and privileges - Semantic model (as mentioned earlier): new SEMANTIC_MODEL entity type and privileges - Tags: new TAG entity type and privileges - Lineage: new privileges and operations - Directories: new DIRECTORY subtype of TABLE_LIKE - Catalog branches: new CATALOG_BRANCH entity - Data Sharing: Share, ExternalConsumer and Listing entities The use cases I heard in Glasgow, from end users and integrators: - I want to add a ML model registry in Polaris (models, versions, stages, MLflow integration). Generic tables is not enough because that needs its own privleges (who can register or promote a model to production), not TABLE_* privileges. - I want to add Data Products / Data Contracts to Polaris, that is a governed group of tables with an owner, SLA and lifecycle, and some additional metadata. Generic tables is not enough here because that needs its own privileges (who can publish), and it refers to tables in several namespaces. - I want to manage Connections in Polaris, which is grantable definitions of external sources (JDBC, Kafka, ActiveMQ, Salesforce APIs, ...) with a USAGE privilege. It belongs at catalog or root level, not inside a namespace. USAGE isn't a table privilege, so Generic Table is not enough here. - I'm coming from Unity Catalog, I have to create external locations / storage credentials, with admins who grant "create table under this path". It's at catalog or root level, with its own privileges. Today storage config is buried in the catalog. - I'm an engine or AI-agent vendor and I want to store functions/UDFs that AI agents can call. It needs its own privilege (EXECUTE) with a different lifecycle from a table. - I'm airline company, and I want to extend Polaris to store my business glossary used by my governance team. The terms are "linked" to tables and views. It's its own hierarchy outside namespaces, and its own stewardship privileges. Imho, we can see three groups: - Only own privileges needed (models, data products, functions): the same need as lineage. A way for an extension to add its own privileges would cover most of the request. - A different place in the hierarchy (connections, external locations, glossary): needs a real new entity type. - New grantee types (for example, groups): none of these guesses needs one. That suggests the grantee side of the type system can stay closed. Regards JB On Mon, Oct 19, 2026 at 8:02 PM Yufei Gu <[email protected]> wrote: > Thanks JB, documenting the current extension mechanisms and > limitations is useful. Before introducing an Entity SPI, I’d like to > understand a concrete use case that requires a new entity type and why > the existing extension points are insufficient. Entity types interact > with authorization, persistence, and other core semantics, so making > them extensible has broader implications than replacing an enum. I’d > prefer to work through those requirements first and introduce the > smallest extension point they justify, rather than generalize the type > system for hypothetical downstream needs. > > Yufei > > On Tue, Oct 13, 2026 at 3:03 PM Dmitri Bourlatchkov <[email protected]> > wrote: > > > > Hi JB, > > > > This is the kind of question, that, I believe, will come up more and more > > often in the future. So, I think it is a good idea to allow extending the > > Polaris type system. It will be in line with OO design principles. > > > > As for specific changes required for that, I do not have anything on my > > side, but if anybody has concrete reviewable changes - contributions are > > welcome. > > > > Cheers, > > Dmitri. > > > > On Tue, Oct 13, 2026 at 3:55 AM Jean-Baptiste Onofré <[email protected]> > > wrote: > > > > > Hi folks, > > > > > > During Lakehouse Day/CommunityOverCode here in Glasgow, several Polaris > > > users shared their experiences with building custom Polaris servers and > > > adding extensions. > > > > > > While certain components like Authorizer and Event are straightforward > to > > > extend (by providing a module with a Jandex index and repackaging), > > > creating custom entities is significantly more difficult. Currently, > users > > > have to modify Polaris core code, essentially adding their custom > entities > > > directly to the entities enum. > > > > > > I have created a PR to document the current state of extensions and > custom > > > entities in Polaris: https://github.com/apache/polaris/pull/5776 > > > > > > To open a broader community discussion: what can we do to improve the > > > developer experience for extending and customizing Polaris? Should we > > > consider introducing an "Entity SPI" at some point? > > > > > > I would love to get your thoughts on this. > > > > > > Thanks! > > > > > > Regards, > > > JB > > > >
