Hi Yufei, Agreed. The purpose here is not to skip that work: an extension would still have to define its authorization, hierarchy, persistence and lifecycle semantics.
The goal is to foster more community discussion on how we can facilitate that, perhaps by doing that in its own module, through stable extension points, instead of patching enums in polaris-core. I would like to hear more from the community. For us, working in Polaris, it's not a problem. But I'm not sure the whole community is happy about that. My point in #5776 is already to provide a clear way to do extensions today. While doing that (because several users asked me how they can create their own extensions) I realized why people are struggling. If this discussion ends with "it's fine and good as is", I'm fine with that. Let's see what the community thinks about that. Regards JB On Tue, Oct 20, 2026 at 6:41 PM Yufei Gu <[email protected]> wrote: > Hi JB, > > Thanks for sharing these use cases. The distinction between custom > privileges and new entity types is helpful. > > I agree that maintaining downstream patches is painful. However, I > don't think we can avoid the underlying work required to introduce a > new first class entity. Authorization, hierarchy, persistence, and > lifecycle semantics still need to be defined and integrated. I'm open > to ideas to make Polaris more extensible, but we don't need to work > out the full design in this email thread. I'd suggest putting together > a formal design proposal, starting with one concrete use case and > showing how it would work through the proposed extension points. That > would give us something concrete to review, including what the > extension owns, what core guarantees, and how persisted types and > privileges remain compatible across upgrades. > > Yufei > > On Tue, Oct 20, 2026 at 12:42 AM Jean-Baptiste Onofré <[email protected]> > wrote: > > > > Hi Yufei, > > > > I'm not proposing to generalize the type system upfront. > > Let's start from concrete use cases and introduce/facilitate the > extension > > point where it makes sense. > > > > To clarify the problem: the documentation PR (#5776) tells users to first > > try modeling a new concept on an existing entity, for example a generic > > table with a dedicated format and properties. That works and doesn't > > require any core change. It falls short when users need: > > - their own privileges, instead of reusing the TABLE_* ones, > > - their own REST API and path resolution > > - their own "entity" to be store and manage by Polaris > > > > Today the only option is to patch polaris-core. Semantic models show what > > that costs: besides the extension module itself, adding SEMANTIC_MODEL > > touched PolarisEntityType, PolarisPrivilege, > PolarisAuthorizableOperation, > > RbacOperationSemantics, PolarisAuthorizerImpl, ResolvePathKey, > > PolarisResourcePaths, PolarisAdminService, PolarisSecurableMapper and the > > NoSQL EntityObjMappings. I think we can say it's pretty "down dive" and > > painful for user extension :) > > A downstream user has to carry that patch and rebase it on every > > release. They also have to choose persisted entity type and privilege > codes > > and hope that upstream never allocates them. > > > > I looked at what the community has proposed/discussed during the last 18 > > months. Almost every new catalog concept needed changes to the closed > enums > > in polaris-core: > > - Policies: new POLICY entity type and privileges > > - Semantic model (as mentioned earlier): new SEMANTIC_MODEL entity type > and > > privileges > > - Tags: new TAG entity type and privileges > > - Lineage: new privileges and operations > > - Directories: new DIRECTORY subtype of TABLE_LIKE > > - Catalog branches: new CATALOG_BRANCH entity > > - Data Sharing: Share, ExternalConsumer and Listing entities > > > > The use cases I heard in Glasgow, from end users and integrators: > > - I want to add a ML model registry in Polaris (models, versions, stages, > > MLflow integration). Generic tables is not enough because that needs its > > own privleges (who can register or promote a model to production), not > > TABLE_* privileges. > > - I want to add Data Products / Data Contracts to Polaris, that is a > > governed group of tables with an owner, SLA and lifecycle, and some > > additional metadata. Generic tables is not enough here because that needs > > its own privileges (who can publish), and it refers to tables in several > > namespaces. > > - I want to manage Connections in Polaris, which is grantable definitions > > of external sources (JDBC, Kafka, ActiveMQ, Salesforce APIs, ...) with a > > USAGE privilege. It belongs at catalog or root level, not inside a > > namespace. USAGE isn't a table privilege, so Generic Table is not enough > > here. > > - I'm coming from Unity Catalog, I have to create external locations / > > storage credentials, with admins who grant "create table under this > path". > > It's at catalog or root level, with its own privileges. Today storage > > config is buried in the catalog. > > - I'm an engine or AI-agent vendor and I want to store functions/UDFs > that > > AI agents can call. It needs its own privilege (EXECUTE) with a different > > lifecycle from a table. > > - I'm airline company, and I want to extend Polaris to store my business > > glossary used by my governance team. The terms are "linked" to tables and > > views. It's its own hierarchy outside namespaces, and its own stewardship > > privileges. > > > > Imho, we can see three groups: > > - Only own privileges needed (models, data products, functions): the same > > need as lineage. A way for an extension to add its own privileges would > > cover most of the request. > > - A different place in the hierarchy (connections, external locations, > > glossary): needs a real new entity type. > > - New grantee types (for example, groups): none of these guesses needs > one. > > That suggests the grantee side of the type system can stay closed. > > > > Regards > > JB > > > > On Mon, Oct 19, 2026 at 8:02 PM Yufei Gu <[email protected]> wrote: > > > > > Thanks JB, documenting the current extension mechanisms and > > > limitations is useful. Before introducing an Entity SPI, I’d like to > > > understand a concrete use case that requires a new entity type and why > > > the existing extension points are insufficient. Entity types interact > > > with authorization, persistence, and other core semantics, so making > > > them extensible has broader implications than replacing an enum. I’d > > > prefer to work through those requirements first and introduce the > > > smallest extension point they justify, rather than generalize the type > > > system for hypothetical downstream needs. > > > > > > Yufei > > > > > > On Tue, Oct 13, 2026 at 3:03 PM Dmitri Bourlatchkov <[email protected]> > > > wrote: > > > > > > > > Hi JB, > > > > > > > > This is the kind of question, that, I believe, will come up more and > more > > > > often in the future. So, I think it is a good idea to allow > extending the > > > > Polaris type system. It will be in line with OO design principles. > > > > > > > > As for specific changes required for that, I do not have anything on > my > > > > side, but if anybody has concrete reviewable changes - contributions > are > > > > welcome. > > > > > > > > Cheers, > > > > Dmitri. > > > > > > > > On Tue, Oct 13, 2026 at 3:55 AM Jean-Baptiste Onofré < > [email protected]> > > > > wrote: > > > > > > > > > Hi folks, > > > > > > > > > > During Lakehouse Day/CommunityOverCode here in Glasgow, several > Polaris > > > > > users shared their experiences with building custom Polaris > servers and > > > > > adding extensions. > > > > > > > > > > While certain components like Authorizer and Event are > straightforward > > > to > > > > > extend (by providing a module with a Jandex index and repackaging), > > > > > creating custom entities is significantly more difficult. > Currently, > > > users > > > > > have to modify Polaris core code, essentially adding their custom > > > entities > > > > > directly to the entities enum. > > > > > > > > > > I have created a PR to document the current state of extensions and > > > custom > > > > > entities in Polaris: https://github.com/apache/polaris/pull/5776 > > > > > > > > > > To open a broader community discussion: what can we do to improve > the > > > > > developer experience for extending and customizing Polaris? Should > we > > > > > consider introducing an "Entity SPI" at some point? > > > > > > > > > > I would love to get your thoughts on this. > > > > > > > > > > Thanks! > > > > > > > > > > Regards, > > > > > JB > > > > > > > > >
