Thanks Robert for the clarification. I think a lot of your points are valid.
This is where I get a bit confused. For the OSI impl, I don't see how it introduces any transitive dependencies. It mainly adds another REST API over the existing Polaris metadata model. It doesn't bring in additional libraries or runtimes, so I'm not sure what dependency concern we're trying to address. I'm also not sure what is meant by "optional Polaris APIs." Once we decide to support an API in Polaris, I don't think the REST contract itself should be optional. We should keep that contract stable so downstream clients aren't easily broken. To me, the question is whether the feature should exist at all, not whether its REST API lives in a separate module. If an operator doesn't need the feature, we already have feature flags to disable the endpoints. I also think it's worth distinguishing between modularizing implementations and modularizing APIs. Modules make a lot of sense for alternative implementations behind a stable interface. For example, the authorizer interface belongs in the core, while different authorizer implementations(e.g., OPA, ranger) can live in separate modules so downstream projects can choose the implementation they want. An OSI REST API is different. It is part of Polaris's external contract. Once that contract exists, I think our priority should be keeping it stable, not making it optional through module boundaries. I completely agree that we should have clean extension points where they add value. I'm just not convinced the OSI REST APIs are a good example that justifies another module. Yufei On Wed, Jun 24, 2026 at 12:41 PM Robert Stupp <[email protected]> wrote: > I agree we should avoid module proliferation for its own sake. > I do not think the rule should be: "Every new feature always gets its own > module". > > To Yufei's question: I think the important distinction is feature flag vs. > assembly boundary. > A feature flag can stop runtime behavior, but it does not remove > compile-time coupling, generated API surface, transitive > dependencies, schema/migration footprint, CDI beans, privileges, or > downstream packaging exposure. > For users building Polaris from Maven artifacts, those are real boundaries. > > I think this also points at a broader design gap: we do not yet have a > clean extension boundary for optional Polaris APIs. > That is why the discussion keeps turning into "core vs separate module". > Ideally, a feature could provide its REST service, endpoint discovery > metadata, persistence SPI, and runtime wiring without requiring unrelated > core/runtime code to know about it directly. > Until we have that kind of boundary, module placement and feature flags are > the tools we have to avoid accidental coupling. > > So I would not frame this as an absolute rule. I would frame it as > criteria: > use a separate module, or at least an explicit boundary, when a feature > adds optional REST endpoints, feature-owned persistence/schema, alternate > downstream implementations, substantial new dependencies, or an API > contract that is still expected to change. > For small coherent changes that are clearly part of the core model, keeping > them in existing modules can be fine. > > For semantic models specifically, I agree they may be closer to core > catalog metadata than metrics or events. > But that only answers the entity-model question. > It does not automatically settle where the REST service, > endpoint discovery, validation, persistence, schema, or future > discovery/query APIs should live, especially while the current v1 shape is > still an initial OSI document registry. > > Robert > > On Wed, Jun 24, 2026 at 8:15 PM Yufei Gu <[email protected]> wrote: > > > > The problem with bundling all new REST APIs into runtime/service is > that > > it _forces_ all downstream builds to expose the new endpoints. > > > > I'm not fully convinced by that argument. > > > > If a downstream project already maintains a custom Polaris build, it > > currently can disable the feature with a feature flag. So the mere > presence > > of an endpoint in runtime/service does not necessarily mean downstream > > deployments must expose it. > > > > I also don't think the metrics reporting work is a strong example for > this > > particular point. IIUC, the metrics query REST API hasn't even been > merged > > yet. Most of the work has been around the SPI, persistence layer, and > > reporting infrastructure. In fact, metrics reporting REST APIs were > present > > in Polaris from the very beginning. Should we move it to a separate > module? > > With that, it's not clear to me how placing a future REST endpoint in > > runtime/service would have materially impacted downstream integrations. > > > > More broadly, I think we should distinguish between different kinds of > > features. Metrics and events are optional capabilities that naturally fit > > separate modules and SPIs. An OSI semantic model is different. It is a > > first class catalog entity, just like a table, view, namespace, or > policy, > > with its own lifecycle, authorization model, and metadata. I would > > therefore expect it to live alongside the rest of the Polaris metadata > > model, while still being disabled via a feature flag if an operator > doesn't > > need it. > > > > To me, the stronger arguments for modularization are separation of > > concerns, clear extension points, or supporting alternate > implementations. > > Those are architectural benefits that can justify additional modules. > But I > > don't think "all new REST APIs should live in separate modules" should > be a > > blanket rule. > > > > Could you elaborate on scenarios where a feature flag would not be > > sufficient and a separate module is required? > > > > Yufei > > > > > > On Wed, Jun 24, 2026 at 10:28 AM Anand Kumar Sankaran via dev < > > [email protected]> wrote: > > > > > All, > > > > > > We (Workday) currently uptake Polaris as Maven dependencies and add > > > functionality on top of the core libraries. It started as > authentication > > > enhancements, but now I have custom listeners for events, custom > > listeners > > > for metrics persistence, a SCIM interface, and there is an ask from a > > data > > > partner to expose semantic information via OSI. Also, till the Iceberg > > > community sorts out FGAC and / or metadata labels, there are asks to > > share > > > some form of hints for enforcing authorization on the data partners' > > side. > > > Our current use-cases are all federated data where our data partners > > > (Salesforce, Snowflake, Databricks, Google and Amazon) federate Workday > > > data through their catalog services. > > > > > > Our timelines are very aggressive, so I have added the functionality > that > > > I need on our custom build, then turn around and contribute it to the > > > Polaris OSS and when we uptake new features, delete the code I added. > > > Thankfully, Quarkus makes this relatively easy. There are some places > > where > > > I can not extend easily and need it to be done in the Polaris OSS first > > > (AWS session tags was one such feature). > > > > > > I want to share a concrete data point from recent work on the metrics > > > reporting feature (PR #4115 / #4756) that illustrates why each of these > > > matters. > > > > > > We initially bundled the reporting SPI, the default no-op/logging > > > implementations, the REST query service, the OpenAPI spec, auth > privilege > > > additions, and JDBC persistence all in a single PR. The review rightly > > > called it out as too broad. Splitting it required two rounds of > > significant > > > rework: extracting the SPI into its own module > > > (extensions/metrics-reports/spi), moving the durable query path and > REST > > > service into a follow-up PR, and re-wiring CDI producers and downstream > > > builds multiple times as the shape changed. > > > > > > A few specifics that map directly to Dmitri's points: > > > > > > - Isolated Gradle modules for REST API code (point 1): Putting the > > metrics > > > query REST service (api/metrics-reports-service) in its own module > meant > > > the core runtime doesn't have to expose those endpoints. Downstream > > builds > > > that don't want the query API simply omit that dependency. We learned > > this > > > the hard way after bundling it into runtime/service initially. > > > - Feature-specific Persistence SPIs (point 5): MetricsPersistence and > > > IcebergMetricsReporter both needed their own SPI layer so downstream > > > implementations (JDBC, no-op, custom) can be swapped without touching > > > polaris-core. Getting this layering right was the bulk of the review > > churn. > > > - Separate PRs for entity changes (point 4): The SQL schema additions > for > > > metrics tables are in PR2 specifically because mixing schema migrations > > > with SPI/API changes, reviewers couldn't easily assess the surface area > > of > > > each concern. > > > > > > The discipline Dmitri is proposing would have saved us multiple rebase > > > cycles and reviewer back-and-forth. > > > > > > — > > > anand > > > > > > From: Dmitri Bourlatchkov <[email protected]> > > > Date: Wednesday, June 24, 2026 at 9:43 AM > > > To: [email protected] <[email protected]> > > > Subject: Re: [DISCUSS] Modular design for new features > > > > > > This Message Is From an External Sender > > > This message came from outside your organization. > > > Report Suspicious< > > > > > > https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1yLOWMt3qBhAE0f6q0IIB7I3WbI-PfK7oi8_Xzj6bOk39QvB5F7ynZ0mNWbr6KptyL4uYInsNJfnl_Eaf4R_c1kMmyaKH_hCt9dIdpgQuj98mWrMS2Y$ > > > > > > > > > > > > > HI Russell, > > > > > > I'm less convinced we need a blanket policy of isolated Gradle modules, > > > feature-specific SPIs, and staged entity-then-REST PRs for every new > > > proposal. > > > > > > > > > Fair enough. That might indeed have been overkill in my initial email. > > > > > > I think it should be fine to combine Entity changes with REST API > changes > > > in the same PR as long as the PR remains reasonably small for ease of > > > review. > > > > > > My concern was mainly driven by the reviewer's perspective, since > > > validating feature boundaries is harder when entity changes are > > > interspersed with REST changes. > > > > > > Most deployments run the standard server release anyway; [...] > > > > > > > > > I cannot agree with that. I believe we have several OSS users with > custom > > > Polaris-based builds. I'm pretty sure all "intergators" have custom > > builds > > > too. This cannot be decided based on undifferentiated deployment counts > > > alone. We need to consider the usability of the project for downstream > > > builds. > > > > > > The problem with bundling all new REST APIs into `runtime/service` is > > that > > > it _forces_ all downstream builds to expose the new endpoints. > > > > > > I think the inconvenience of dealing with multiple source modules is > > rather > > > minor in modern IDEs, while the benefit for downstream project > > flexibility > > > is clear. > > > > > > I wonder if Anand could share his contributor experience based on > [4115]. > > > > > > [4115] > > > > > > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/4115__;!!Iz9xO38YGHZK!6uV890MsdBYLWklaXHA3BkjgieXFNpSHU7qwX6QObHmvU31sQaVFT-4ZCkpnmffb_CH6mgTHwpjV6Rf6rw$ > > > > > > Cheers, > > > Dmitri. > > > > > > On Wed, Jun 24, 2026 at 12:02 PM Russell Spitzer < > > > [email protected]> > > > wrote: > > > > > > > Thanks for raising this, Dmitri. I agree we should be careful about > > > > coupling new features into core and runtime in ways that are hard to > > > > unwind, and I'm on board with feature flags and keeping optional > > behavior > > > > off default paths where that makes sense. > > > > > > > > I'm less convinced we need a blanket policy of isolated Gradle > modules, > > > > feature-specific SPIs, and staged entity-then-REST PRs for every new > > > > proposal. A big reason is contributor experience: most of the > existing > > > > server code lives in a small number of well-understood modules > > > > (polaris-core, polaris-runtime-service, the generated API jars). > > Someone > > > > proposing scan metrics or a semantic layer API can find similar REST > > > > handlers, persistence patterns, and tests without first learning a > > module > > > > taxonomy or deciding which of several new jars their change belongs > in. > > > > > > > > I'd rather optimize for single-module (or few-module) contributions > by > > > > default: add the endpoint, service logic, and tests alongside > existing > > > > similar code. I can understand bundling sets of functionality > together > > > like > > > > if we did want to separate Polaris specific from Iceberg specific > > modules > > > > but I think the codebase is actually more usable with fewer modules > > > rather > > > > than more. If a feature later proves it needs independent > deployment, a > > > > pluggable backend, or a separate schema lifecycle, we can extract it > > > then. > > > > > > > > Most deployments run the standard server release anyway; downstream > > > custom > > > > assembly is possible but doesn't seem to be a widespread pattern > today. > > > > Given that, I'd support lighter guidelines: > > > > > > > > 1. Don't entangle new features into core call paths unnecessarily > > > > 2. Use flags for optional capabilities > > > > 3. Default to adding code where contributors already look > > > > 4. Split into a separate module only when there's a demonstrated need > > > > > > > > Happy to discuss where that line should be. > > > > > > > > On Wed, Jun 24, 2026 at 9:06 AM Dmitri Bourlatchkov < > [email protected]> > > > > wrote: > > > > > > > > > Hi All, > > > > > > > > > > Polaris has been getting many new and interesting proposals lately. > > > This > > > > is > > > > > certainly good for the project. > > > > > > > > > > On the other hand, we need to think about the stability and > usability > > > of > > > > > the system as new features are introduced. > > > > > > > > > > Polaris is currently used in two modes: a) as a ready-made server > for > > > the > > > > > default set of features (source or binary releases) and/or b) as a > > > basis > > > > > for custom downstream builds (from Maven artifacts). > > > > > > > > > > I'd like to propose the following general principles, which I hope > > will > > > > > allow quick feature development without adding risks to either of > the > > > > usage > > > > > avenues. > > > > > > > > > > 1) Put code for new REST API services in isolated Gradle modules. > > > > > > > > > > 2) Wire those services into the runtime/server explicitly, behind > > > feature > > > > > flags where appropriate. > > > > > > > > > > 3) Do not add hard dependencies from runtime/service or > polaris-core > > to > > > > > REST API modules. > > > > > > > > > > 4) If a feature requires new Polaris entity types, add those core > > model > > > > > changes in a dedicated PR so the entity and persistence contract > can > > be > > > > > reviewed on its own. New REST service modules can depend on those > > core > > > > > entities, but existing core call paths should not depend on > > > > > feature-specific entities. > > > > > > > > > > 5) Add new Persistence SPI(s) for non-entity storage (e.g. > > Scan/Commit > > > > > Metrics). Keep new SPI classes in feature-specific Gradle modules. > > > > > > > > > > 6) Use isolated SQL schema definition files for each feature > > involving > > > > > non-entity JDBC persistence. > > > > > > > > > > That is: separate .sql files for Metrics, Events, etc. > > > > > > > > > > Thoughts? > > > > > > > > > > Cheers, > > > > > Dmitri. > > > > > > > > > > > > > > > > > >
