Hi all, I think two largely independent questions are getting mixed together here: module boundaries and compatibility guarantees.
`polaris-core` and `polaris-runtime-service` are already very broad aggregation modules. Treating either of them as the default home for every new functional area compounds the dependency and coupling problems. I do not think that every interface needs its own `X-spi` module. However, a coherent functional area should normally have a separate module when it has distinct dependencies, implementations, framework bindings, optional assembly, or lifecycle requirements. In particular, framework-neutral contracts should not have to depend on the complete Quarkus runtime merely because that is where their current implementation lives. For terminology, API versus SPI describes the direction of the contract: * an API is called by a consumer, while * an SPI is implemented by a provider. That distinction says nothing by itself about whether the contract is internal, public, experimental, or stable. Both API and SPI are useful terms for internal contracts as well. The documentation for a package or module can state its intended audience and compatibility guarantees. I would therefore distinguish at least: * Internal APIs and SPIs, which can evolve together with their callers and implementations. * Downstream-facing SPIs, where compatibility is valuable, but deliberate breaking changes remain possible and should be documented. * Externally consumed APIs and wire protocols, which usually need stronger compatibility and versioning guarantees. A broader term such as "public contract" can be useful as an inventory, but I do not think all of those contract types should automatically receive the same evolution policy. In particular, a strict backward-compatibility requirement for every downstream-facing SPI is neither practical nor always correct. When an SPI changes because implementations must satisfy a new semantic contract or a new correctness or authorization invariant, preserving source compatibility through a default method may merely hide an incompatible implementation. A compilation failure can be the safer outcome. That does not make SPI breakage free. It should be intentional and documented. When migration is non-obvious, the release notes should explain the required downstream changes. I would treat compatibility as one design consideration, not as a strict requirement that overrides the semantics of the contract or necessary architectural evolution. The existing Polaris Evolution documentation [1] already says that Java classes may change between releases. I think that should remain the baseline for SPIs. On the concrete question in the original post, I agree with Dmitri that source-breaking SPI changes can be appropriate. If a particular SPI promises stronger compatibility, that guarantee should be explicit for that SPI rather than inferred from the terms "SPI" or "public contract." Module placement should likewise follow concrete dependency and assembly boundaries rather than a universal rule. That seems like sufficient general guidance without imposing a single compatibility or module-layout policy on every SPI. Cheers, Robert [1] https://polaris.apache.org/releases/1.6.0/evolution/#using-polaris-as-a-build-time-dependency On Tue, Jul 21, 2026 at 1:42 AM Dmitri Bourlatchkov <[email protected]> wrote: > HI EJ, > > Thanks for following up on PR 5052! > > > 1. The API/SPI split hinges on "when multiple alternatives exist",[...] > > I totally agree with your point. Alternatives here would mean something > that can be a plugin. Even if Polaris itself has only one implementation, > if we expect downstream builds to provide an alternative impl., it will > fit the "alternatives exist" principle. > > > 2. Same question on the placement rule, you allow the SPI to live inside > > X when X is already guaranteed to be a dependency of its own > > implementations, curious what governs it when that's true for some > > implementers and not others. > > From my POV, both putting SPI class in X or in X-spi is ok. > > An SPI implementation works only when invoked from X. Therefore, X > must be present at runtime anyway. Having a dependency from the SPI > implementation on X at compile time is mildly intrusive, but not too > critical from my POV. The positive side is that it reduces the number > of modules in the system. > > I think in the current codebase, this principle applies mainly to > runtime/service. > > If X has a lot of external dependencies that do not make sense in all > SPI implementations, then of course, having a separate X-spi module > will make more sense. > > Also if the SPI can be called from multiple modules, a separate X-spi > module is advisable too (although I do not think we have these cases > in current code). > > > 3. And on point 4, the compile-error safety net assumes downstream > > rebuilds against new releases before it matters, not every consumer does. > > Probably its own thread though, I'm thinking of more on the client and > > server side (schema versioning on the wire, not something to hash out > here > > I do not know how it might be possible to provide a custom Polaris SPI > impl. > without a downstream build, ATM. > > Polaris Servers are based on Quarkus, which performs all introspection > tasks > for CDI purposes at build time. > > Plugging something in without recompilation is possible via ServiceLoader, > and Polaris has a few use cases for it. Still, a custom SPI provider would > do well to cover those implementations by integration tests. The test > should catch incompatibilities even if recompilation was not performed in > the > downstream build. > > > Polaris's own dependency graph, every extensions/* implementation > > already has to depend on polaris-core, lands on keeping SPIs like > > PolarisAuthorizer and PolarisMetaStoreManager inside core rather than > > split out. > > From my POV the dependency chain has to start from the SPI caller. > > PolarisMetaStoreManager is called from other core classes in a few places, > so I guess it fits into polaris-core. Plus it was there historically. > > PolarisAuthorizer is not called from core. So by my logic it is not a > "core SPI", indeed :) > > PolarisAuthorizer is a "service SPI" (used by runtime/service). > > I do not mind keeping it in polaris-core for now since it has > historically been there. I mean this discussion to apply to new > code mostly. > > > And on GenericTableEndpoints, the thing I raised on the same PR, > > it's not an SPI either [...] > > Could you give me a pointer to that? I might have lost context here :) > > > [...] Quarkus being today's default impl, a hypothetical Spring > > binding being another, it points at a real in-between layer: > > framework-agnostic REST-serving logic that isn't quite core and > > isn't quite runtime either. > > Good point. Much depends on how we expect custom builds to > function. > > Currently Quarkus is assumed to be the platform for Polaris. > > If we (hypothetically) wanted to support different platforms, I'd > guess specific REST API service code would not be 100% portable > due to framework differences. A sub-set of frameworks may be > interchangeable, but not all, I'm sure. > > (side note: this was very apparent in the migration from Dropwizard > to Quarkus early in the project's history. The old code is still in git). > > So the REST service layer binding to HTTP would have to be implemented > separately in each case and then call into a shared layer through an API. > The shared layer would call Persistence and AuthZ through an SPI. > > The role of polaris-core is to provide a shared Entity model (with its > own API) and a shared SPI for persisting those entities. > > Does that make sense? > > > [...] this would be a build-time, different-deployable choice, > > a different axis than what you described [...] > > Loading SPI implementations without rebuilding is possible. I've done > that before :) However, I can say from experience that it is a lot more > difficult to support in a backward-compatible way and usually not worth > the effort. > > Requiring a rebuild (with CI) to catch SPI discrepancies is more robust > and usually leads to more clarity in downstream components. Check > out the related message from Anand [1]. > > Quarkus enhances this approach by handling CDI wiring and related > build-time checks. > > [1] https://lists.apache.org/thread/s26pj6vp95gq13ltmcvj72s9f1870foj > > Cheers, > Dmitri. > > On Mon, Jul 20, 2026 at 12:42 PM EJ Wang <[email protected]> > wrote: > > > Hi Dmitri, > > > > Following up here since it connects with PR5052, which just merged. > > CatalogConfigEndpointContributor landed in runtime/service for now. On > that > > PR I argued core is too broad and runtime is too Quarkus-specific for it, > > and suggested punting the real "where does this go long-term" question > to a > > separate issue rather than blocking the merge on it. You agreed, merged > as > > is. I don't see that issue filed yet, so figured this thread works as > well > > as any. > > > > A few things weren't fully clear to me in the writeup: > > > > 1. The API/SPI split hinges on "when multiple alternatives exist", is > > that about how many alternatives exist today, or design intent to > allow > > more later? A plugin point with exactly one default implementation, > > built that way on purpose, still reads like an SPI to me even with a > > single > > implementer. > > 2. Same question on the placement rule, you allow the SPI to live > inside > > X when X is already guaranteed to be a dependency of its own > > implementations, curious what governs it when that's true for some > > implementers and not others. > > 3. And on point 4, the compile-error safety net assumes downstream > > rebuilds against new releases before it matters, not every consumer > > does. > > Probably its own thread though, I'm thinking of more on the client and > > server side (schema versioning on the wire, not something to hash out > > here). > > > > Separately, curious how general the placement rule is meant to be. > > Polaris's own dependency graph, every extensions/* implementation already > > has to depend on polaris-core, lands on keeping SPIs like > PolarisAuthorizer > > and PolarisMetaStoreManager inside core rather than split out. > > > > Is "ideally X-spi per module" the default shape with the in-X case being > > rare, or are both equally "ideal" depending on which way the dependency > > graph points, no inherent preference either way? > > And on GenericTableEndpoints, the thing I raised on the same PR, it's not > > an SPI either, nobody implements a wire-format constant. Does your model > > have a read on where that kind of thing goes, or is it genuinely outside > > what you're describing here? > > > > One more thing, more analogy than a literal claim, not saying the > > REST-serving layer is an SPI. But if you squint at it that way, Quarkus > > being today's default impl, a hypothetical Spring binding being another, > it > > points at a real in-between layer: framework-agnostic REST-serving logic > > that isn't quite core and isn't quite runtime either, the same shape as > the > > GenericTableEndpoints question, just one layer up. Your examples are all > > in-process, CDI-selected at runtime, this would be a build-time, > > different-deployable choice, a different axis than what you described. > Not > > asking you to settle whether it's an SPI, more whether that in-between > > layer is something your placement thinking has a slot for. > > > > -ej > > > > On Mon, Jul 13, 2026 at 5:42 PM Dmitri Bourlatchkov <[email protected]> > > wrote: > > > > > Hi All, > > > > > > I'm sending this email hoping to confirm alignment of the Polaris > > community > > > members on SPI development principles in this project. I do not think > we > > > need strong and formal rules / guidelines as long as the community has > > > consensus on the general approach. > > > > > > Recap: SPI = Service Provider Interface > > > > > > * What is the difference between API and SPI? > > > > > > API is meant to be invoked by a client on a library (or service) in > order > > > to cause the library (service) to perform a certain action. > > > > > > SPI is meant to be implemented by a library / service plugin in order > to > > > provide a method for performing certain actions when multiple > > alternatives > > > exist. > > > > > > Example from Polaris: > > > > > > IcebergRestCatalogApiService is an API that represents the IRC spec in > > > terms of java classes. > > > > > > Authenticator is an SPI that defines what Polaris code requires an > > > authorization framework to implement. The default impl. > > > is DefaultAuthenticator, OPA and Ranger implementation and alternatives > > > that can be plugged in via CDI + configuration. > > > > > > * Where should SPI classes be located normally? > > > > > > Without considering the legacy code in Polaris, I believe SPI classes > > > should be located such that they require minimal depdency leaks into > the > > > module implementing the SPI. Ideally all SPI classes used by module X > > > should be located in a module "X-spi" to convey that they are > associated > > > with X. > > > > > > This is not to say that existing code must be refactored immediately to > > fit > > > these patterns. This is a proposal for future evolution. > > > > > > If module X can be expected to be a dependency of its SPI > > implementations, > > > the SPI classes can be placed inside X itself. > > > > > > * Should SPI classe be in polaris-core? > > > > > > If polaris-core requires defines a plugin point, the SPI for that can > > > certainly be in polaris-core. > > > > > > BasePersistence is an example of a core SPI. > > > > > > * Can SPI interfaces have breaking changes from the java language > > > perspective? > > > > > > Yes. Unline API, SPI interfaces define what plugins must implement. not > > > what clients rely on. > > > > > > For example, adding a parameter to an SPI method will cause a > compilation > > > error downstream, but it's ok. This is a signal to the downstream SPI > > > implementation that it is expected to process the new parameter. It is > a > > > good breakage in the sense that it will inform the downstram > > implementation > > > and prevent invalid behaviour that might occur as a result of ignoring > > the > > > new parameter. > > > > > > In many cases this is not a hard failure downstream. Normally, the > > > downstream project will get those compilation errors inside a PR that > > > updates its Polaris dependencies. It will not be a deployment time > error > > > (which is common with service API changes). > > > > > > Thoughts? > > > > > > Thanks, > > > Dmitri. > > > > > >
