I don’t think the fact that the plugin can connect to any server makes the shared module pointless. It won’t guarantee runtime compatibility, but it does keep the plugin and server code aligned when they’re built from the same source.
The only caveat of the new module is that polaris-api-catalog-common would not be self-contained. Any future consumer must also provide Iceberg core. It's already there now. AFAIK, there is no way to avoid the dependency unless Polaris creates its own classes for the following ones: import org.apache.iceberg.catalog.Namespace; import org.apache.iceberg.catalog.TableIdentifier; import org.apache.iceberg.rest.RESTCatalogProperties; import org.apache.iceberg.rest.RESTUtil; import org.apache.iceberg.rest.Endpoint; Do we plan to do so? Yufei On Fri, Nov 13, 2026 at 10:40 AM Ayush Saxena <[email protected]> wrote: > Hi Alex, all, > > Thanks for the write-up, and for putting both approaches up rather > than arguing them in the abstract. > > My preference is #6084, though I recognise the module had support > before Dmitri's point. What persuades me is that the shared artifact > does not buy the guarantee it looks like it buys: the plugin runs > against whatever server it is pointed at, so sharing a JAR aligns the > constants with one particular build of Polaris rather than with the > server actually answering. If the paths have to be stable within API > v1 regardless, a local copy carries the same contract with less > machinery — and the new module needing iceberg-core undercuts the > "thin" part of the original idea. > > The real cost is divergence later, not correctness today: someone > changes PolarisResourcePaths server-side and the Spark copy quietly > does not follow. Nothing in the build ties them together. The Spark > copy of PolarisResourcePaths already carries a "keep in sync with the > server" note; GenericTableEndpoints does not, and nothing on the > polaris-core side points back. Worth making it bidirectional and > naming the sibling class in each, so whoever edits one knows the other > exists. > > A CHANGELOG note is worth having either way, since the bundle no > longer ships org.apache.polaris.core.* at all — that is visible to > anyone who was relying on those classes being present. > > -Ayush > > > -Ayush > > On Fri, 13 Nov 2026 at 21:45, Alex Dutra via dev <[email protected]> > wrote: > > > > Hi Dmitri, > > > > You are right that creating a module just for reusing API URI paths > > looks overkill, especially given that the new module needs > > iceberg-core, and thus is not a "lightweight" module. > > > > I created an alternative PR without the separate module (but with a > > bit of code duplication instead): > > > > https://github.com/apache/polaris/pull/6053 : separate module > > https://github.com/apache/polaris/pull/6084 : no separate module > > > > Which one does the community prefer? > > > > Thanks, > > Alex > > > > On Wed, Nov 11, 2026 at 5:59 PM Dmitri Bourlatchkov via dev > > <[email protected]> wrote: > > > > > > Hi Alex, > > > > > > Sorry for coming late to this discussion. > > > > > > I left some comments in GH. However, I would not mind if PR 6053 got > merged > > > "as is". I think it is still a net positive change. > > > > > > More specifically (repeating my points from GH), I do not see much > value in > > > reusing API URI paths across independent runtime artifacts. > > > > > > The Spark Plugin may run against any server in practice. Reusing > Polaris > > > modules does not guarantee that paths are aligned. > > > > > > On the other hand, API paths are expected to remain the same within > the API > > > major version (v1), so copying constants will not add maintenance > overhead. > > > > > > Cheers, > > > Dmitri. > > > > > > On Wed, Nov 11, 2026 at 4:10 AM Alex Dutra via dev < > [email protected]> > > > wrote: > > > > > > > And there it is: > > > > > > > > https://github.com/apache/polaris/pull/6053 > > > > > > > > Thanks, > > > > Alex > > > > > > > > On Wed, Nov 11, 2026 at 9:26 AM Alex Dutra <[email protected]> > wrote: > > > > > > > > > > Hi Yufei, > > > > > > > > > > Thanks, that's very helpful. Since it's just 2 classes, I think it > > > > > makes sense to favor option 2. I will open a PR shortly. > > > > > > > > > > Thanks, > > > > > Alex > > > > > > > > > > On Tue, Nov 10, 2026 at 6:38 PM Yufei Gu <[email protected]> > wrote: > > > > > > > > > > > > Hi Alex, > > > > > > > > > > > > The Spark client uses only two classes from polaris-core: > > > > GenericTableEndpoints and PolarisResourcePaths. > > > > > > > > > > > > I'd favor option 2: move them into a small > polaris-api-catalog-common > > > > module under the dir "api", with explicit dependencies. Both Spark > and the > > > > server could use it, and Spark would no longer depend on core. > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > On Tue, Nov 10, 2026 at 8:30 AM Alex Dutra via dev < > > > > [email protected]> wrote: > > > > > >> > > > > > >> Hi all, > > > > > >> > > > > > >> While verifying the 1.9.0-rc0 vote I found a few LICENSE issues > in > > > > the Spark > > > > > >> bundle JARs, which I'm fixing separately. Tracing one of them > raised > > > > a broader > > > > > >> question that I'd like to discuss here. > > > > > >> > > > > > >> Background: > > > > > >> > > > > > >> The Spark plugin includes polaris-core as: > > > > > >> > > > > > >> implementation(project(":polaris-core")) { isTransitive = > false } > > > > > >> > > > > > >> See [1] for example. This was introduced in #1933 to "avoid > pulling > > > > > >> unused dependencies". The bundle JAR then contains all > polaris-core > > > > classes > > > > > >> but none of their runtime dependencies. There is a TODO next to > the > > > > > >> line about extracting a thin "polaris-rest" module for clients. > > > > > >> > > > > > >> What I observed: > > > > > >> > > > > > >> Running jdeps on the bundled org.apache.polaris.core classes > shows > > > > references to > > > > > >> packages that are not in the bundle, e.g. Jackson 3 > > > > (tools.jackson.*), Guava, > > > > > >> Immutables, the AWS/Azure/GCP SDKs, and > > > > org.apache.polaris.core.admin.model > > > > > >> (0 classes in the bundle, ~100 references to it). Spark > provides some > > > > of these > > > > > >> at runtime (Jackson 2 annotations, SLF4J), but not Jackson 3 or > the > > > > cloud SDKs. > > > > > >> > > > > > >> I have NOT determined which of these classes the Spark client > > > > actually loads at > > > > > >> runtime, so this may be harmless today. But it means the > bundle's > > > > correctness > > > > > >> depends on the client never touching a class whose dependency is > > > > missing, and > > > > > >> nothing in the build enforces that. > > > > > >> > > > > > >> Why not just drop isTransitive = false? > > > > > >> > > > > > >> polaris-core's runtime classpath resolves to ~300 modules. > Bundling > > > > them all > > > > > >> would make the JAR much larger, risk conflicts with Spark's own > > > > Jackson/Guava/ > > > > > >> Netty, and require a full review of the bundle LICENSE/NOTICE. > > > > > >> > > > > > >> Here are our options: > > > > > >> > > > > > >> 1. Keep as is, and document/test the supported surface (e.g. an > > > > integration > > > > > >> test that loads every polaris-core class the client uses > from the > > > > bundle). > > > > > >> 2. Extract a thin client-facing module (the existing TODO) > containing > > > > only what > > > > > >> the Spark client needs, with its own explicit dependencies, > and > > > > bundle that. > > > > > >> 3. Drop isTransitive = false and handle conflicts via > > > > relocation/exclusions, > > > > > >> re-auditing LICENSE/NOTICE. > > > > > >> > > > > > >> What are your thoughts? In particular, if anyone knows which > > > > polaris-core > > > > > >> classes the Spark client loads at runtime, that would help > scope this. > > > > > >> > > > > > >> Thanks, > > > > > >> Alex > > > > > >> > > > > > >> [1]: > > > > > https://github.com/apache/polaris/blob/65befed8407eb8e6a1a5bb0d1a242fcdf5e82c2d/plugins/spark/v3.5/spark/build.gradle.kts#L55-L57 > > > > >
