Hi Alex, all, Thanks for the write-up, and for putting both approaches up rather than arguing them in the abstract.
My preference is #6084, though I recognise the module had support before Dmitri's point. What persuades me is that the shared artifact does not buy the guarantee it looks like it buys: the plugin runs against whatever server it is pointed at, so sharing a JAR aligns the constants with one particular build of Polaris rather than with the server actually answering. If the paths have to be stable within API v1 regardless, a local copy carries the same contract with less machinery — and the new module needing iceberg-core undercuts the "thin" part of the original idea. The real cost is divergence later, not correctness today: someone changes PolarisResourcePaths server-side and the Spark copy quietly does not follow. Nothing in the build ties them together. The Spark copy of PolarisResourcePaths already carries a "keep in sync with the server" note; GenericTableEndpoints does not, and nothing on the polaris-core side points back. Worth making it bidirectional and naming the sibling class in each, so whoever edits one knows the other exists. A CHANGELOG note is worth having either way, since the bundle no longer ships org.apache.polaris.core.* at all — that is visible to anyone who was relying on those classes being present. -Ayush -Ayush On Fri, 13 Nov 2026 at 21:45, Alex Dutra via dev <[email protected]> wrote: > > Hi Dmitri, > > You are right that creating a module just for reusing API URI paths > looks overkill, especially given that the new module needs > iceberg-core, and thus is not a "lightweight" module. > > I created an alternative PR without the separate module (but with a > bit of code duplication instead): > > https://github.com/apache/polaris/pull/6053 : separate module > https://github.com/apache/polaris/pull/6084 : no separate module > > Which one does the community prefer? > > Thanks, > Alex > > On Wed, Nov 11, 2026 at 5:59 PM Dmitri Bourlatchkov via dev > <[email protected]> wrote: > > > > Hi Alex, > > > > Sorry for coming late to this discussion. > > > > I left some comments in GH. However, I would not mind if PR 6053 got merged > > "as is". I think it is still a net positive change. > > > > More specifically (repeating my points from GH), I do not see much value in > > reusing API URI paths across independent runtime artifacts. > > > > The Spark Plugin may run against any server in practice. Reusing Polaris > > modules does not guarantee that paths are aligned. > > > > On the other hand, API paths are expected to remain the same within the API > > major version (v1), so copying constants will not add maintenance overhead. > > > > Cheers, > > Dmitri. > > > > On Wed, Nov 11, 2026 at 4:10 AM Alex Dutra via dev <[email protected]> > > wrote: > > > > > And there it is: > > > > > > https://github.com/apache/polaris/pull/6053 > > > > > > Thanks, > > > Alex > > > > > > On Wed, Nov 11, 2026 at 9:26 AM Alex Dutra <[email protected]> wrote: > > > > > > > > Hi Yufei, > > > > > > > > Thanks, that's very helpful. Since it's just 2 classes, I think it > > > > makes sense to favor option 2. I will open a PR shortly. > > > > > > > > Thanks, > > > > Alex > > > > > > > > On Tue, Nov 10, 2026 at 6:38 PM Yufei Gu <[email protected]> wrote: > > > > > > > > > > Hi Alex, > > > > > > > > > > The Spark client uses only two classes from polaris-core: > > > GenericTableEndpoints and PolarisResourcePaths. > > > > > > > > > > I'd favor option 2: move them into a small polaris-api-catalog-common > > > module under the dir "api", with explicit dependencies. Both Spark and the > > > server could use it, and Spark would no longer depend on core. > > > > > > > > > > Yufei > > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > On Tue, Nov 10, 2026 at 8:30 AM Alex Dutra via dev < > > > [email protected]> wrote: > > > > >> > > > > >> Hi all, > > > > >> > > > > >> While verifying the 1.9.0-rc0 vote I found a few LICENSE issues in > > > the Spark > > > > >> bundle JARs, which I'm fixing separately. Tracing one of them raised > > > a broader > > > > >> question that I'd like to discuss here. > > > > >> > > > > >> Background: > > > > >> > > > > >> The Spark plugin includes polaris-core as: > > > > >> > > > > >> implementation(project(":polaris-core")) { isTransitive = false } > > > > >> > > > > >> See [1] for example. This was introduced in #1933 to "avoid pulling > > > > >> unused dependencies". The bundle JAR then contains all polaris-core > > > classes > > > > >> but none of their runtime dependencies. There is a TODO next to the > > > > >> line about extracting a thin "polaris-rest" module for clients. > > > > >> > > > > >> What I observed: > > > > >> > > > > >> Running jdeps on the bundled org.apache.polaris.core classes shows > > > references to > > > > >> packages that are not in the bundle, e.g. Jackson 3 > > > (tools.jackson.*), Guava, > > > > >> Immutables, the AWS/Azure/GCP SDKs, and > > > org.apache.polaris.core.admin.model > > > > >> (0 classes in the bundle, ~100 references to it). Spark provides some > > > of these > > > > >> at runtime (Jackson 2 annotations, SLF4J), but not Jackson 3 or the > > > cloud SDKs. > > > > >> > > > > >> I have NOT determined which of these classes the Spark client > > > actually loads at > > > > >> runtime, so this may be harmless today. But it means the bundle's > > > correctness > > > > >> depends on the client never touching a class whose dependency is > > > missing, and > > > > >> nothing in the build enforces that. > > > > >> > > > > >> Why not just drop isTransitive = false? > > > > >> > > > > >> polaris-core's runtime classpath resolves to ~300 modules. Bundling > > > them all > > > > >> would make the JAR much larger, risk conflicts with Spark's own > > > Jackson/Guava/ > > > > >> Netty, and require a full review of the bundle LICENSE/NOTICE. > > > > >> > > > > >> Here are our options: > > > > >> > > > > >> 1. Keep as is, and document/test the supported surface (e.g. an > > > integration > > > > >> test that loads every polaris-core class the client uses from the > > > bundle). > > > > >> 2. Extract a thin client-facing module (the existing TODO) containing > > > only what > > > > >> the Spark client needs, with its own explicit dependencies, and > > > bundle that. > > > > >> 3. Drop isTransitive = false and handle conflicts via > > > relocation/exclusions, > > > > >> re-auditing LICENSE/NOTICE. > > > > >> > > > > >> What are your thoughts? In particular, if anyone knows which > > > polaris-core > > > > >> classes the Spark client loads at runtime, that would help scope > > > > >> this. > > > > >> > > > > >> Thanks, > > > > >> Alex > > > > >> > > > > >> [1]: > > > https://github.com/apache/polaris/blob/65befed8407eb8e6a1a5bb0d1a242fcdf5e82c2d/plugins/spark/v3.5/spark/build.gradle.kts#L55-L57 > > >
