Hi Alex,

Sorry for coming late to this discussion.

I left some comments in GH. However, I would not mind if PR 6053 got merged
"as is". I think it is still a net positive change.

More specifically (repeating my points from GH), I do not see much value in
reusing API URI paths across independent runtime artifacts.

The Spark Plugin may run against any server in practice. Reusing Polaris
modules does not guarantee that paths are aligned.

On the other hand, API paths are expected to remain the same within the API
major version (v1), so copying constants will not add maintenance overhead.

Cheers,
Dmitri.

On Wed, Nov 11, 2026 at 4:10 AM Alex Dutra via dev <[email protected]>
wrote:

> And there it is:
>
> https://github.com/apache/polaris/pull/6053
>
> Thanks,
> Alex
>
> On Wed, Nov 11, 2026 at 9:26 AM Alex Dutra <[email protected]> wrote:
> >
> > Hi Yufei,
> >
> > Thanks, that's very helpful. Since it's just 2 classes, I think it
> > makes sense to favor option 2. I will open a PR shortly.
> >
> > Thanks,
> > Alex
> >
> > On Tue, Nov 10, 2026 at 6:38 PM Yufei Gu <[email protected]> wrote:
> > >
> > > Hi Alex,
> > >
> > > The Spark client uses only two classes from polaris-core:
> GenericTableEndpoints and PolarisResourcePaths.
> > >
> > > I'd favor option 2: move them into a small polaris-api-catalog-common
> module under the dir "api", with explicit dependencies. Both Spark and the
> server could use it, and Spark would no longer depend on core.
> > >
> > > Yufei
> > >
> > >
> > > Yufei
> > >
> > >
> > > On Tue, Nov 10, 2026 at 8:30 AM Alex Dutra via dev <
> [email protected]> wrote:
> > >>
> > >> Hi all,
> > >>
> > >> While verifying the 1.9.0-rc0 vote I found a few LICENSE issues in
> the Spark
> > >> bundle JARs, which I'm fixing separately. Tracing one of them raised
> a broader
> > >> question that I'd like to discuss here.
> > >>
> > >> Background:
> > >>
> > >> The Spark plugin includes polaris-core as:
> > >>
> > >>   implementation(project(":polaris-core")) { isTransitive = false }
> > >>
> > >> See [1] for example. This was introduced in #1933 to "avoid pulling
> > >> unused dependencies". The bundle JAR then contains all polaris-core
> classes
> > >> but none of their runtime dependencies. There is a TODO next to the
> > >> line about extracting a thin "polaris-rest" module for clients.
> > >>
> > >> What I observed:
> > >>
> > >> Running jdeps on the bundled org.apache.polaris.core classes shows
> references to
> > >> packages that are not in the bundle, e.g. Jackson 3
> (tools.jackson.*), Guava,
> > >> Immutables, the AWS/Azure/GCP SDKs, and
> org.apache.polaris.core.admin.model
> > >> (0 classes in the bundle, ~100 references to it). Spark provides some
> of these
> > >> at runtime (Jackson 2 annotations, SLF4J), but not Jackson 3 or the
> cloud SDKs.
> > >>
> > >> I have NOT determined which of these classes the Spark client
> actually loads at
> > >> runtime, so this may be harmless today. But it means the bundle's
> correctness
> > >> depends on the client never touching a class whose dependency is
> missing, and
> > >> nothing in the build enforces that.
> > >>
> > >> Why not just drop isTransitive = false?
> > >>
> > >> polaris-core's runtime classpath resolves to ~300 modules. Bundling
> them all
> > >> would make the JAR much larger, risk conflicts with Spark's own
> Jackson/Guava/
> > >> Netty, and require a full review of the bundle LICENSE/NOTICE.
> > >>
> > >> Here are our options:
> > >>
> > >> 1. Keep as is, and document/test the supported surface (e.g. an
> integration
> > >>    test that loads every polaris-core class the client uses from the
> bundle).
> > >> 2. Extract a thin client-facing module (the existing TODO) containing
> only what
> > >>    the Spark client needs, with its own explicit dependencies, and
> bundle that.
> > >> 3. Drop isTransitive = false and handle conflicts via
> relocation/exclusions,
> > >>    re-auditing LICENSE/NOTICE.
> > >>
> > >> What are your thoughts? In particular, if anyone knows which
> polaris-core
> > >> classes the Spark client loads at runtime, that would help scope this.
> > >>
> > >> Thanks,
> > >> Alex
> > >>
> > >> [1]:
> https://github.com/apache/polaris/blob/65befed8407eb8e6a1a5bb0d1a242fcdf5e82c2d/plugins/spark/v3.5/spark/build.gradle.kts#L55-L57
>

Reply via email to