Hi Alex,

The Spark client uses only two classes from polaris-core:
GenericTableEndpoints and PolarisResourcePaths.

I'd favor option 2: move them into a small polaris-api-catalog-common
module under the dir "api", with explicit dependencies. Both Spark and the
server could use it, and Spark would no longer depend on core.
Yufei


Yufei


On Tue, Nov 10, 2026 at 8:30 AM Alex Dutra via dev <[email protected]>
wrote:

> Hi all,
>
> While verifying the 1.9.0-rc0 vote I found a few LICENSE issues in the
> Spark
> bundle JARs, which I'm fixing separately. Tracing one of them raised a
> broader
> question that I'd like to discuss here.
>
> Background:
>
> The Spark plugin includes polaris-core as:
>
>   implementation(project(":polaris-core")) { isTransitive = false }
>
> See [1] for example. This was introduced in #1933 to "avoid pulling
> unused dependencies". The bundle JAR then contains all polaris-core classes
> but none of their runtime dependencies. There is a TODO next to the
> line about extracting a thin "polaris-rest" module for clients.
>
> What I observed:
>
> Running jdeps on the bundled org.apache.polaris.core classes shows
> references to
> packages that are not in the bundle, e.g. Jackson 3 (tools.jackson.*),
> Guava,
> Immutables, the AWS/Azure/GCP SDKs, and org.apache.polaris.core.admin.model
> (0 classes in the bundle, ~100 references to it). Spark provides some of
> these
> at runtime (Jackson 2 annotations, SLF4J), but not Jackson 3 or the cloud
> SDKs.
>
> I have NOT determined which of these classes the Spark client actually
> loads at
> runtime, so this may be harmless today. But it means the bundle's
> correctness
> depends on the client never touching a class whose dependency is missing,
> and
> nothing in the build enforces that.
>
> Why not just drop isTransitive = false?
>
> polaris-core's runtime classpath resolves to ~300 modules. Bundling them
> all
> would make the JAR much larger, risk conflicts with Spark's own
> Jackson/Guava/
> Netty, and require a full review of the bundle LICENSE/NOTICE.
>
> Here are our options:
>
> 1. Keep as is, and document/test the supported surface (e.g. an integration
>    test that loads every polaris-core class the client uses from the
> bundle).
> 2. Extract a thin client-facing module (the existing TODO) containing only
> what
>    the Spark client needs, with its own explicit dependencies, and bundle
> that.
> 3. Drop isTransitive = false and handle conflicts via
> relocation/exclusions,
>    re-auditing LICENSE/NOTICE.
>
> What are your thoughts? In particular, if anyone knows which polaris-core
> classes the Spark client loads at runtime, that would help scope this.
>
> Thanks,
> Alex
>
> [1]:
> https://github.com/apache/polaris/blob/65befed8407eb8e6a1a5bb0d1a242fcdf5e82c2d/plugins/spark/v3.5/spark/build.gradle.kts#L55-L57
>

Reply via email to