markhoerth opened a new issue, #13171:
URL: https://github.com/apache/gravitino/issues/13171
### Version
1.3.0
### Describe what's wrong
The published POMs for the shaded connector runtime artifacts declare
runtime-scoped dependencies on the modules the jar was built from. Runtime
scope is transitive, so resolving the connector by coordinate downloads those
modules and their full dependency trees alongside the shaded jar, placing
unshaded copies of classes the jar already contains, plus unshaded third-party
libraries, on the engine classpath.
`gravitino-spark-connector-runtime-3.5_2.12-1.3.0.pom` declares:
- `org.apache.gravitino:gravitino-client-java-runtime:1.3.0` (runtime)
- `org.apache.gravitino:gravitino-spark-3.5_2.12:1.3.0` (runtime)
- `org.apache.kyuubi:kyuubi-spark-connector-hive_2.12:1.11.0` (runtime)
`gravitino-flink-connector-runtime-1.20_2.12-1.3.0.pom` declares:
- `org.apache.gravitino:gravitino-client-java-runtime:1.3.0` (runtime)
- `org.apache.gravitino:gravitino-flink-1.20_2.12:1.3.0` (runtime)
The Spark runtime jar already bundles the Gravitino client (96 classes under
`org/apache/gravitino/client/`).
Resolving the Spark 3.5 connector with `--packages` on Spark 3.5.3 copies 31
artifacts (46 MB) where one is expected:
- Unshaded Gravitino modules: `gravitino-client-java-runtime`,
`gravitino-client-java`, `gravitino-api`, `gravitino-common`,
`gravitino-catalog-common`, `gravitino-spark-common`,
`gravitino-spark-3.5_2.12`, and `gravitino-spark-3.4_2.12`
- Third-party: `guava-32.1.3-jre`,
`jackson-core/databind/annotations-2.15.2`,
`jackson-datatype-jdk8/jsr310-2.15.2`, `httpclient5-5.4.4`, `httpcore5-5.3.4`,
`httpcore5-h2-5.3.4`, `caffeine-2.9.3`, `commons-lang3-3.14.0`,
`commons-io-2.18.0`, `commons-collections4-4.4`, `slf4j-api-2.0.17`,
`jakarta.validation-api-2.0.2`, and annotation jars
- `kyuubi-spark-connector-hive_2.12-1.11.0`
The point of a shaded runtime artifact is that it is self-contained and
cannot conflict with the libraries the engine already carries. Coordinate
resolution undoes that for every user of `spark.jars.packages` or equivalent.
Users who place the jar on the classpath by hand are not affected.
### Error message and/or stacktrace
None on a trivial query. The failure mode is class and version conflicts
between the unshaded libraries and the engine's own, surfacing when a catalog
loads.
### How to reproduce
/opt/spark/bin/spark-sql --conf spark.jars.ivy=/tmp/ivy \
--packages
org.apache.gravitino:gravitino-spark-connector-runtime-3.5_2.12:1.3.0 \
-e 'select 1' 2>&1 | grep -E 'found |artifacts copied'
ls -1 /tmp/ivy/jars
Expected: one artifact, the runtime jar. Actual: 31 artifacts copied.
### Additional context
Requirement: every `*-connector-runtime-*` artifact is published with a POM
that declares no dependencies the jar already contains, so resolving the
coordinate yields the runtime jar alone. The same applies to all Spark and
Flink runtime variants.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
