james-willis opened a new issue, #3095: URL: https://github.com/apache/sedona/issues/3095
## Expected behavior The shaded fat jars (`sedona-spark-shaded`, `sedona-flink-shaded`, `sedona-snowflake`) should never bundle unrelocated Scala toolchain classes (`scala-library`, `scala-reflect`, `scala-compiler`). The runtime (Spark/Flink) provides its own Scala distribution, and an unrelocated copy inside the fat jar shadows it. ## Actual behavior The maven-shade-plugin `artifactSet` in all three shaded modules only excludes `org.scala-lang:scala-library`: - `spark-shaded/pom.xml` - `flink-shaded/pom.xml` - `snowflake/pom.xml` If any dependency (present or future) transitively pulls in `org.scala-lang:scala-reflect` or `scala-compiler` at compile scope, it gets silently bundled unrelocated into the shaded jar. The published Sedona 1.9.0 shaded jars are currently clean (verified: no `scala/reflect/` or `scala/tools/` entries), so this is preventive hardening rather than an active bug upstream. However, we hit the failure mode for real in a downstream build derived from this pom: a transitive dependency chain (`geomesa-z3 -> scala-logging -> scala-reflect:2.13.8`) silently bundled an old unrelocated `scala.reflect.internal.*` (including all 16 `StdNames` classes) into the shaded jar. Under `spark-submit` that stale copy shadowed Spark's `scala-reflect-2.13.16`, and any Scala-compiler user on the classpath broke — e.g. almond/Ammonite kernels failed with: ``` java.lang.NoSuchMethodError: scala.reflect.internal.StdNames$TypeNames.PermittedSubclassesATTR() ``` (`PermittedSubclassesATTR` was added to `StdNames` in Scala 2.13.12, so any pre-2.13.12 `scala-reflect` copy shadowing a newer one produces this.) ## Proposed fix Change the exclude in all three shaded modules from: ```xml <exclude>org.scala-lang:scala-library</exclude> ``` to: ```xml <exclude>org.scala-lang:*</exclude> ``` This drops `scala-library`, `scala-reflect`, and `scala-compiler` from the fat jar. The groupId match is exact, so `org.scala-lang.modules:scala-collection-compat` (which Spark does not ship and must stay bundled) is unaffected. ## Steps to reproduce the problem Add any compile-scope dependency that transitively pulls `org.scala-lang:scala-reflect` to `common` or `spark/common`, build `spark-shaded`, and observe unrelocated `scala/reflect/**` classes in the jar (`unzip -l ... | grep scala/reflect`). ## Settings Sedona version = 1.9.0 (master) API type = Scala, Java Apache Spark version = 3.5 / 4.0 Environment = any -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
