This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 8ad1886 [SPARK-57844] Exclude Hadoop YARN classes from shadow jar
8ad1886 is described below
commit 8ad18867ed92cb36e110235830a1367dc7823266
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Jul 1 08:15:22 2026 -0700
[SPARK-57844] Exclude Hadoop YARN classes from shadow jar
### What changes were proposed in this pull request?
This PR excludes Hadoop YARN classes (`org/apache/hadoop/yarn/**`) from the
operator shadow jar by adding a class-path filter to the `shadowJar` block in
`spark-operator/build.gradle`.
Note that the individual `org.apache.hadoop:hadoop-yarn-api`,
`hadoop-yarn-client`, and `hadoop-yarn-common` artifacts are not present in the
dependency graph. Spark's `spark-core` / `spark-kubernetes` pull in the shaded
`hadoop-client-api` uber-jar, which bundles the YARN classes internally.
Therefore these classes are filtered at packaging time rather than excluded as
Maven modules. This follows the existing precedent of excluding Hadoop
`webapps/**` (HDFS/YARN UI).
### Why are the changes needed?
The operator manages Spark applications on Kubernetes and never uses YARN.
The bundled YARN classes are dead weight in the runtime classpath and shaded
jar. Removing them drops 2,766 classes (~20 MB uncompressed) that originate
from `hadoop-client-api`.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8
Closes #727 from dongjoon-hyun/SPARK-57844.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
spark-operator/build.gradle | 2 ++
1 file changed, 2 insertions(+)
diff --git a/spark-operator/build.gradle b/spark-operator/build.gradle
index 716dda0..760a66b 100644
--- a/spark-operator/build.gradle
+++ b/spark-operator/build.gradle
@@ -96,6 +96,8 @@ shadowJar {
exclude("org/apache/spark/ui/static/**")
// Hadoop webapps (HDFS/YARN UI)
exclude("webapps/**")
+ // Hadoop YARN classes - not used for K8s deployment
+ exclude("org/apache/hadoop/yarn/**")
// Exclude 3rd party metadata to avoid conflicts
exclude("META-INF/maven/**")
// Netty QUIC native libraries - not needed for K8s API server communication
(HTTP/1.1 & HTTP/2)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
