This is an automated email from the ASF dual-hosted git repository.
rexxiong pushed a commit to branch branch-0.6
in repository https://gitbox.apache.org/repos/asf/celeborn.git
The following commit(s) were added to refs/heads/branch-0.6 by this push:
new 215b51ae1 [CELEBORN-2022] Spark4 Client should package commons-io
215b51ae1 is described below
commit 215b51ae15775a5d507b992efe19889f3302e2d5
Author: sychen <[email protected]>
AuthorDate: Wed Jun 4 17:56:45 2025 +0800
[CELEBORN-2022] Spark4 Client should package commons-io
### What changes were proposed in this pull request?
### Why are the changes needed?
```java
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/celeborn/shaded/org/apache/commons/io/output/ByteArrayOutputStream
at
org.apache.spark.shuffle.celeborn.CelebornShuffleDataIO.<init>(CelebornShuffleDataIO.java:41)
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:75)
at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:53)
at
java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
at
java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
at
org.apache.spark.util.Utils$.$anonfun$loadExtensions$1(Utils.scala:2834)
at scala.collection.immutable.List.flatMap(List.scala:294)
at scala.collection.immutable.List.flatMap(List.scala:79)
at org.apache.spark.util.Utils$.loadExtensions(Utils.scala:2826)
at
org.apache.spark.shuffle.ShuffleDataIOUtils$.loadShuffleDataIO(ShuffleDataIOUtils.scala:35)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:633)
at
org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:3055)
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
local test
Closes #3304 from cxzl25/CELEBORN-2022.
Authored-by: sychen <[email protected]>
Signed-off-by: Shuang <[email protected]>
(cherry picked from commit c0512c67fd4352f90067af6b2e07384fd0a44230)
Signed-off-by: Shuang <[email protected]>
---
client-spark/spark-4-shaded/pom.xml | 1 +
1 file changed, 1 insertion(+)
diff --git a/client-spark/spark-4-shaded/pom.xml
b/client-spark/spark-4-shaded/pom.xml
index 28589c7f8..633eabbec 100644
--- a/client-spark/spark-4-shaded/pom.xml
+++ b/client-spark/spark-4-shaded/pom.xml
@@ -77,6 +77,7 @@
<include>io.netty:*</include>
<include>org.apache.commons:commons-lang3</include>
<include>org.roaringbitmap:RoaringBitmap</include>
+ <include>commons-io:commons-io</include>
</includes>
</artifactSet>
<filters>
