Faiz Halde created SPARK-45255:
----------------------------------
Summary: Spark connect client failing with
java.lang.NoClassDefFoundError
Key: SPARK-45255
URL: https://issues.apache.org/jira/browse/SPARK-45255
Project: Spark
Issue Type: New Feature
Components: Connect
Affects Versions: 3.5.0
Reporter: Faiz Halde
I have a very simple repo with the following dependency in `build.sbt`
```
{{libraryDependencies ++= Seq("org.apache.spark" %% "spark-connect-client-jvm"
% "3.5.0")}}
```
A simple application
```
{{object Main extends App {}}
{{ val s = SparkSession.builder().remote("sc://localhost").getOrCreate()}}
{{ s.read.json("/tmp/input.json").repartition(10).show(false)}}
{{}}}
```
But when I run it, I get the following error
```
Exception in thread "main" java.lang.NoClassDefFoundError:
org/sparkproject/connect/client/com/google/common/cache/CacheLoader
at Main$.delayedEndpoint$Main$1(Main.scala:4)
at Main$delayedInit$body.apply(Main.scala:3)
at scala.Function0.apply$mcV$sp(Function0.scala:39)
at scala.Function0.apply$mcV$sp$(Function0.scala:39)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
at scala.App.$anonfun$main$1$adapted(App.scala:80)
at scala.collection.immutable.List.foreach(List.scala:431)
at scala.App.main(App.scala:80)
at scala.App.main$(App.scala:78)
at Main$.main(Main.scala:3)
at Main.main(Main.scala)
Caused by: java.lang.ClassNotFoundException:
org.sparkproject.connect.client.com.google.common.cache.CacheLoader
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 11 more
```
I know the connect does a bunch of shading during assembly so it could be
related to that. This application is not started via spark-submit or anything.
Neither under `SPARK_HOME` ( I guess that's the whole point of connect client )
I followed the doc exactly as described. Can somebody help?
BTW it did work if I copied the exact shading rules in my project but I wonder
if that's the right thing to do?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
