[
https://issues.apache.org/jira/browse/SPARK-45571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liran Y resolved SPARK-45571.
-----------------------------
Resolution: Invalid
I had wrong permissions on the files in the jars directory
> Spark connect executor ignore jars in classpath
> -----------------------------------------------
>
> Key: SPARK-45571
> URL: https://issues.apache.org/jira/browse/SPARK-45571
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 3.5.0
> Reporter: Liran Y
> Priority: Major
>
> I am starting a spark connect server in k8s.
> While trying to access S3 I'm getting the following error from the executor
> {code:java}
> Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most
> recent failure: Lost task 0.3 in stage 0.0 (TID 3) (10.1.0.174 executor 1):
> java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> org.apache.hadoop.fs.s3a.S3AFileSystem not found
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2688)
> at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3431)
> ... {code}
> The driver is able to access S3 without trouble.
> I tried adding the hadoop-aws jar in multiple ways: --packcages, --jars,
> SPARK_EXTRA_CLASSPATH env and adding to the spark Jars folder in my
> dockerfile.
>
> When looking in my executor pod, I'm seeing the classpath is set up properly
> and should have access to the jar located in multiple places.
>
> Only by using the addArtifact API I can add the missing jar.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
