Liran Y created SPARK-45571:
-------------------------------
Summary: Spark connect executor ignore jars in classpath
Key: SPARK-45571
URL: https://issues.apache.org/jira/browse/SPARK-45571
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 3.5.0
Reporter: Liran Y
I am starting a spark connect server in k8s.
While trying to access S3 I'm getting the following error from the executor
{code:java}
Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most
recent failure: Lost task 0.3 in stage 0.0 (TID 3) (10.1.0.174 executor 1):
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
org.apache.hadoop.fs.s3a.S3AFileSystem not found
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2688)
at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3431)
... {code}
The driver is able to access S3 without trouble.
I tried adding the hadoop-aws jar in multiple ways: --packcages, --jars,
SPARK_EXTRA_CLASSPATH env and adding to the spark Jars folder in my dockerfile.
When looking in my executor pod, I'm seeing the classpath is set up properly
and should have access to the jar located in multiple places.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
