Hi, This has been fixed here: https://github.com/apache/spark/pull/23546. Will be available with Spark 3.0.0
Best, Stavros On Wed, Jun 5, 2019 at 11:18 PM pacuna <[email protected]> wrote: > I'm trying to run a sample code that reads a file from s3 so I need the aws > sdk and aws hadoop dependencies. > If I assemble these deps into the main jar everything works fine. But when > I > try using --packages, the deps are not seen by the pods. > > This is my submit command: > > spark-submit > --master k8s://https://xx.xx.xx.xx > --class "SimpleApp" > --deploy-mode cluster > --conf spark.kubernetes.container.image=docker.io/pacuna/spark:0.2 > --conf > spark.kubernetes.authenticate.driver.serviceAccountName=spark-test-user > --packages > com.amazonaws:aws-java-sdk:1.7.4,org.apache.hadoop:hadoop-aws:2.7.3 > --conf spark.hadoop.fs.s3a.access.key=... > --conf spark.hadoop.fs.s3a.secret.key=... > https://xxxxx/simple-project_2.11-1.0.jar > > And the error I'm getting in the driver pod is: > > 19/06/05 20:13:50 ERROR SparkContext: Failed to add > > file:///home/dev/.ivy2/jars/com.fasterxml.jackson.core_jackson-core-2.2.3.jar > to Spark environment > java.io.FileNotFoundException: Jar > /home/dev/.ivy2/jars/com.fasterxml.jackson.core_jackson-core-2.2.3.jar not > found > > I'm getting that error for all the deps jars needed. > > Any ideas? > > Thanks. > > > > > -- > Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ > > --------------------------------------------------------------------- > To unsubscribe e-mail: [email protected] > >
