Re: Spark kubernetes s3 connectivity issue

actually can you create an Uber jar file in a conventional way using those two hadoop versions? You have HADOOP_AWS_VERSION=3.3.0 besides 3.2. HTH view my Linkedin profile https://en.everybodywiki.com/Mich_Talebzadeh

Re: Spark kubernetes s3 connectivity issue

I understand what you are saying . However, I am not sure how to implement when i create a docker image using spark 3.2.1 with hadoop 3.2 which has guava jar already added as part of distribution. On Tue, Feb 15, 2022, 01:17 Mich Talebzadeh wrote: > Hi Raj, > > I found the old email. That is

Re: Spark kubernetes s3 connectivity issue

Hi Raj, I found the old email. That is what I did but it is 2018 stuff. The email says I sorted out this problem. I rewrote the assembly with shade rules to avoid old jar files as follows: lazy val root = (project in file(".")). settings( name := "${APPLICATION}", version := "1.0",

Re: Spark kubernetes s3 connectivity issue

Should we remove the existing jar and upgrade it to some recent version? On Tue, Feb 15, 2022, 01:08 Mich Talebzadeh wrote: > I recall I had similar issues running Spark on Google Dataproc. > > sounds like it gets Hadoop's jars on the classpath which include an older > version of Guava. The

Re: Spark kubernetes s3 connectivity issue

I recall I had similar issues running Spark on Google Dataproc. sounds like it gets Hadoop's jars on the classpath which include an older version of Guava. The solution is to shade/relocate Guava in your distribution HTH view my Linkedin profile

Spark kubernetes s3 connectivity issue

Hi Team , We are trying to build a docker image using Centos and trying to connect through S3. Same works with Hadoop 3.2.0 and spark.3.1.2 #Installing spark binaries ENV SPARK_HOME /opt/spark ENV SPARK_VERSION 3.2.1 ENV HADOOP_VERSION 3.2.0 ARG HADOOP_VERSION_SHORT=3.2 ARG