Okay! It is a good approach too, I'm happy that you got it working.
Cheers,
Pol Santamaria
On Tue, Apr 20, 2021 at 10:23 AM Nisd wrote:
> I ended up with this solution:
> https://stackoverflow.com/a/67173899/1020941
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
I ended up with this solution: https://stackoverflow.com/a/67173899/1020941
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Hi Pol
Thanks for the reply!
"spark:latest" is based on the Dockerfile included with Spark Release 3.1.1
- "Pre-build for Apache Hadoop 3.2 and later"
I have simplified my Dockerfile quite a bit, but its still not loading at
all.
FROM spark:latest
ARG jetty_version=9.4.40.v20210413
ARG wildfly_
Hi Nick,
You should look which spark version is "latest", understand which Hadoop
version was built "spark:latest" on top, and then check the compatibility
of Hadoop with the Azure libraries. In the past, I used the following
Dockerfile to experiment:
FROM gcr.io/spark-operator/spark:v3.0.0
USER
Hello
I am trying to load the Hadoop-Azure driver in Apache Spark, but so far I have
failed.
The plan is to include the required files in the docker image, as we plan on
using a Client-mode SparkSession.
My current Dockerfile looks like this:
FROM spark:latest
