You need to set the spark configuration property: spark.yarn.access.namenodes
to your namenode.
e.g. spark.yarn.access.namenodes=hdfs://mynamenode:8020
Similarly, I'm curious if you're also running high availability HDFS with an
HA nameservice.
I currently have HA HDFS and kerberos and I've noti
Only YARN mode is supported with kerberos. You can't use a spark:// master
with kerberos.
Tobias Pfeiffer wrote
> When you give a "spark://*" master, Spark will run on a different machine,
> where you have not yet authenticated to HDFS, I think. I don't know how to
> solve this, though, maybe som
