Edit. From here <http://www.cloudera.com/documentation/enterprise/latest/topics/sg_spark_auth.html> I read that you can pass a `key tab` option to spark-submit. I thus tried
spark-submit --class "graphx_sp" --master yarn --keytab /path/to/keytab --deploy-mode cluster --executor-memory 13G --total-executor-cores 32 target/scala-2.10/graphx_sp_2.10-1.0.jar However, the error persists Any ideas? Thanks Geroid > On 5 Dec 2016, at 13:35, Gerard Casey <gerardhughca...@gmail.com> wrote: > > Hello all, > > I am using Spark with Kerberos authentication. > > I can run my code using `spark-shell` fine and I can also use `spark-submit` > in local mode (e.g. —master local[16]). Both function as expected. > > local mode - > > spark-submit --class "graphx_sp" --master local[16] --driver-memory 20G > target/scala-2.10/graphx_sp_2.10-1.0.jar > > I am now progressing to run in cluster mode using YARN. > > cluster mode with YARN - > > spark-submit --class "graphx_sp" --master yarn --deploy-mode cluster > --executor-memory 13G --total-executor-cores 32 > target/scala-2.10/graphx_sp_2.10-1.0.jar > > However, this returns: > > diagnostics: User class threw exception: > org.apache.hadoop.security.AccessControlException: Authentication required > > Before I run using spark-shell or on local mode in spark-submit I do the > following kerberos setup: > > kinit -k -t ~/keytab -r 7d `whoami` > > Clearly, this setup is not extending to the YARN setup. How do I fix the > Kerberos issue with YARN in cluster mode? Is this something which must be in > my /src/main/scala/graphx_sp.scala file? > > Many thanks > > Geroid
