There's generally an exception in these cases, and you haven't posted it, so it's hard to tell you what's wrong. The most probable cause, without the extra information the exception provides, is that you're using the wrong Hadoop configuration when submitting the job to YARN.
On Mon, Dec 5, 2016 at 4:35 AM, Gerard Casey <gerardhughca...@gmail.com> wrote: > Hello all, > > I am using Spark with Kerberos authentication. > > I can run my code using `spark-shell` fine and I can also use `spark-submit` > in local mode (e.g. —master local[16]). Both function as expected. > > local mode - > > spark-submit --class "graphx_sp" --master local[16] --driver-memory 20G > target/scala-2.10/graphx_sp_2.10-1.0.jar > > I am now progressing to run in cluster mode using YARN. > > cluster mode with YARN - > > spark-submit --class "graphx_sp" --master yarn --deploy-mode cluster > --executor-memory 13G --total-executor-cores 32 > target/scala-2.10/graphx_sp_2.10-1.0.jar > > However, this returns: > > diagnostics: User class threw exception: > org.apache.hadoop.security.AccessControlException: Authentication required > > Before I run using spark-shell or on local mode in spark-submit I do the > following kerberos setup: > > kinit -k -t ~/keytab -r 7d `whoami` > > Clearly, this setup is not extending to the YARN setup. How do I fix the > Kerberos issue with YARN in cluster mode? Is this something which must be in > my /src/main/scala/graphx_sp.scala file? > > Many thanks > > Geroid -- Marcelo --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
