Re: Kerberos setup in Apache spark connecting to remote HDFS/Yarn

[Sudarshan Rangarajan]

Hi Ami,

Did you try setting spark.yarn.principal and spark.yarn.keytab as
configuration properties, passing in their corresponding Kerberos values ?

Search for these properties on
http://spark.apache.org/docs/latest/running-on-yarn.html to learn more
about what's expected for them.

Regards,
Sudarshan

On Fri, Jun 17, 2016 at 12:01 PM, akhandeshi 
wrote:

> Little more progress...
>
> I add few enviornment variables, not I get following error message:
>
>  InvocationTargetException: Can't get Master Kerberos principal for use as
> renewer -> [Help 1]
>
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Kerberos-setup-in-Apache-spark-connecting-to-remote-HDFS-Yarn-tp27181p27189.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>

User impersonation with Kerberos and Delegation tokens

[Sudarshan Rangarajan]

Hi there,

I'm using SparkLauncher API from Spark v1.6.1, to submit a Spark job to
YARN. The service from where this API will be invoked will need to talk to
other services on our cluster via Kerberos (ex. HDFS, YARN etc.). Also, my
service expects to impersonate its then logged-in user during job
submission to YARN.

Can someone comment on/confirm, the following two observations ?

1. From this Github PR
,
it seems that SparkSubmit doesn't support user impersonation when Kerberos
credentials of the real user are specified.

2. SPARK-14743  - It
seems that Spark doesn't yet support acquisition of delegation tokens
during job submission to YARN irrespective of user impersonation. This Github
discussion  is also relevant to
this scenario.

Thanks,
Sudarshan