GitHub user vanzin opened a pull request:
https://github.com/apache/spark/pull/22624
[SPARK-23781][CORE] Add base class for token renewal functionality.
The base class is based on YARN's token renewer, with some new
functionality to support the current usage in Mesos. It supports
both renewal (which you get when you provide a principal and keytab
to Spark) and just creating an arbitrary set of tokens, which is
used by Mesos in the non-renewal case.
The internal API is a little sub-optimal because YARN needs to defer
some of the initialization of the renewer; so the code needs to handle
the case where the renewer doesn't yet have a reference to the driver
endpoint, for example.
YARN was tested with custom app to stress the token renewal code in
a real cluster. Tested the driver behavior with a hacked up Mesos
cluster (access to Hive and HDFS worked both with TGT and keytab,
delegation tokens were obtained, but I couldn't get executors to run).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vanzin/spark SPARK-23781
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22624.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22624
----
commit 01e39466d622665b63576e90b8fd0da7cae7e43f
Author: Marcelo Vanzin <vanzin@...>
Date: 2018-10-02T20:24:21Z
[SPARK-23781][CORE] Add base class for token renewal functionality.
The base class is based on YARN's token renewer, with some new
functionality to support the current usage in Mesos. It supports
both renewal (which you get when you provide a principal and keytab
to Spark) and just creating an arbitrary set of tokens, which is
used by Mesos in the non-renewal case.
The internal API is a little sub-optimal because YARN needs to defer
some of the initialization of the renewer; so the code needs to handle
the case where the renewer doesn't yet have a reference to the driver
endpoint, for example.
YARN was tested with custom app to stress the token renewal code in
a real cluster. Tested the driver behavior with a hacked up Mesos
clusters (access to Hive and HDFS worked both with TGT and keytab,
delegation tokens were obtained, but I couldn't get executors to run).
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
