Github user skonto commented on the issue:
https://github.com/apache/spark/pull/20945
@vanzin @susanxhuynh
> - require the launcher to have a kerberos login, and send DTs to the
application. a.k.a. what Spark-on-YARN does.
> - in the code that launches the driver on the Mesos side, create the DTs
in a safe context (e.g. not as part of the spark-submit invocation) and provide
them to the Spark driver using the HADOOP_TOKEN_FILE_LOCATION env var.
Option 1: Spark submit (launcher) could create the DTs locally as a utility
function.
Then the user could upload them to secret store.
And then run the spark submit as usual referencing the DTs secret.
Due to https://issues.apache.org/jira/browse/SPARK-20982 we can not pass
any info from the launcher side to the mesos dispatcher directly. Also we
cannot write code in Spark Submit to integrate with secret store because this
depends on the env. DC/OS has its own implementation and libs/API to integrate
with it.
Option 2: means that the dispatcher should create the DTs and pass them to
the driverâs container as secrets. That means it should be able to fetch the
superusers TGT from the secret store create the delegation tokens in an
isolated manner (other drivers could be launched in parallel) and store them
back to the secret store so that the driver to be launched can use them. Again
this would require for the mesos dispatcher to be integrated with DCOS APIs,
for example to access the secret store you need an auth token to be passed and
call a specific api
https://docs.mesosphere.com/1.11.0/administration/secrets/secrets-api/.
Option 3: Spark submit within the container before it runs the userâs
main it could create the DTs, save them to the local filesystem, point to them
with HADOOP_TOKEN_FILE_LOCATION and then remove the TGT (/tmp/krb5cc_uid) (like
kdestroy), so user code cannot use it to impersonate anyone.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
