[
https://issues.apache.org/jira/browse/SPARK-21842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16168278#comment-16168278
]
Arthur Rand edited comment on SPARK-21842 at 9/18/17 12:42 AM:
---------------------------------------------------------------
Hey [~kalvinnchau]
I'm currently of the mind that using the RPC/broadcast approach is better (for
Mesos) for a couple reasons.
1. We recently added Secret support to Spark on Mesos, this uses a temporary
file system to put the keytab or TGT in the sandbox of the Spark driver. They
are packed into the SparkAppConfig (CoarseGrainedSchedulerBackend.scala:L236)
which is broadcast to the executors, so using RPC/broadcast is consistent with
this.
2. Keeps all transfers of secure information within Spark.
3. Doesn't _require_ HDFS.
However I understand that there is a potential risk with executors falsely
registering with the Driver and getting tokens. I know in the case of DC/OS
this is less of a concern (we have some protections around this). But this
could still happen today due to the code mentioned above. We could prevent this
by keeping track of the executor IDs and only allowing executors to register
when they have an expected ID..?
was (Author: arand):
Hey [~kalvinnchau]
I'm currently of the mind that using the RPC/broadcast approach is better (for
Mesos) for a couple reasons.
1. We recently added Secret support to Spark on Mesos, this uses a temporary
file system to put the keytab or TGT in the sandbox of the Spark driver. They
are packed into the SparkAppConfig (CoarseGrainedSchedulerBackend.scala:L236)
which is broadcast to the executors, so using RPC/broadcast is consistent with
this.
2. Keeps all transfers of secure information within Spark.
3. Doesn't require HDFS. There is a little bit of a chicken-and-egg situation
here w.r.t. YARN, but I'm obviously not familiar enough with how
Spark-YARN-HDFS work together.
However I understand that there is a potential risk with executors falsely
registering with the Driver and getting tokens. I know in the case of DC/OS
this is less of a concern (we have some protections around this). But this
could still happen today due to the code mentioned above. We could prevent this
by keeping track of the executor IDs and only allowing executors to register
when they have an expected ID..?
> Support Kerberos ticket renewal and creation in Mesos
> ------------------------------------------------------
>
> Key: SPARK-21842
> URL: https://issues.apache.org/jira/browse/SPARK-21842
> Project: Spark
> Issue Type: New Feature
> Components: Mesos
> Affects Versions: 2.3.0
> Reporter: Arthur Rand
>
> We at Mesosphere have written Kerberos support for Spark on Mesos. The code
> to use Kerberos on a Mesos cluster has been added to Apache Spark
> (SPARK-16742). This ticket is to complete the implementation and allow for
> ticket renewal and creation. Specifically for long running and streaming jobs.
> Mesosphere design doc (needs revision, wip):
> https://docs.google.com/document/d/1xyzICg7SIaugCEcB4w1vBWp24UDkyJ1Pyt2jtnREFqc/edit#heading=h.tdnq7wilqrj6
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
