[
https://issues.apache.org/jira/browse/SPARK-13110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Beauvois updated SPARK-13110:
------------------------------------
Description:
Hello,
Spark is installed on several machines of my cluster. These machines are used
by the clients (we'll call these machines "CM"). Note that Spark is configured
on Yarn and not on standalone mode.
I installed a Spark History server on a different machine which is unaccessible
to the clients (we'll call this machine "SHS"). Now I'm trying to configure
Spark History web UI access for the users of my cluster who are authenticated
with Kerberos. For the moment I have been able to make Spark History working
with its kerberos principal and keytab.
The spark-defaults.conf of the SHS is the following:
{code}
# Spark history server configurations
spark.history.provider = org.apache.spark.deploy.history.FsHistoryProvider
spark.history.fs.logDirectory = hdfs:///Products/SPARK/logs/
spark.history.fs.update.interval = 10s
spark.history.retainedApplications = 100
spark.history.ui.port = 18080
spark.history.kerberos.enabled = true
spark.history.kerberos.principal =
sparkhistory/[email protected]
spark.history.kerberos.keytab =
/opt/application/Spark/current/keytabs/sparkhistory.keytab
spark.history.ui.acls.enable = true
spark.history.fs.cleaner.enabled = false
spark.yarn.historyServer.address = sparkhistoryserver.dns.fr:18080
{code}
The spark-defaults.conf of the CM is the following:
{code}
# Spark history server configurations
spark.history.provider =
spark.history.fs.logDirectory =
spark.history.fs.update.interval =
spark.history.retainedApplications =
spark.history.ui.port =
spark.history.kerberos.enabled =
spark.history.kerberos.principal =
spark.history.kerberos.keytab =
spark.history.ui.acls.enable = true
spark.history.fs.cleaner.enabled =
spark.yarn.historyServer.address = sparkhistoryserver.dns.fr:18080
{code}
First I would like to know if my configurations are good (both SHS and CM).
Secondly I would like to know how to restrict the web UI access for the users
who are kerberos authenticated. Let me explain more what is the expected
behaviour here:
- the user Obelix does a Spark job and finishes it properly
- Obelix can go to the ResourceManager web UI and click on "history". He's
redirected to the Spark History web UI and he can have the details of its
previous job. Note that Obelix is kerberos authenticated in order to be able to
go to the ResourceManager web UI and the Spark History web UI.
- Asterix goes to the ResourceManager web UI and click on "history" of Obelix's
job. Asterix is not redirected because he's not the user who launched the job.
was:
Hello,
Spark is installed on several machines of my cluster. These machines are used
by the clients (we'll call these machines "CM"). Note that Spark is configured
on Yarn and not on standalone mode.
I installed a Spark History server on a different machine which is unaccessible
to the clients (we'll call this machine "SHS"). Now I'm trying to configure
Spark History web UI access for the users of my cluster who are authenticated
with Kerberos. For the moment I have been able to make Spark History working
with its kerberos principal and keytab.
The spark-defaults.conf of the SHS is the following:
{code}
# Spark history server configurations
spark.history.provider = org.apache.spark.deploy.history.FsHistoryProvider
spark.history.fs.logDirectory = hdfs:///Products/SPARK/logs/
spark.history.fs.update.interval = 10s
spark.history.retainedApplications = 100
spark.history.ui.port = 18080
spark.history.kerberos.enabled = true
spark.history.kerberos.principal =
sparkhistory/[email protected]
spark.history.kerberos.keytab =
/opt/application/Spark/current/keytabs/sparkhistory.keytab
spark.history.ui.acls.enable = true
spark.history.fs.cleaner.enabled = false
spark.yarn.historyServer.address = sparkhistoryserver.dns.fr:18080
{code}
The spark-defaults.conf of the CM is the following:
{code}
# Spark history server configurations
spark.history.provider =
spark.history.fs.logDirectory =
spark.history.fs.update.interval =
spark.history.retainedApplications =
spark.history.ui.port =
spark.history.kerberos.enabled =
spark.history.kerberos.principal =
spark.history.kerberos.keytab =
spark.history.ui.acls.enable = true
spark.history.fs.cleaner.enabled =
spark.yarn.historyServer.address = sparkhistoryserver.dns.fr:18080
{code}
First I would like to know if my configurations are good.
Secondly I would like to know how to restrict the web UI access for the users
who are kerberos authenticated. Let me explain more what is the expected
behaviour here:
- the user Obelix does a Spark job and finishes it properly
- Obelix can go to the ResourceManager web UI and click on "history". He's
redirected to the Spark History web UI and he can have the details of its
previous job. Note that Obelix is kerberos authenticated in order to be able to
go to the ResourceManager web UI and the Spark History web UI.
- Asterix goes to the ResourceManager web UI and click on "history" of Obelix's
job. Asterix is not redirected because he's not the user who launched the job.
> How to configure the access of the Spark History Web UI with Kerberos
> authentication?
> -------------------------------------------------------------------------------------
>
> Key: SPARK-13110
> URL: https://issues.apache.org/jira/browse/SPARK-13110
> Project: Spark
> Issue Type: Question
> Components: Web UI
> Affects Versions: 1.5.1, 1.5.2, 1.6.0
> Environment: Spark 1.6.0 / Hadoop 2.7.1 / Zookeeper 3.4.5 /
> Authentication done through Kerberos
> Reporter: Pierre Beauvois
>
> Hello,
> Spark is installed on several machines of my cluster. These machines are used
> by the clients (we'll call these machines "CM"). Note that Spark is
> configured on Yarn and not on standalone mode.
> I installed a Spark History server on a different machine which is
> unaccessible to the clients (we'll call this machine "SHS"). Now I'm trying
> to configure Spark History web UI access for the users of my cluster who are
> authenticated with Kerberos. For the moment I have been able to make Spark
> History working with its kerberos principal and keytab.
> The spark-defaults.conf of the SHS is the following:
> {code}
> # Spark history server configurations
> spark.history.provider = org.apache.spark.deploy.history.FsHistoryProvider
> spark.history.fs.logDirectory = hdfs:///Products/SPARK/logs/
> spark.history.fs.update.interval = 10s
> spark.history.retainedApplications = 100
> spark.history.ui.port = 18080
> spark.history.kerberos.enabled = true
> spark.history.kerberos.principal =
> sparkhistory/[email protected]
> spark.history.kerberos.keytab =
> /opt/application/Spark/current/keytabs/sparkhistory.keytab
> spark.history.ui.acls.enable = true
> spark.history.fs.cleaner.enabled = false
> spark.yarn.historyServer.address = sparkhistoryserver.dns.fr:18080
> {code}
> The spark-defaults.conf of the CM is the following:
> {code}
> # Spark history server configurations
> spark.history.provider =
> spark.history.fs.logDirectory =
> spark.history.fs.update.interval =
> spark.history.retainedApplications =
> spark.history.ui.port =
> spark.history.kerberos.enabled =
> spark.history.kerberos.principal =
> spark.history.kerberos.keytab =
> spark.history.ui.acls.enable = true
> spark.history.fs.cleaner.enabled =
> spark.yarn.historyServer.address = sparkhistoryserver.dns.fr:18080
> {code}
> First I would like to know if my configurations are good (both SHS and CM).
> Secondly I would like to know how to restrict the web UI access for the users
> who are kerberos authenticated. Let me explain more what is the expected
> behaviour here:
> - the user Obelix does a Spark job and finishes it properly
> - Obelix can go to the ResourceManager web UI and click on "history". He's
> redirected to the Spark History web UI and he can have the details of its
> previous job. Note that Obelix is kerberos authenticated in order to be able
> to go to the ResourceManager web UI and the Spark History web UI.
> - Asterix goes to the ResourceManager web UI and click on "history" of
> Obelix's job. Asterix is not redirected because he's not the user who
> launched the job.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
