This is an automated email from the ASF dual-hosted git repository.
yao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new f6059a9 [KYUUBI #1090] Add deployment document about Hadoop
Credentials Manager
f6059a9 is described below
commit f6059a9b300453189a71fd2080dcbab17ad223a2
Author: zhouyifan279 <[email protected]>
AuthorDate: Wed Sep 15 10:02:09 2021 +0800
[KYUUBI #1090] Add deployment document about Hadoop Credentials Manager
### _Why are the changes needed?_
Umbrella issue #915 is finished. Document is needed to tell user how to
enable it.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including
negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run
test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests)
locally before make a pull request
Closes #1101 from zhouyifan279/KYUUBI#1090.
Closes #1090
fdd4da8a [zhouyifan279] [KYUUBI #1090] Add deployment document about Hadoop
Credentials Manager
418da1dc [zhouyifan279] [KYUUBI #1090] Add deployment document about Hadoop
Credentials Manager
0e8e44ba [zhouyifan279] [KYUUBI #1090] Add deployment document about Hadoop
Credentials Manager
Authored-by: zhouyifan279 <[email protected]>
Signed-off-by: Kent Yao <[email protected]>
---
docs/deployment/settings.md | 2 +-
docs/security/hadoop_credentials_manager.md | 77 ++++++++++++++++++++++
docs/security/index.rst | 1 +
.../org/apache/kyuubi/config/KyuubiConf.scala | 2 +-
4 files changed, 80 insertions(+), 2 deletions(-)
diff --git a/docs/deployment/settings.md b/docs/deployment/settings.md
index a77eaed..8840875 100644
--- a/docs/deployment/settings.md
+++ b/docs/deployment/settings.md
@@ -155,7 +155,7 @@ kyuubi\.credentials<br>\.hadoopfs\.enabled|<div
style='width: 65pt;word-wrap: br
kyuubi\.credentials<br>\.hadoopfs\.uris|<div style='width: 65pt;word-wrap:
break-word;white-space: normal'></div>|<div style='width: 170pt;word-wrap:
break-word;white-space: normal'>Extra Hadoop filesystem URIs for which to
request delegation tokens. The filesystem that hosts fs.defaultFS does not need
to be listed here.</div>|<div style='width: 30pt'>seq</div>|<div style='width:
20pt'>1.4.0</div>
kyuubi\.credentials<br>\.hive\.enabled|<div style='width: 65pt;word-wrap:
break-word;white-space: normal'>true</div>|<div style='width: 170pt;word-wrap:
break-word;white-space: normal'>Whether to renew Hive metastore delegation
token</div>|<div style='width: 30pt'>boolean</div>|<div style='width:
20pt'>1.4.0</div>
kyuubi\.credentials<br>\.renewal\.interval|<div style='width: 65pt;word-wrap:
break-word;white-space: normal'>PT1H</div>|<div style='width: 170pt;word-wrap:
break-word;white-space: normal'>How often Kyuubi renews one user's delegation
tokens</div>|<div style='width: 30pt'>duration</div>|<div style='width:
20pt'>1.4.0</div>
-kyuubi\.credentials<br>\.renewal\.retryWait|<div style='width: 65pt;word-wrap:
break-word;white-space: normal'>PT1M</div>|<div style='width: 170pt;word-wrap:
break-word;white-space: normal'>How long to wait before retrying to fetch new
credentials after a failure.</div>|<div style='width: 30pt'>duration</div>|<div
style='width: 20pt'>1.4.0</div>
+kyuubi\.credentials<br>\.renewal\.retry\.wait|<div style='width:
65pt;word-wrap: break-word;white-space: normal'>PT1M</div>|<div style='width:
170pt;word-wrap: break-word;white-space: normal'>How long to wait before
retrying to fetch new credentials after a failure.</div>|<div style='width:
30pt'>duration</div>|<div style='width: 20pt'>1.4.0</div>
### Delegation
diff --git a/docs/security/hadoop_credentials_manager.md
b/docs/security/hadoop_credentials_manager.md
new file mode 100644
index 0000000..32ce02e
--- /dev/null
+++ b/docs/security/hadoop_credentials_manager.md
@@ -0,0 +1,77 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+<div align=center>
+
+
+
+</div>
+
+# Hadoop Credentials Manager
+
+In order to pass the authentication of a kerberos secured hadoop cluster,
kyuubi currently submits
+engines in two ways:
+1. Submits with current kerberos user and extra `SparkSubmit` argument
`--proxy-user`.
+2. Submits with `spark.kerberos.principal` and `spark.kerberos.keytab`
specified.
+
+If engine is submitted with `--proxy-user` specified, its delegation tokens of
hadoop cluster
+services are obtained by current kerberos user and can not be renewed by
itself.
+Thus, engine's lifetime is limited by the lifetime of delegation tokens.
+To remove this limitation, kyuubi renews delegation tokens at server side in
Hadoop Credentials Manager.
+
+Engine submitted with principal and keytab can renew delegation tokens by
itself.
+But for implementation simplicity, kyuubi server will also renew delegation
tokens for it.
+
+## Configurations
+
+### Cluster Services
+Kyuubi currently supports renew delegation tokens of Hadoop filesystems and
Hive metastore servers.
+
+#### Hadoop client configurations
+Set `HADOOP_CONF_DIR` in `$KYUUBI_HOME/conf/kyuubi-env.sh` if it hasn't been
set yet, e.g.
+
+```bash
+$ echo "export HADOOP_CONF_DIR=/path/to/hadoop/conf" >>
$KYUUBI_HOME/conf/kyuubi-env.sh
+```
+Extra Hadoop filesystems can be specified in
`$KYUUBI_HOME/conf/kyuubi-defaults.conf`
+by `kyuubi.credentials.hadoopfs.uris` in comma separated list.
+
+#### Hive metastore configurations
+
+##### Via kyuubi-defaults.conf
+Specify Hive metastore configurations In
`$KYUUBI_HOME/conf/kyuubi-defaults.conf`. Hadoop Credentials
+Manager will load the configurations when initialized.
+
+##### Via hive-site.xml
+Place your copy of `hive-site.xml` into `$KYUUBI_HOME/conf`, Kyuubi will load
this config file to
+its classpath.
+
+This version of configuration has lower priority than those in
`$KYUUBI_HOME/conf/kyuubi-defaults.conf`.
+
+##### Via JDBC Connection URL
+Hive configurations specified in JDBC connection URL are ignored by Hadoop
Credentials Manager as
+Hadoop Credentials Manager is initialized when Kyuubi server starts.
+
+### Credentials Renewal
+
+Key | Default | Meaning | Type | Since
+--- | --- | --- | --- | ---
+kyuubi\.credentials<br>\.hadoopfs\.enabled|<div style='width: 65pt;word-wrap:
break-word;white-space: normal'>true</div>|<div style='width: 170pt;word-wrap:
break-word;white-space: normal'>Whether to renew Hadoop filesystem delegation
tokens</div>|<div style='width: 30pt'>boolean</div>|<div style='width:
20pt'>1.4.0</div>
+kyuubi\.credentials<br>\.hadoopfs\.uris|<div style='width: 65pt;word-wrap:
break-word;white-space: normal'></div>|<div style='width: 170pt;word-wrap:
break-word;white-space: normal'>Extra Hadoop filesystem URIs for which to
request delegation tokens. The filesystem that hosts fs.defaultFS does not need
to be listed here.</div>|<div style='width: 30pt'>seq</div>|<div style='width:
20pt'>1.4.0</div>
+kyuubi\.credentials<br>\.hive\.enabled|<div style='width: 65pt;word-wrap:
break-word;white-space: normal'>true</div>|<div style='width: 170pt;word-wrap:
break-word;white-space: normal'>Whether to renew Hive metastore delegation
token</div>|<div style='width: 30pt'>boolean</div>|<div style='width:
20pt'>1.4.0</div>
+kyuubi\.credentials<br>\.renewal\.interval|<div style='width: 65pt;word-wrap:
break-word;white-space: normal'>PT1H</div>|<div style='width: 170pt;word-wrap:
break-word;white-space: normal'>How often Kyuubi renews one user's delegation
tokens</div>|<div style='width: 30pt'>duration</div>|<div style='width:
20pt'>1.4.0</div>
+kyuubi\.credentials<br>\.renewal\.retry\.wait|<div style='width:
65pt;word-wrap: break-word;white-space: normal'>PT1M</div>|<div style='width:
170pt;word-wrap: break-word;white-space: normal'>How long to wait before
retrying to fetch new credentials after a failure.</div>|<div style='width:
30pt'>duration</div>|<div style='width: 20pt'>1.4.0</div>
diff --git a/docs/security/index.rst b/docs/security/index.rst
index e8eacf9..63e6d89 100644
--- a/docs/security/index.rst
+++ b/docs/security/index.rst
@@ -25,5 +25,6 @@ Kyuubi Security Overview
Authentication <authentication>
kinit
+ hadoop_credentials_manager
authorization
diff --git
a/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
b/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
index c2480ef..21f9c13 100644
--- a/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
+++ b/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
@@ -242,7 +242,7 @@ object KyuubiConf {
.createWithDefault(Duration.ofHours(1).toMillis)
val CREDENTIALS_RENEWAL_RETRY_WAIT: ConfigEntry[Long] =
- buildConf("credentials.renewal.retryWait")
+ buildConf("credentials.renewal.retry.wait")
.doc("How long to wait before retrying to fetch new credentials after a
failure.")
.version("1.4.0")
.timeConf
