This is an automated email from the ASF dual-hosted git repository.
chengpan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new 0ce697eef [KYUUBI #5654] Introduce new config to forcibly rewrite pod
name
0ce697eef is described below
commit 0ce697eefa72091181e81f2e656d84ff54083c39
Author: hezhao2 <[email protected]>
AuthorDate: Mon Nov 13 10:21:05 2023 +0800
[KYUUBI #5654] Introduce new config to forcibly rewrite pod name
### _Why are the changes needed?_
close #5654
This pr is to introduce a config for rewriting pod name, which can help
users forcibly rewrite driver or executor pod name prefix.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including
negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run
test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests)
locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes #5672 from zhaohehuhu/dev-1111.
Closes #5654
6a450e444 [hezhao2] introduce new config to force rewrite pod name
Authored-by: hezhao2 <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
---
docs/configuration/settings.md | 30 ++++++++++++----------
.../org/apache/kyuubi/config/KyuubiConf.scala | 20 +++++++++++++++
.../kyuubi/engine/spark/SparkProcessBuilder.scala | 16 ++++++++++--
.../org/apache/kyuubi/util/KubernetesUtils.scala | 24 ++++++++++-------
.../engine/spark/SparkProcessBuilderSuite.scala | 21 ++++++++++++++-
5 files changed, 85 insertions(+), 26 deletions(-)
diff --git a/docs/configuration/settings.md b/docs/configuration/settings.md
index 2673a6903..0fa6d8e5a 100644
--- a/docs/configuration/settings.md
+++ b/docs/configuration/settings.md
@@ -309,20 +309,22 @@ You can configure the Kyuubi properties in
`$KYUUBI_HOME/conf/kyuubi-defaults.co
### Kubernetes
-| Key | Default |
Meaning
| Type | Since |
-|-----------------------------------------------------|-------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
-| kyuubi.kubernetes.authenticate.caCertFile | <undefined> |
Path to the CA cert file for connecting to the Kubernetes API server over TLS
from the kyuubi. Specify this as a path as opposed to a URI (i.e. do not
provide a scheme) | string | 1.7.0 |
-| kyuubi.kubernetes.authenticate.clientCertFile | <undefined> |
Path to the client cert file for connecting to the Kubernetes API server over
TLS from the kyuubi. Specify this as a path as opposed to a URI (i.e. do not
provide a scheme) | string | 1.7.0 |
-| kyuubi.kubernetes.authenticate.clientKeyFile | <undefined> |
Path to the client key file for connecting to the Kubernetes API server over
TLS from the kyuubi. Specify this as a path as opposed to a URI (i.e. do not
provide a scheme) | string | 1.7.0 |
-| kyuubi.kubernetes.authenticate.oauthToken | <undefined> |
The OAuth token to use when authenticating against the Kubernetes API server.
Note that unlike, the other authentication options, this must be the exact
string value of the token to use for the authentication. | string | 1.7.0 |
-| kyuubi.kubernetes.authenticate.oauthTokenFile | <undefined> |
Path to the file containing the OAuth token to use when authenticating against
the Kubernetes API server. Specify this as a path as opposed to a URI (i.e. do
not provide a scheme) | string | 1.7.0 |
-| kyuubi.kubernetes.context | <undefined> |
The desired context from your kubernetes config file used to configure the K8s
client for interacting with the cluster.
| string | 1.6.0 |
-| kyuubi.kubernetes.context.allow.list ||
The allowed kubernetes context list, if it is empty, there is no kubernetes
context limitation.
| set | 1.8.0 |
-| kyuubi.kubernetes.master.address | <undefined> |
The internal Kubernetes master (API server) address to be used for kyuubi.
| string | 1.7.0 |
-| kyuubi.kubernetes.namespace | default |
The namespace that will be used for running the kyuubi pods and find engines.
| string | 1.7.0 |
-| kyuubi.kubernetes.namespace.allow.list ||
The allowed kubernetes namespace list, if it is empty, there is no kubernetes
namespace limitation.
| set | 1.8.0 |
-| kyuubi.kubernetes.terminatedApplicationRetainPeriod | PT5M |
The period for which the Kyuubi server retains application information after
the application terminates.
| duration | 1.7.1 |
-| kyuubi.kubernetes.trust.certificates | false | If
set to true then client can submit to kubernetes cluster only with token
| boolean | 1.7.0 |
+| Key |
Default |
Meaning
| Type | Since |
+|----------------------------------------------------------------------|-------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
+| kyuubi.kubernetes.authenticate.caCertFile |
<undefined> | Path to the CA cert file for connecting to the Kubernetes
API server over TLS from the kyuubi. Specify this as a path as opposed to a URI
(i.e. do not provide a scheme)
| string |
1.7.0 |
+| kyuubi.kubernetes.authenticate.clientCertFile |
<undefined> | Path to the client cert file for connecting to the
Kubernetes API server over TLS from the kyuubi. Specify this as a path as
opposed to a URI (i.e. do not provide a scheme)
|
string | 1.7.0 |
+| kyuubi.kubernetes.authenticate.clientKeyFile |
<undefined> | Path to the client key file for connecting to the
Kubernetes API server over TLS from the kyuubi. Specify this as a path as
opposed to a URI (i.e. do not provide a scheme)
|
string | 1.7.0 |
+| kyuubi.kubernetes.authenticate.oauthToken |
<undefined> | The OAuth token to use when authenticating against the
Kubernetes API server. Note that unlike, the other authentication options, this
must be the exact string value of the token to use for the authentication.
| string |
1.7.0 |
+| kyuubi.kubernetes.authenticate.oauthTokenFile |
<undefined> | Path to the file containing the OAuth token to use when
authenticating against the Kubernetes API server. Specify this as a path as
opposed to a URI (i.e. do not provide a scheme)
| string
| 1.7.0 |
+| kyuubi.kubernetes.context |
<undefined> | The desired context from your kubernetes config file used
to configure the K8s client for interacting with the cluster.
| string |
1.6.0 |
+| kyuubi.kubernetes.context.allow.list
|| The allowed kubernetes context list, if it is empty, there is no
kubernetes context limitation.
| set | 1.8.0 |
+| kyuubi.kubernetes.master.address |
<undefined> | The internal Kubernetes master (API server) address to be
used for kyuubi.
| string |
1.7.0 |
+| kyuubi.kubernetes.namespace |
default | The namespace that will be used for running the kyuubi pods
and find engines.
| string | 1.7.0 |
+| kyuubi.kubernetes.namespace.allow.list
|| The allowed kubernetes namespace list, if it is empty, there is
no kubernetes namespace limitation.
| set | 1.8.0 |
+| kyuubi.kubernetes.spark.forciblyRewriteDriverPodName.enabled | false
| Whether to forcibly rewrite Spark driver pod name with
'kyuubi-<uuid>-driver'. If disabled, Kyuubi will try to preserve the
application name while satisfying K8s' pod name policy, but some vendors may
have stricter pod name policies, thus the generated name may become illegal.
| boolean | 1.8.1 |
+| kyuubi.kubernetes.spark.forciblyRewriteExecutorPodNamePrefix.enabled | false
| Whether to forcibly rewrite Spark executor pod name prefix with
'kyuubi-<uuid>'. If disabled, Kyuubi will try to preserve the application name
while satisfying K8s' pod name policy, but some vendors may have stricter Pod
name policies, thus the generated name may become illegal. | boolean | 1.8.1 |
+| kyuubi.kubernetes.terminatedApplicationRetainPeriod | PT5M
| The period for which the Kyuubi server retains application
information after the application terminates.
| duration | 1.7.1
|
+| kyuubi.kubernetes.trust.certificates | false
| If set to true then client can submit to kubernetes cluster only
with token
| boolean | 1.7.0 |
### Lineage
diff --git
a/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
b/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
index c06672ad2..042a211a2 100644
--- a/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
+++ b/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
@@ -3153,4 +3153,24 @@ object KyuubiConf {
.serverOnly
.intConf
.createOptional
+
+ val KUBERNETES_FORCIBLY_REWRITE_DRIVER_POD_NAME: ConfigEntry[Boolean] =
+ buildConf("kyuubi.kubernetes.spark.forciblyRewriteDriverPodName.enabled")
+ .doc("Whether to forcibly rewrite Spark driver pod name with
'kyuubi-<uuid>-driver'. " +
+ "If disabled, Kyuubi will try to preserve the application name while
satisfying K8s' " +
+ "pod name policy, but some vendors may have stricter pod name
policies, thus the " +
+ "generated name may become illegal.")
+ .version("1.8.1")
+ .booleanConf
+ .createWithDefault(false)
+
+ val KUBERNETES_FORCIBLY_REWRITE_EXEC_POD_NAME_PREFIX: ConfigEntry[Boolean] =
+
buildConf("kyuubi.kubernetes.spark.forciblyRewriteExecutorPodNamePrefix.enabled")
+ .doc("Whether to forcibly rewrite Spark executor pod name prefix with
'kyuubi-<uuid>'. " +
+ "If disabled, Kyuubi will try to preserve the application name while
satisfying K8s' " +
+ "pod name policy, but some vendors may have stricter Pod name
policies, thus the " +
+ "generated name may become illegal.")
+ .version("1.8.1")
+ .booleanConf
+ .createWithDefault(false)
}
diff --git
a/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala
b/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala
index afc96fb5e..086ca057d 100644
---
a/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala
+++
b/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala
@@ -30,6 +30,7 @@ import org.apache.hadoop.security.UserGroupInformation
import org.apache.kyuubi._
import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.config.KyuubiConf._
import org.apache.kyuubi.engine.{ApplicationManagerInfo,
KyuubiApplicationManager, ProcBuilder}
import
org.apache.kyuubi.engine.KubernetesApplicationOperation.{KUBERNETES_SERVICE_HOST,
KUBERNETES_SERVICE_PORT}
import org.apache.kyuubi.engine.ProcBuilder.KYUUBI_ENGINE_LOG_PATH_KEY
@@ -229,17 +230,28 @@ class SparkProcessBuilder(
kubernetesNamespace())
}
+ private val forciblyRewriteDriverPodName: Boolean =
+ conf.get(KUBERNETES_FORCIBLY_REWRITE_DRIVER_POD_NAME)
+ private val forciblyRewriteExecPodNamePrefix: Boolean =
+ conf.get(KUBERNETES_FORCIBLY_REWRITE_EXEC_POD_NAME_PREFIX)
+
def appendPodNameConf(conf: Map[String, String]): Map[String, String] = {
val appName = conf.getOrElse(APP_KEY, "spark")
val map = mutable.Map.newBuilder[String, String]
if (clusterManager().exists(cm =>
cm.toLowerCase(Locale.ROOT).startsWith("k8s"))) {
if (!conf.contains(KUBERNETES_EXECUTOR_POD_NAME_PREFIX)) {
- val prefix = KubernetesUtils.generateExecutorPodNamePrefix(appName,
engineRefId)
+ val prefix = KubernetesUtils.generateExecutorPodNamePrefix(
+ appName,
+ engineRefId,
+ forciblyRewriteExecPodNamePrefix)
map += (KUBERNETES_EXECUTOR_POD_NAME_PREFIX -> prefix)
}
if (deployMode().exists(_.toLowerCase(Locale.ROOT) == "cluster")) {
if (!conf.contains(KUBERNETES_DRIVER_POD_NAME)) {
- val name = KubernetesUtils.generateDriverPodName(appName,
engineRefId)
+ val name = KubernetesUtils.generateDriverPodName(
+ appName,
+ engineRefId,
+ forciblyRewriteDriverPodName)
map += (KUBERNETES_DRIVER_POD_NAME -> name)
}
}
diff --git
a/kyuubi-server/src/main/scala/org/apache/kyuubi/util/KubernetesUtils.scala
b/kyuubi-server/src/main/scala/org/apache/kyuubi/util/KubernetesUtils.scala
index 9da3408a3..929897d48 100644
--- a/kyuubi-server/src/main/scala/org/apache/kyuubi/util/KubernetesUtils.scala
+++ b/kyuubi-server/src/main/scala/org/apache/kyuubi/util/KubernetesUtils.scala
@@ -130,21 +130,27 @@ object KubernetesUtils extends Logging {
.replaceAll("^[0-9]", "x")
}
- def generateDriverPodName(appName: String, engineRefId: String): String = {
- val resolvedResourceName = s"kyuubi-${getResourceNamePrefix(appName,
engineRefId)}-driver"
- if (resolvedResourceName.length <= DRIVER_POD_NAME_MAX_LENGTH) {
- resolvedResourceName
- } else {
+ def generateDriverPodName(
+ appName: String,
+ engineRefId: String,
+ forciblyRewrite: Boolean): String = {
+ lazy val resolvedResourceName = s"kyuubi-${getResourceNamePrefix(appName,
engineRefId)}-driver"
+ if (forciblyRewrite || resolvedResourceName.length >
DRIVER_POD_NAME_MAX_LENGTH) {
s"kyuubi-$engineRefId-driver"
+ } else {
+ resolvedResourceName
}
}
- def generateExecutorPodNamePrefix(appName: String, engineRefId: String):
String = {
+ def generateExecutorPodNamePrefix(
+ appName: String,
+ engineRefId: String,
+ forciblyRewrite: Boolean): String = {
val resolvedResourceName = s"kyuubi-${getResourceNamePrefix(appName,
engineRefId)}"
- if (resolvedResourceName.length <= EXECUTOR_POD_NAME_PREFIX_MAX_LENGTH) {
- resolvedResourceName
- } else {
+ if (forciblyRewrite || resolvedResourceName.length >
EXECUTOR_POD_NAME_PREFIX_MAX_LENGTH) {
s"kyuubi-$engineRefId"
+ } else {
+ resolvedResourceName
}
}
}
diff --git
a/kyuubi-server/src/test/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilderSuite.scala
b/kyuubi-server/src/test/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilderSuite.scala
index 408f42f64..7bbe4ad06 100644
---
a/kyuubi-server/src/test/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilderSuite.scala
+++
b/kyuubi-server/src/test/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilderSuite.scala
@@ -28,7 +28,7 @@ import org.scalatestplus.mockito.MockitoSugar
import org.apache.kyuubi._
import org.apache.kyuubi.config.KyuubiConf
-import org.apache.kyuubi.config.KyuubiConf.{ENGINE_LOG_TIMEOUT,
ENGINE_SPARK_MAIN_RESOURCE}
+import org.apache.kyuubi.config.KyuubiConf.{ENGINE_LOG_TIMEOUT,
ENGINE_SPARK_MAIN_RESOURCE, KUBERNETES_FORCIBLY_REWRITE_DRIVER_POD_NAME,
KUBERNETES_FORCIBLY_REWRITE_EXEC_POD_NAME_PREFIX}
import org.apache.kyuubi.engine.ProcBuilder.KYUUBI_ENGINE_LOG_PATH_KEY
import org.apache.kyuubi.engine.spark.SparkProcessBuilder._
import org.apache.kyuubi.ha.HighAvailabilityConf
@@ -336,6 +336,15 @@ class SparkProcessBuilderSuite extends
KerberizedTestHelper with MockitoSugar {
val conf4 = Map(APP_KEY -> chineseAppName)
val driverPodName4 =
processBuilder.appendPodNameConf(conf4).get(KUBERNETES_DRIVER_POD_NAME)
assert(driverPodName4 === Some(s"kyuubi-test-$engineRefId-driver"))
+ val newProcessBuilder = new SparkProcessBuilder(
+ "kyuubi",
+ conf.set(MASTER_KEY, "k8s://internal").set(DEPLOY_MODE_KEY,
"cluster").set(
+ KUBERNETES_FORCIBLY_REWRITE_DRIVER_POD_NAME,
+ true),
+ engineRefId)
+ val conf5 = Map(APP_KEY -> "test-forcibly-rewrite-app")
+ val driverPodName5 =
newProcessBuilder.appendPodNameConf(conf5).get(KUBERNETES_DRIVER_POD_NAME)
+ assert(driverPodName5 === Some(s"kyuubi-$engineRefId-driver"))
}
test("[KYUUBI #5165] Test SparkProcessBuilder#appendExecutorPodPrefix") {
@@ -363,6 +372,16 @@ class SparkProcessBuilderSuite extends
KerberizedTestHelper with MockitoSugar {
val execPodNamePrefix3 = processBuilder
.appendPodNameConf(conf3).get(KUBERNETES_EXECUTOR_POD_NAME_PREFIX)
assert(execPodNamePrefix3 === Some(s"kyuubi-$engineRefId"))
+ val newProcessBuilder = new SparkProcessBuilder(
+ "kyuubi",
+ conf.set(MASTER_KEY, "k8s://internal").set(DEPLOY_MODE_KEY,
"cluster").set(
+ KUBERNETES_FORCIBLY_REWRITE_EXEC_POD_NAME_PREFIX,
+ true),
+ engineRefId)
+ val conf5 = Map(APP_KEY -> "test-forcibly-rewrite-app")
+ val execPodNamePrefix4 = newProcessBuilder
+ .appendPodNameConf(conf5).get(KUBERNETES_EXECUTOR_POD_NAME_PREFIX)
+ assert(execPodNamePrefix4 === Some(s"kyuubi-$engineRefId"))
}
test("extract spark core scala version") {
