This is an automated email from the ASF dual-hosted git repository.
chengpan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new c454bfdcb [KYUUBI #3385] Set spark.kubernetes.executor.podNamePrefix
if missing
c454bfdcb is described below
commit c454bfdcbeb336d47d670ccf5ff84e45bf79a20b
Author: Hanna Liashchuk <[email protected]>
AuthorDate: Thu Oct 20 17:30:39 2022 +0800
[KYUUBI #3385] Set spark.kubernetes.executor.podNamePrefix if missing
### _Why are the changes needed?_
In Spark, if Kubernetes executor pod is not conforming to Kubernetes naming
conventions, a user gets the next error:
```
org.apache.kyuubi.KyuubiSQLException: org.apache.kyuubi.KyuubiSQLException:
java.lang.IllegalArgumentException:
'kyuubi-connection-spark-sql-USER-8fc2847d-b65d-4a46-bb41-4a86385cb61c-988995839f29e15f'
in spark.kubernetes.executor.podNamePrefix is invalid.
```
It can be fixed by manually setting
'spark.kubernetes.executor.podNamePrefix' in configurations, but since you
cannot dynamically set a value for every user to contain their username, it
will miss this essential information that helps to distinguish executors one
from another. I suggest Kyuubi server adds this config automatically if it's
not coming from user.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including
negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [X] [Run
test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests)
locally before make a pull request
Closes #3590 from hanna-liashchuk/master.
Closes #3385
94613145 [Hanna Liashchuk] Fixed style check
4b2281ce [Hanna Liashchuk] Merged if clauses
2c52988d [hanna-liashchuk] Merge branch 'apache:master' into master
f397cdc7 [Hanna Liashchuk] Added if clause
8b91109c [Hanna Liashchuk] Added EpochMilli to the podNamePrefix
13e8fb53 [hanna-liashchuk] Merge branch 'apache:master' into master
dcef3e10 [Hanna Liashchuk] Set executor.podNamePrefix for k8s if missing
Lead-authored-by: Hanna Liashchuk <[email protected]>
Co-authored-by: hanna-liashchuk
<[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
---
.../src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala | 2 ++
1 file changed, 2 insertions(+)
diff --git
a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
index ff96ef4f9..108dd2dc6 100644
---
a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
+++
b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
@@ -168,6 +168,8 @@ object SparkSQLEngine extends Logging {
if (Utils.isOnK8s) {
kyuubiConf.setIfMissing(FRONTEND_CONNECTION_URL_USE_HOSTNAME, false)
+ val podNamePrefix = s"kyuubi-${user}-${Instant.now().toEpochMilli}"
+ _sparkConf.setIfMissing("spark.kubernetes.executor.podNamePrefix",
podNamePrefix)
}
// Set web ui port 0 to avoid port conflicts during non-k8s cluster mode
