(kyuubi) branch master updated: [KYUUBI #5680] Optimize Spark engine pod name generation

[chengpan]

This is an automated email from the ASF dual-hosted git repository.

chengpan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kyuubi.git


The following commit(s) were added to refs/heads/master by this push:
     new 25eb53dcd [KYUUBI #5680] Optimize Spark engine pod name generation
25eb53dcd is described below

commit 25eb53dcdf089ed5cad2cdebda66c2a8aa6317bb
Author: Ocean22 <1058853...@qq.com>
AuthorDate: Mon Dec 4 12:33:09 2023 +0800

    [KYUUBI #5680] Optimize Spark engine pod name generation
    
    ### _Why are the changes needed?_
    
    Close #5680
    
    **Background**:
    1) In default case, when kyuubi submit spark sql engine on kubernetes, will 
generate spark driver pod name with 
"kyuubi-${app_name}-${engine_ref_id}-driver".
    2) And app_name will be 
"kyuubi_${shareLevel}_${engineType}_${appUser}_${engineRefId}" if not set by 
user.
    3) In result, we may get spark driver pod name with 
"kyuubi-kyuubi-${shareLevel}-${engineType}-${appUser}-${engineRefId}-${engineRefId}-driver".
    4) We were hoping for a more concise and readable name, such as 
"kyuubi-${shareLevel}-${engineType}-${appUser}-${app_name}-${engineRefId}-driver".
    
    ### _How was this patch tested?_
    - [ ] Add some test cases that check the changes thoroughly including 
negative and positive cases if possible
    
    - [ ] Add screenshots for manual tests if appropriate
    **1) Before modification：**
    Left is unset `spark.app.name`, and right is set to “ocean”
    
![image](https://github.com/apache/kyuubi/assets/45907917/350aa09f-a9cb-4cc4-bdc6-71b085ba0824)
    
    **2) Modify the code**
    
![image](https://github.com/apache/kyuubi/assets/45907917/f9cbef91-9e1a-482a-82bb-a78caf669438)
    
    **3) Build module and get the jar**
    
![image](https://github.com/apache/kyuubi/assets/45907917/b2cf7595-dd54-46eb-bb79-354f3301499f)
    
    **4) Build a new images**
    
![image](https://github.com/apache/kyuubi/assets/45907917/c0418f2d-dcde-4845-a568-84e920892359)
    
    **5) After modification:**
    
![image](https://github.com/apache/kyuubi/assets/45907917/5db95d5b-2550-41b7-a11c-d8e4a6c0f09f)
    
    - [ ] [Run 
test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests)
 locally before make a pull request
    
    ### _Was this patch authored or co-authored using generative AI tooling?_
    
    No
    
    Closes #5695 from Ocean22/master.
    
    Closes #5680
    
    6dbac57b3 [Cheng Pan] Update 
kyuubi-server/src/main/scala/org/apache/kyuubi/util/KubernetesUtils.scala
    5ddc9f6ad [no 会 English] Update KubernetesUtils.scala
    b685b60aa [no 会 English] Update KubernetesUtils.scala
    6a64e17ea [no 会 English] Update KubernetesUtils.scala
    1335bbd21 [no 会 English] Update EngineRef.scala
    f2af95535 [no 会 English] Update KubernetesUtils.scala
    891f1720d [no 会 English] Update KubernetesUtils.scala
    c20b75549 [no 会 English] Update EngineRef.scala
    09f3ed190 [no 会 English] Update EngineRef.scala
    64bd2c156 [no 会 English] Update EngineRef.scala
    
    Lead-authored-by: Ocean22 <1058853...@qq.com>
    Co-authored-by: no 会 English <45907917+ocea...@users.noreply.github.com>
    Co-authored-by: Cheng Pan <pan3...@gmail.com>
    Signed-off-by: Cheng Pan <cheng...@apache.org>
---
 .../org/apache/kyuubi/util/KubernetesUtils.scala   | 26 ++++++++++++++++++----
 1 file changed, 22 insertions(+), 4 deletions(-)

diff --git 
a/kyuubi-server/src/main/scala/org/apache/kyuubi/util/KubernetesUtils.scala 
b/kyuubi-server/src/main/scala/org/apache/kyuubi/util/KubernetesUtils.scala
index 929897d48..02b52f926 100644
--- a/kyuubi-server/src/main/scala/org/apache/kyuubi/util/KubernetesUtils.scala
+++ b/kyuubi-server/src/main/scala/org/apache/kyuubi/util/KubernetesUtils.scala
@@ -120,8 +120,8 @@ object KubernetesUtils extends Logging {
     opt2.foreach { _ => require(opt1.isEmpty, errMessage) }
   }
 
-  private def getResourceNamePrefix(appName: String, engineRefId: String): 
String = {
-    s"$appName-$engineRefId"
+  private def getResourceNamePrefix(appName: String, engineRefId: 
Option[String]): String = {
+    engineRefId.map(refId => s"$appName-$refId").getOrElse(appName)
       .trim
       .toLowerCase(Locale.ROOT)
       .replaceAll("[^a-z0-9\\-]", "-")
@@ -134,7 +134,16 @@ object KubernetesUtils extends Logging {
       appName: String,
       engineRefId: String,
       forciblyRewrite: Boolean): String = {
-    lazy val resolvedResourceName = s"kyuubi-${getResourceNamePrefix(appName, 
engineRefId)}-driver"
+    val resourceNamePrefix = if (appName.contains(engineRefId)) {
+      getResourceNamePrefix(appName, None)
+    } else {
+      getResourceNamePrefix(appName, Some(engineRefId))
+    }
+    val resolvedResourceName = if (resourceNamePrefix.startsWith("kyuubi-")) {
+      s"$resourceNamePrefix-driver"
+    } else {
+      s"kyuubi-$resourceNamePrefix-driver"
+    }
     if (forciblyRewrite || resolvedResourceName.length > 
DRIVER_POD_NAME_MAX_LENGTH) {
       s"kyuubi-$engineRefId-driver"
     } else {
@@ -146,7 +155,16 @@ object KubernetesUtils extends Logging {
       appName: String,
       engineRefId: String,
       forciblyRewrite: Boolean): String = {
-    val resolvedResourceName = s"kyuubi-${getResourceNamePrefix(appName, 
engineRefId)}"
+    val resourceNamePrefix = if (appName.contains(engineRefId)) {
+      getResourceNamePrefix(appName, None)
+    } else {
+      getResourceNamePrefix(appName, Some(engineRefId))
+    }
+    val resolvedResourceName = if (resourceNamePrefix.startsWith("kyuubi-")) {
+      s"$resourceNamePrefix"
+    } else {
+      s"kyuubi-$resourceNamePrefix"
+    }
     if (forciblyRewrite || resolvedResourceName.length > 
EXECUTOR_POD_NAME_PREFIX_MAX_LENGTH) {
       s"kyuubi-$engineRefId"
     } else {