[kyuubi] branch master updated: [KYUUBI #5206] Try to kill pod with label if no ApplicationInfo found to prevent pod leak

[feiwang]

This is an automated email from the ASF dual-hosted git repository.

feiwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kyuubi.git


The following commit(s) were added to refs/heads/master by this push:
     new be48b9404 [KYUUBI #5206] Try to kill pod with label if no 
ApplicationInfo found to prevent pod leak
be48b9404 is described below

commit be48b9404a9e371d277b6f62d40a64e964eccc63
Author: fwang12 <fwan...@ebay.com>
AuthorDate: Tue Aug 29 13:26:38 2023 +0800

    [KYUUBI #5206] Try to kill pod with label if no ApplicationInfo found to 
prevent pod leak
    
    ### _Why are the changes needed?_
    
    Now for `KubernetesApplicationOperation`, it rely on the appInfoStore.
    
    For batch rest api, if the closeBatch request can not be send to the 
kyuubiInstance that created the batch, the current kyuubiInstance will try to 
kill the batch.
    
    I wonder that, in this case, the applicationInfo might can not be found in 
the appInfoStore.
    
    It is better to try the best to kill the pod with  `kyuubi-unique-tag` 
label to prevent pod leak.
    
    ### _How was this patch tested?_
    - [ ] Add some test cases that check the changes thoroughly including 
negative and positive cases if possible
    
    - [ ] Add screenshots for manual tests if appropriate
    
    - [x] [Run 
test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests)
 locally before make a pull request
    
    ### _Was this patch authored or co-authored using generative AI tooling?_
    
    Closes #5206 from turboFei/k8s_pod_kill.
    
    Closes #5206
    
    630bcd04b [fwang12] warning
    a9c22e03f [fwang12] delete with label
    
    Authored-by: fwang12 <fwan...@ebay.com>
    Signed-off-by: fwang12 <fwan...@ebay.com>
---
 .../engine/KubernetesApplicationOperation.scala    | 29 ++++++++++++++--------
 1 file changed, 19 insertions(+), 10 deletions(-)

diff --git 
a/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KubernetesApplicationOperation.scala
 
b/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KubernetesApplicationOperation.scala
index b61abc30b..7ddd0366b 100644
--- 
a/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KubernetesApplicationOperation.scala
+++ 
b/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KubernetesApplicationOperation.scala
@@ -125,18 +125,27 @@ class KubernetesApplicationOperation extends 
ApplicationOperation with Logging {
     val kubernetesClient = getOrCreateKubernetesClient(kubernetesInfo)
     debug(s"[$kubernetesInfo] Deleting application info from Kubernetes 
cluster by $tag tag")
     try {
-      val info = appInfoStore.getOrDefault(tag, ApplicationInfo.NOT_FOUND)
-      debug(s"Application info[tag: $tag] is in ${info.state}")
-      info.state match {
-        case NOT_FOUND | FAILED | UNKNOWN =>
-          (
-            false,
-            s"[$kubernetesInfo] Target application[tag: $tag] is in 
${info.state} status")
-        case _ =>
+      Option(appInfoStore.get(tag)) match {
+        case Some(info) =>
+          debug(s"Application info[tag: $tag] is in ${info.state}")
+          info.state match {
+            case NOT_FOUND | FAILED | UNKNOWN =>
+              (
+                false,
+                s"[$kubernetesInfo] Target application[tag: $tag] is in 
${info.state} status")
+            case _ =>
+              (
+                !kubernetesClient.pods.withName(info.name).delete().isEmpty,
+                s"[$kubernetesInfo] Operation of deleted" +
+                  s" application[appId: ${info.id} ,tag: $tag] is completed")
+          }
+        case None =>
+          warn(s"No application info found for tag[$tag]," +
+            s" trying to delete pod with label [$LABEL_KYUUBI_UNIQUE_KEY -> 
$tag]")
           (
-            !kubernetesClient.pods.withName(info.name).delete().isEmpty,
+            !kubernetesClient.pods.withLabel(LABEL_KYUUBI_UNIQUE_KEY, 
tag).delete().isEmpty,
             s"[$kubernetesInfo] Operation of deleted" +
-              s" application[appId: ${info.id} ,tag: $tag] is completed")
+              s" pod with label [$LABEL_KYUUBI_UNIQUE_KEY -> $tag] is 
completed")
       }
     } catch {
       case e: Exception =>