(kyuubi) branch branch-1.9 updated: [KYUUBI #6463] Release semaphore immediately after startup process exit

[chengpan]

This is an automated email from the ASF dual-hosted git repository.

chengpan pushed a commit to branch branch-1.9
in repository https://gitbox.apache.org/repos/asf/kyuubi.git


The following commit(s) were added to refs/heads/branch-1.9 by this push:
     new 31c1fa163 [KYUUBI #6463] Release semaphore immediately after startup 
process exit
31c1fa163 is described below

commit 31c1fa16374d5319f53d367f7bcda4542089f531
Author: ic4y <i...@apache.org>
AuthorDate: Thu Jun 13 21:10:21 2024 +0800

    [KYUUBI #6463] Release semaphore immediately after startup process exit
    
    # :mag: Description
    ## Issue References 🔗
    
    The concurrency limit for the engine startup process is mainly used to 
avoid overload on the machine(or container) of the Kyuubi server, the current 
implementation holds startupProcessSemaphore until the session is established 
successfully. While for Spark on YARN cluster mode, some YARN queue resource 
insufficiency may block the subsequent Spark application submissions to other 
queues, significantly affecting the Kyuubi server's resource utilization.
    
    ## Describe Your Solution 🔧
    
    We should immediately release the `startupProcessSemaphore` after the 
engine startup process exits (i.e., after the `spark-submit` process exits) as 
the load has already disappeared.
    
    ## Types of changes :bookmark:
    
    - [x] Bugfix (non-breaking change which fixes an issue)
    - [ ] New feature (non-breaking change which adds functionality)
    - [ ] Breaking change (fix or feature that would cause existing 
functionality to change)
    
    ## Test Plan 🧪
    
    I tested it on a cluster of 50 kyuubi Servers, and kyuubi server resource 
utilization increased by 70%
    
    ---
    
    # Checklist 📝
    
    - [ ] This patch was not authored or co-authored using [Generative 
Tooling](https://www.apache.org/legal/generative-tooling.html)
    
    **Be nice. Be informative.**
    
    Closes #6463 from ic4y/master-p003.
    
    Closes #6463
    
    f7de68ce3 [ic4y] Improve code quality
    d8b0248df [ic4y] [Improve][EngineRef] Optimize Engine Startup Concurrency 
Limit
    
    Authored-by: ic4y <i...@apache.org>
    Signed-off-by: Cheng Pan <cheng...@apache.org>
    (cherry picked from commit 95ed74821c3a2d3a3f402033de7b463966a4bc28)
    Signed-off-by: Cheng Pan <cheng...@apache.org>
---
 .../src/main/scala/org/apache/kyuubi/engine/EngineRef.scala          | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git 
a/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/EngineRef.scala 
b/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/EngineRef.scala
index 0bb13b049..664d10080 100644
--- a/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/EngineRef.scala
+++ b/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/EngineRef.scala
@@ -232,7 +232,10 @@ private[kyuubi] class EngineRef(
       while (engineRef.isEmpty) {
         if (exitValue.isEmpty && process.waitFor(1, TimeUnit.SECONDS)) {
           exitValue = Some(process.exitValue())
-          if (!exitValue.contains(0)) {
+          if (exitValue.contains(0)) {
+            acquiredPermit = false
+            startupProcessSemaphore.foreach(_.release())
+          } else {
             val error = builder.getError
             MetricsSystem.tracing { ms =>
               ms.incCount(MetricRegistry.name(ENGINE_FAIL, appUser))