This is an automated email from the ASF dual-hosted git repository.
chengpan pushed a commit to branch branch-1.8
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/branch-1.8 by this push:
new 919b79bea [KYUUBI #6463] Release semaphore immediately after startup
process exit
919b79bea is described below
commit 919b79bea6ae0b3b2c558cfb9546ec0239027b7b
Author: ic4y <[email protected]>
AuthorDate: Thu Jun 13 21:10:21 2024 +0800
[KYUUBI #6463] Release semaphore immediately after startup process exit
The concurrency limit for the engine startup process is mainly used to
avoid overload on the machine(or container) of the Kyuubi server, the current
implementation holds startupProcessSemaphore until the session is established
successfully. While for Spark on YARN cluster mode, some YARN queue resource
insufficiency may block the subsequent Spark application submissions to other
queues, significantly affecting the Kyuubi server's resource utilization.
We should immediately release the `startupProcessSemaphore` after the
engine startup process exits (i.e., after the `spark-submit` process exits) as
the load has already disappeared.
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
I tested it on a cluster of 50 kyuubi Servers, and kyuubi server resource
utilization increased by 70%
---
- [ ] This patch was not authored or co-authored using [Generative
Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6463 from ic4y/master-p003.
Closes #6463
f7de68ce3 [ic4y] Improve code quality
d8b0248df [ic4y] [Improve][EngineRef] Optimize Engine Startup Concurrency
Limit
Authored-by: ic4y <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
(cherry picked from commit 95ed74821c3a2d3a3f402033de7b463966a4bc28)
Signed-off-by: Cheng Pan <[email protected]>
---
.../src/main/scala/org/apache/kyuubi/engine/EngineRef.scala | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git
a/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/EngineRef.scala
b/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/EngineRef.scala
index 952ed1ef5..af5c67906 100644
--- a/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/EngineRef.scala
+++ b/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/EngineRef.scala
@@ -222,7 +222,10 @@ private[kyuubi] class EngineRef(
while (engineRef.isEmpty) {
if (exitValue.isEmpty && process.waitFor(1, TimeUnit.SECONDS)) {
exitValue = Some(process.exitValue())
- if (exitValue != Some(0)) {
+ if (exitValue.contains(0)) {
+ acquiredPermit = false
+ startupProcessSemaphore.foreach(_.release())
+ } else {
val error = builder.getError
MetricsSystem.tracing { ms =>
ms.incCount(MetricRegistry.name(ENGINE_FAIL, appUser))
