pan3793 commented on code in PR #5279:
URL: https://github.com/apache/kyuubi/pull/5279#discussion_r1325285104
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/server/KyuubiBatchService.scala:
##########
@@ -83,26 +84,28 @@ class KyuubiBatchService(
metadata.requestArgs,
Some(metadata),
fromRecovery = false)
- val sessionHandle = sessionManager.openBatchSession(batchSession)
+ sessionManager.openBatchSession(batchSession)
var submitted = false
while (!submitted) { // block until batch job submitted
- submitted = sessionManager.getBatchSession(sessionHandle).map {
batchSession =>
- val batchState =
batchSession.batchJobSubmissionOp.getStatus.state
- batchState == OperationState.RUNNING ||
OperationState.isTerminal(batchState)
- }.getOrElse {
- error(s"Batch Session $batchId is not existed, marked as
finished")
- true
+ submitted = metadataManager.getBatchSessionMetadata(batchId)
match {
+ case Some(metadata) if
OperationState.isTerminal(metadata.opState) =>
+ true
+ case Some(metadata) if metadata.opState ==
OperationState.RUNNING =>
+ metadata.appState match {
+ // app that is not submitted to resource manager
Review Comment:
The purpose of such concurrency limitation is that the SparkSubmit process
swallows too many CPU and memory resources, and the SparkSubmit process won't
exit when the app state is PENDING, thus PENDING should be treated as not
submitted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
