This is an automated email from the ASF dual-hosted git repository.
ulyssesyou pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new a5bb93e5d [KYUUBI #2636] Refine BatchesResourceSuite
a5bb93e5d is described below
commit a5bb93e5d66856e87a38b8c49a3ea17cbd113c7a
Author: Fu Chen <[email protected]>
AuthorDate: Thu May 12 16:15:27 2022 +0800
[KYUUBI #2636] Refine BatchesResourceSuite
### _Why are the changes needed?_
refine BatchesResourceSuite
the BatchesResourceSuite failed after upgrade spark to 3.3.0
https://github.com/apache/incubator-kyuubi/runs/6384637613?check_suite_focus=true
```
BatchesResourceSuite:
- open batch session *** FAILED ***
The code passed to eventually never returned normally. Attempted 20 times
over 10.176988683 seconds. Last failure message: logs.exists(((x$2: String) =>
x$2.contains("/bin/spark-submit"))) was true, but logs.exists(((x$3: String) =>
x$3.contains(scala.StringContext.apply("spark.SparkContext: Submitted
application: ", "").s(appName)))) was false. (BatchesResourceSuite.scala:94)
- get batch session list *** FAILED ***
List(Batch(d860929f-f1a8-4f0d-890d-d06d895b48d8,spark,Map(id -> 21071,
name -> org.apache.spark.deploy.SparkSubmit --conf spark.master=local --conf
spark.kyuubi.session.engine.spark.max.lifetime=5000 --conf
spark.kyuubi.session.engine.check.interval=1000 --conf
spark.yarn.tags=d860929f-f1a8-4f0d-890d-d06d895b48d8 --conf
spark.app.name=spark-batch-submission --class
org.apache.kyuubi.engine.spark.SparkSQLEngine --proxy-user anonymous
/home/runner/work/incubator-kyuubi/incubator-kyuu [...]
```
- spark 3.3 has a different console layer pattern after migrating to log4j2.
Spark 3.3
```
22/05/12 11:19:57 INFO SparkContext: Submitted application: Spark Pi
```
Spark 3.2 and previous version
```
22/05/12 11:42:18 INFO spark.SparkContext: Submitted application: Spark Pi
```
> Actually, I can't figure out what causes this change.
- clean up batch job after each unit test.
### _How was this patch tested?_
Pass CI
Closes #2636 from cfmcgrady/fix-batch-suite.
Closes #2636
815fdf5a [Fu Chen] address comment
9878d1d0 [Fu Chen] address comment
db17e826 [Fu Chen] refine BatchesResourceSuite
Authored-by: Fu Chen <[email protected]>
Signed-off-by: ulysses-you <[email protected]>
---
.../kyuubi/server/api/v1/BatchesResourceSuite.scala | 19 +++++++++++++++++--
1 file changed, 17 insertions(+), 2 deletions(-)
diff --git
a/kyuubi-server/src/test/scala/org/apache/kyuubi/server/api/v1/BatchesResourceSuite.scala
b/kyuubi-server/src/test/scala/org/apache/kyuubi/server/api/v1/BatchesResourceSuite.scala
index e2ff28ab5..a6160357f 100644
---
a/kyuubi-server/src/test/scala/org/apache/kyuubi/server/api/v1/BatchesResourceSuite.scala
+++
b/kyuubi-server/src/test/scala/org/apache/kyuubi/server/api/v1/BatchesResourceSuite.scala
@@ -33,9 +33,24 @@ import
org.apache.kyuubi.config.KyuubiConf.{ENGINE_CHECK_INTERVAL, ENGINE_SPARK_
import org.apache.kyuubi.engine.spark.SparkProcessBuilder
import
org.apache.kyuubi.server.http.authentication.AuthenticationHandler.AUTHORIZATION_HEADER
import org.apache.kyuubi.service.authentication.KyuubiAuthenticationFactory
-import org.apache.kyuubi.session.KyuubiSessionManager
+import org.apache.kyuubi.session.{KyuubiBatchSessionImpl, KyuubiSessionManager}
class BatchesResourceSuite extends KyuubiFunSuite with RestFrontendTestHelper {
+
+ override def afterEach(): Unit = {
+ val sessionManager = fe.be.sessionManager
+ sessionManager.asInstanceOf[KyuubiSessionManager]
+ .getBatchSessionList(null, 0, Int.MaxValue)
+ .map(_.asInstanceOf[KyuubiBatchSessionImpl])
+ .foreach { session =>
+ try {
+ session.batchJobSubmissionOp.killBatchApplication()
+ } finally {
+ sessionManager.closeSession(session.handle)
+ }
+ }
+ }
+
test("open batch session") {
val sparkProcessBuilder = new SparkProcessBuilder("kyuubi", conf)
val appName = "spark-batch-submission"
@@ -104,7 +119,7 @@ class BatchesResourceSuite extends KyuubiFunSuite with
RestFrontendTestHelper {
// check both kyuubi log and engine log
assert(logs.exists(_.contains("/bin/spark-submit")) && logs.exists(
- _.contains(s"spark.SparkContext: Submitted application: $appName")))
+ _.contains(s"SparkContext: Submitted application: $appName")))
}
// invalid user name
