cfmcgrady opened a new pull request, #2636:
URL: https://github.com/apache/incubator-kyuubi/pull/2636
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in
https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your
PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g.,
'[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
refine BatchesResourceSuite
the BatchesResourceSuite fail after upgrade spark to 3.3.0
https://github.com/apache/incubator-kyuubi/runs/6384637613?check_suite_focus=true
```
BatchesResourceSuite:
- open batch session *** FAILED ***
The code passed to eventually never returned normally. Attempted 20 times
over 10.176988683 seconds. Last failure message: logs.exists(((x$2: String) =>
x$2.contains("/bin/spark-submit"))) was true, but logs.exists(((x$3: String) =>
x$3.contains(scala.StringContext.apply("spark.SparkContext: Submitted
application: ", "").s(appName)))) was false. (BatchesResourceSuite.scala:94)
- get batch session list *** FAILED ***
List(Batch(d860929f-f1a8-4f0d-890d-d06d895b48d8,spark,Map(id -> 21071,
name -> org.apache.spark.deploy.SparkSubmit --conf spark.master=local --conf
spark.kyuubi.session.engine.spark.max.lifetime=5000 --conf
spark.kyuubi.session.engine.check.interval=1000 --conf
spark.yarn.tags=d860929f-f1a8-4f0d-890d-d06d895b48d8 --conf
spark.app.name=spark-batch-submission --class
org.apache.kyuubi.engine.spark.SparkSQLEngine --proxy-user anonymous
/home/runner/work/incubator-kyuubi/incubator-kyuubi/externals/kyuubi-spark-sql-engine/target/kyuubi-spark-sql-engine_2.12-1.6.0-SNAPSHOT.jar,
state -> RUNNING),localhost:37119,CLOSED)) was not empty
(BatchesResourceSuite.scala:169)
```
- spark 3.3 has a different console layer pattern after migrating to log4j2.
Spark 3.2 and previous version
```
22/05/12 11:19:57 INFO SparkContext: Submitted application: Spark Pi
```
Spark 3.3
```
22/05/12 11:42:18 INFO spark.SparkContext: Submitted application: Spark Pi
```
> Actually, I can't figure out what causes this change.
- clean up batch job after each unit test.
### _How was this patch tested?_
Pass CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
