This is an automated email from the ASF dual-hosted git repository.
chengpan pushed a commit to branch branch-1.10
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/branch-1.10 by this push:
new e41e4ff11b [KYUUBI #6998] [TEST] Harness SparkProcessBuilderSuite
e41e4ff11b is described below
commit e41e4ff11b7e86e5e9a353c4cb6c0e4f5b76b350
Author: Cheng Pan <[email protected]>
AuthorDate: Tue Mar 25 17:28:58 2025 +0800
[KYUUBI #6998] [TEST] Harness SparkProcessBuilderSuite
### Why are the changes needed?
Fix the missing `assert` in `SparkProcessBuilderSuite - spark process
builder`.
Fix the flaky test `SparkProcessBuilderSuite - capture error from spark
process builder` by increasing `kyuubi.session.engine.startup.maxLogLines` from
10 to 4096, this is easy to fail, especially in Spark 4.0 due to increased
error stack trace. for example,
https://github.com/apache/kyuubi/actions/runs/13974413470/job/39290129824
```
SparkProcessBuilderSuite:
- spark process builder
- capture error from spark process builder *** FAILED ***
The code passed to eventually never returned normally. Attempted 167
times over 1.5007926256666668 minutes. Last failure message:
"org.apache.kyuubi.KyuubiSQLException: Suppressed:
org.apache.spark.util.Utils$OriginalTryStackTraceException: Full stacktrace of
original doTryWithCallerStacktrace caller
See more:
/home/runner/work/kyuubi/kyuubi/kyuubi-server/target/work/kentyao/kyuubi-spark-sql-engine.log.2
at
org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69)
at
org.apache.kyuubi.engine.ProcBuilder.$anonfun$start$1(ProcBuilder.scala:239)
at java.base/java.lang.Thread.run(Thread.java:1583)
.
FYI: The last 10 line(s) of log are:
25/03/24 12:53:39 INFO MemoryStore: MemoryStore started with capacity
434.4 MiB
25/03/24 12:53:39 INFO MemoryStore: MemoryStore cleared
25/03/24 12:53:39 INFO BlockManager: BlockManager stopped
25/03/24 12:53:39 INFO BlockManagerMaster: BlockManagerMaster stopped
25/03/24 12:53:39 INFO
OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
OutputCommitCoordinator stopped!
25/03/24 12:53:39 INFO SparkContext: Successfully stopped SparkContext
25/03/24 12:53:39 INFO ShutdownHookManager: Shutdown hook called
25/03/24 12:53:39 INFO ShutdownHookManager: Deleting directory
/tmp/spark-18455622-344e-48ac-92eb-4b368c35e697
25/03/24 12:53:39 INFO ShutdownHookManager: Deleting directory
/home/runner/work/kyuubi/kyuubi/kyuubi-server/target/work/kentyao/artifacts/spark-7479249b-44a2-4fe5-aa0f-544074f9c356
25/03/24 12:53:39 INFO ShutdownHookManager: Deleting directory
/tmp/spark-5ba8250f-1ff2-4e0d-a365-27d7518308e1" did not contain
"org.apache.hadoop.hive.ql.metadata.HiveException:".
(SparkProcessBuilderSuite.scala:77)
```
### How was this patch tested?
Pass GHA, and verified locally with Spark 4.0.0 RC3 by running tests 10
times with constant success.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #6998 from pan3793/spark-pb-ut.
Closes #6998
a4290b413 [Cheng Pan] harness SparkProcessBuilderSuite
Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
(cherry picked from commit 6459680d89726cc7c1d483caae2b48992d168e3f)
Signed-off-by: Cheng Pan <[email protected]>
---
.../apache/kyuubi/engine/spark/SparkProcessBuilderSuite.scala | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git
a/kyuubi-server/src/test/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilderSuite.scala
b/kyuubi-server/src/test/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilderSuite.scala
index 5f3bae1249..972b410922 100644
---
a/kyuubi-server/src/test/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilderSuite.scala
+++
b/kyuubi-server/src/test/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilderSuite.scala
@@ -28,7 +28,7 @@ import org.scalatestplus.mockito.MockitoSugar
import org.apache.kyuubi._
import org.apache.kyuubi.config.KyuubiConf
-import org.apache.kyuubi.config.KyuubiConf.{ENGINE_LOG_TIMEOUT,
ENGINE_SPARK_MAIN_RESOURCE, KUBERNETES_FORCIBLY_REWRITE_DRIVER_POD_NAME,
KUBERNETES_FORCIBLY_REWRITE_EXEC_POD_NAME_PREFIX}
+import org.apache.kyuubi.config.KyuubiConf._
import org.apache.kyuubi.engine.ProcBuilder.KYUUBI_ENGINE_LOG_PATH_KEY
import org.apache.kyuubi.engine.spark.SparkProcessBuilder._
import org.apache.kyuubi.ha.HighAvailabilityConf
@@ -38,15 +38,17 @@ import org.apache.kyuubi.util.AssertionUtils._
import org.apache.kyuubi.util.command.CommandLineUtils._
class SparkProcessBuilderSuite extends KerberizedTestHelper with MockitoSugar {
- private def conf = KyuubiConf().set("kyuubi.on", "off")
+ private def conf = KyuubiConf()
+ .set(SESSION_ENGINE_STARTUP_MAX_LOG_LINES, 4096)
+ .set("kyuubi.on", "off")
test("spark process builder") {
val builder = new SparkProcessBuilder("kentyao", true, conf)
val commands = builder.toString.split(' ')
assert(commands(2) === "org.apache.kyuubi.engine.spark.SparkSQLEngine")
assert(commands.contains("spark.kyuubi.on=off"))
- builder.toString.contains("\\\n\t--class")
- builder.toString.contains("\\\n\t--conf spark.kyuubi.on=off")
+ assert(builder.toString.contains("\\\n\t--class"))
+ assert(builder.toString.contains("\\\n\t--conf spark.kyuubi.on=off"))
val pb = new ProcessBuilder(commands.head, "--help")
assert(pb.start().waitFor() === 0)
assert(Files.exists(Paths.get(commands.last)))
