This is an automated email from the ASF dual-hosted git repository.
feiwang pushed a commit to branch branch-1.8
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/branch-1.8 by this push:
new e66ef4540 [KYUUBI #5196][FOLLOWUP] Extract spark core scala version
lazily and respect engine env
e66ef4540 is described below
commit e66ef4540131778f958675580dd5abee87569cdb
Author: fwang12 <[email protected]>
AuthorDate: Mon Oct 16 23:46:55 2023 +0800
[KYUUBI #5196][FOLLOWUP] Extract spark core scala version lazily and
respect engine env
### _Why are the changes needed?_
Only extract the spark core scala version if `SPARK_SCALA_VERSION` env is
empty, and respect engine env.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including
negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run
test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests)
locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes #5434 from turboFei/lazy_scala_version.
Closes #5196
fdccef77b [fwang12] lazy extract spark core scala version
Authored-by: fwang12 <[email protected]>
Signed-off-by: fwang12 <[email protected]>
(cherry picked from commit c60f5b7e8872efb15597b962da1873a4ba2e71ef)
Signed-off-by: fwang12 <[email protected]>
---
.../scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala
b/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala
index 02f4064af..afc96fb5e 100644
---
a/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala
+++
b/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala
@@ -108,9 +108,9 @@ class SparkProcessBuilder(
}
override protected val engineScalaBinaryVersion: String = {
- val sparkCoreScalaVersion =
+ env.get("SPARK_SCALA_VERSION").filter(StringUtils.isNotBlank).getOrElse {
extractSparkCoreScalaVersion(Paths.get(sparkHome, "jars").toFile.list())
- StringUtils.defaultIfBlank(System.getenv("SPARK_SCALA_VERSION"),
sparkCoreScalaVersion)
+ }
}
override protected lazy val engineHomeDirFilter: FileFilter = file => {
