This is an automated email from the ASF dual-hosted git repository.
chengpan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new e4a064470 [KYUUBI #6129] [BUILD] Remove the extension spark jars
before build to avoid include unspecified version
e4a064470 is described below
commit e4a06447065e5a91ff4ab98713d8b820ca45e524
Author: zwangsheng <[email protected]>
AuthorDate: Wed Mar 6 11:59:23 2024 +0800
[KYUUBI #6129] [BUILD] Remove the extension spark jars before build to
avoid include unspecified version
# :mag: Description
## Issue References ๐
This pull request fixes
https://github.com/apache/kyuubi/pull/5836#discussion_r1424963477
## Describe Your Solution ๐ง
Remove the remain extension spark jars in related target dir before start
to package.
Aims to avoid include unspecified version.
## Types of changes :bookmark:
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
## Test Plan ๐งช
#### Behavior Without This Pull Request :coffin:
Before this PR:
```shel
./build/dist -Pspark-3.1 ...
# your will get expected package, then you run the following command
./build/dist -Pspark-3.3
# you will find the final package include the spark 3.1 extension jar,
which is un-expected.
```
#### Behavior With This Pull Request :tada:
After this PR, will clean up those extension spark jars before build to
ensure we won't package the unspecified extension jar.
#### Related Unit Tests
None
---
# Checklist ๐
- [x] This patch was not authored or co-authored using [Generative
Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6129 from zwangsheng/KYUUBI/clean_up_jasrs.
Closes #6129
3e5f43415 [Cheng Pan] Update build/dist
d824491be [zwangsheng] Remove support spark 3.1
c83b79af6 [zwangsheng] [BUILD] Remove the extension spark jars before build
Lead-authored-by: zwangsheng <[email protected]>
Co-authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
---
build/dist | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/build/dist b/build/dist
index cde515dc6..74c04679c 100755
--- a/build/dist
+++ b/build/dist
@@ -233,6 +233,13 @@ if [[ "$HIVE_PROVIDED" == "true" ]]; then
MVN_DIST_OPT="$MVN_DIST_OPT -Phive-provided"
fi
+# clean up all versions of Spark extension, to avoid collecting outdated
artifacts
+SPARK_EXTENSION_VERSIONS=('3-2' '3-3' '3-4' '3-5')
+# shellcheck disable=SC2068
+for SPARK_EXTENSION_VERSION in ${SPARK_EXTENSION_VERSIONS[@]}; do
+ rm -f
"$KYUUBI_HOME/extensions/spark/kyuubi-extension-spark-$SPARK_EXTENSION_VERSION/target/kyuubi-extension-spark-${SPARK_EXTENSION_VERSION}_${SCALA_VERSION}-${VERSION}.jar"
+done
+
export MAVEN_OPTS="${MAVEN_OPTS:--Xmx2g}"
BUILD_COMMAND=("$MVN" clean install $MVN_DIST_OPT $@)
@@ -331,7 +338,6 @@ for jar in $(ls "$DISTDIR/jars/"); do
done
# Copy Kyuubi Spark extension
-SPARK_EXTENSION_VERSIONS=('3-1' '3-2' '3-3' '3-4' '3-5')
# shellcheck disable=SC2068
for SPARK_EXTENSION_VERSION in ${SPARK_EXTENSION_VERSIONS[@]}; do
if [[ -f
$"$KYUUBI_HOME/extensions/spark/kyuubi-extension-spark-$SPARK_EXTENSION_VERSION/target/kyuubi-extension-spark-${SPARK_EXTENSION_VERSION}_${SCALA_VERSION}-${VERSION}.jar"
]]; then
