craigcondit commented on code in PR #553:
URL: https://github.com/apache/yunikorn-k8shim/pull/553#discussion_r1152538539
##########
scripts/run-e2e-tests.sh:
##########
@@ -101,12 +101,15 @@ function install_kind() {
}
function install_spark() {
- check_cmd "wget"
- wget -qO-
https://archive.apache.org/dist/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz |
tar xzvf - && chmod +x spark-3.3.1-bin-hadoop3 && sudo mv
spark-3.3.1-bin-hadoop3 /opt/.
- exit_on_error "install spark failed."
- sudo ln -s /opt/spark-3.3.1-bin-hadoop3 /opt/spark
- exit_on_error "install spark failed. unable to create symlink"
- export SPARK_HOME=/opt/spark/
+ PWD=$(pwd)
+ if [ ! -d "${PWD}/_spark" ]; then
+ echo "Installing spark in yunikorn rep."
+ check_cmd "wget"
+ mkdir "${PWD}/_spark"
+ curl
https://archive.apache.org/dist/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz |
tar xzvf - && chmod +x spark-3.3.1-bin-hadoop3 && mv spark-3.3.1-bin-hadoop3
"${PWD}/_spark/."
Review Comment:
Instead of moving, change into the _spark directory first -- this way the
archive will go into the correct place and we don't litter up the top level
directory if the script aborts.
##########
scripts/run-e2e-tests.sh:
##########
@@ -101,12 +101,15 @@ function install_kind() {
}
function install_spark() {
- check_cmd "wget"
- wget -qO-
https://archive.apache.org/dist/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz |
tar xzvf - && chmod +x spark-3.3.1-bin-hadoop3 && sudo mv
spark-3.3.1-bin-hadoop3 /opt/.
- exit_on_error "install spark failed."
- sudo ln -s /opt/spark-3.3.1-bin-hadoop3 /opt/spark
- exit_on_error "install spark failed. unable to create symlink"
- export SPARK_HOME=/opt/spark/
+ PWD=$(pwd)
+ if [ ! -d "${PWD}/_spark" ]; then
+ echo "Installing spark in yunikorn rep."
+ check_cmd "wget"
Review Comment:
Remove this as we no longer need wget.
##########
scripts/run-e2e-tests.sh:
##########
@@ -101,12 +101,15 @@ function install_kind() {
}
function install_spark() {
- check_cmd "wget"
- wget -qO-
https://archive.apache.org/dist/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz |
tar xzvf - && chmod +x spark-3.3.1-bin-hadoop3 && sudo mv
spark-3.3.1-bin-hadoop3 /opt/.
- exit_on_error "install spark failed."
- sudo ln -s /opt/spark-3.3.1-bin-hadoop3 /opt/spark
- exit_on_error "install spark failed. unable to create symlink"
- export SPARK_HOME=/opt/spark/
+ PWD=$(pwd)
+ if [ ! -d "${PWD}/_spark" ]; then
Review Comment:
This should probably check for the whole directory we want to use (i.e.
_spark/spark-3.3.1-bin-hadoop3), and if not, start with `rm -rf
"${PWD}_spark"`. This way we can transparently upgrade in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
