wilfred-s commented on a change in pull request #193:
URL:
https://github.com/apache/incubator-yunikorn-k8shim/pull/193#discussion_r495694075
##########
File path: deployments/examples/spark30/cmd/run.sh
##########
@@ -16,12 +16,40 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-# replace these env var values accordingly
-SPARK_HOME=/Users/example/repository/spark
+SCRIPT_PATH=$(cd `dirname $0`; pwd)
+# set up root directory
+WORK_SPACE_ROOT=`dirname $SCRIPT_PATH`
+
+SPARK_VERSION=3.0.0
+SPARK_HADOOP_VERSION=2.7
+
+SPARK_BINARY_FILE_NAME=spark-${SPARK_VERSION}-bin-hadoop${SPARK_HADOOP_VERSION}.tgz
+SPARK_BINARY_CHECKSUM_FILE_NAME=${SPARK_BINARY_FILE_NAME}.sha512
+SPARK_BINARY_FILE_PATH=$WORK_SPACE_ROOT/$SPARK_BINARY_FILE_NAME
+SPARK_BINARY_CHECKSUM_FILE_PATH=$WORK_SPACE_ROOT/$SPARK_BINARY_CHECKSUM_FILE_NAME
+SPARK_HOME=$WORK_SPACE_ROOT/spark-${SPARK_VERSION}-bin-hadoop${SPARK_HADOOP_VERSION}
+SPARK_EXAMPLE_JAR=local://$SPARK_HOME/examples/jars/spark-examples_2.12-${SPARK_VERSION}.jar
+
K8S_ENDPOINT=http://localhost:8001
SPARK_EXECUTOR_NUM=1
SPARK_DOCKER_IMAGE=yunikorn/spark:latest
-SPARK_EXAMPLE_JAR=local:///opt/spark/examples/jars/spark-examples_2.12-3.0.0-SNAPSHOT.jar
+
+if [ -f $SPARK_BINARY_FILE_PATH ]; then
+ echo "The binary file $SPARK_BINARY_FILE_NAME has been cached!"
+else
+ echo "The binary file $SPARK_BINARY_FILE_NAME did not exist, try to
download."
+ wget
http://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/${SPARK_BINARY_FILE_NAME}
-O ${SPARK_BINARY_FILE_PATH}
+fi
+
+# check signature to verify the completeness
+if [[ $(cat $SPARK_BINARY_CHECKSUM_FILE_PATH) == $(shasum -a 512
$SPARK_BINARY_FILE_PATH | awk '{print $1}') ]]; then
+ echo "The checksum is matched!"
+ rm -rf
$WORK_SPACE_ROOT/spark-${SPARK_VERSION}-bin-hadoop${SPARK_HADOOP_VERSION}
Review comment:
Make the behaviour clear, always using a clean unpacked version is OK
but the user needs to be made aware that you are doing it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
