yanghua commented on a change in pull request #193:
URL:
https://github.com/apache/incubator-yunikorn-k8shim/pull/193#discussion_r495666540
##########
File path: deployments/examples/spark30/cmd/run.sh
##########
@@ -16,12 +16,40 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-# replace these env var values accordingly
-SPARK_HOME=/Users/example/repository/spark
+SCRIPT_PATH=$(cd `dirname $0`; pwd)
+# set up root directory
+WORK_SPACE_ROOT=`dirname $SCRIPT_PATH`
+
+SPARK_VERSION=3.0.0
+SPARK_HADOOP_VERSION=2.7
+
+SPARK_BINARY_FILE_NAME=spark-${SPARK_VERSION}-bin-hadoop${SPARK_HADOOP_VERSION}.tgz
+SPARK_BINARY_CHECKSUM_FILE_NAME=${SPARK_BINARY_FILE_NAME}.sha512
+SPARK_BINARY_FILE_PATH=$WORK_SPACE_ROOT/$SPARK_BINARY_FILE_NAME
+SPARK_BINARY_CHECKSUM_FILE_PATH=$WORK_SPACE_ROOT/$SPARK_BINARY_CHECKSUM_FILE_NAME
+SPARK_HOME=$WORK_SPACE_ROOT/spark-${SPARK_VERSION}-bin-hadoop${SPARK_HADOOP_VERSION}
+SPARK_EXAMPLE_JAR=local://$SPARK_HOME/examples/jars/spark-examples_2.12-${SPARK_VERSION}.jar
+
K8S_ENDPOINT=http://localhost:8001
SPARK_EXECUTOR_NUM=1
SPARK_DOCKER_IMAGE=yunikorn/spark:latest
-SPARK_EXAMPLE_JAR=local:///opt/spark/examples/jars/spark-examples_2.12-3.0.0-SNAPSHOT.jar
+
+if [ -f $SPARK_BINARY_FILE_PATH ]; then
+ echo "The binary file $SPARK_BINARY_FILE_NAME has been cached!"
+else
+ echo "The binary file $SPARK_BINARY_FILE_NAME did not exist, try to
download."
+ wget
http://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/${SPARK_BINARY_FILE_NAME}
-O ${SPARK_BINARY_FILE_PATH}
+fi
+
+# check signature to verify the completeness
+if [[ $(cat $SPARK_BINARY_CHECKSUM_FILE_PATH) == $(shasum -a 512
$SPARK_BINARY_FILE_PATH | awk '{print $1}') ]]; then
+ echo "The checksum is matched!"
+ rm -rf
$WORK_SPACE_ROOT/spark-${SPARK_VERSION}-bin-hadoop${SPARK_HADOOP_VERSION}
Review comment:
I thought about this problem, but we cannot guarantee whether the
existing folder has been changed. If it is unpredictable, why not just delete
it based on the same standard file first. The decompression process does not
consume more time.
Here, I think our goal is to let new users experience it. If it is not
sufficiently automated and requires too high learning costs, then it will bring
some rebellious emotions.
WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
