wilfred-s commented on a change in pull request #193:
URL:
https://github.com/apache/incubator-yunikorn-k8shim/pull/193#discussion_r495692795
##########
File path: deployments/examples/spark30/cmd/run.sh
##########
@@ -16,12 +16,40 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-# replace these env var values accordingly
-SPARK_HOME=/Users/example/repository/spark
+SCRIPT_PATH=$(cd `dirname $0`; pwd)
+# set up root directory
+WORK_SPACE_ROOT=`dirname $SCRIPT_PATH`
+
+SPARK_VERSION=3.0.0
+SPARK_HADOOP_VERSION=2.7
+
+SPARK_BINARY_FILE_NAME=spark-${SPARK_VERSION}-bin-hadoop${SPARK_HADOOP_VERSION}.tgz
+SPARK_BINARY_CHECKSUM_FILE_NAME=${SPARK_BINARY_FILE_NAME}.sha512
+SPARK_BINARY_FILE_PATH=$WORK_SPACE_ROOT/$SPARK_BINARY_FILE_NAME
+SPARK_BINARY_CHECKSUM_FILE_PATH=$WORK_SPACE_ROOT/$SPARK_BINARY_CHECKSUM_FILE_NAME
+SPARK_HOME=$WORK_SPACE_ROOT/spark-${SPARK_VERSION}-bin-hadoop${SPARK_HADOOP_VERSION}
+SPARK_EXAMPLE_JAR=local://$SPARK_HOME/examples/jars/spark-examples_2.12-${SPARK_VERSION}.jar
+
K8S_ENDPOINT=http://localhost:8001
SPARK_EXECUTOR_NUM=1
SPARK_DOCKER_IMAGE=yunikorn/spark:latest
-SPARK_EXAMPLE_JAR=local:///opt/spark/examples/jars/spark-examples_2.12-3.0.0-SNAPSHOT.jar
+
+if [ -f $SPARK_BINARY_FILE_PATH ]; then
+ echo "The binary file $SPARK_BINARY_FILE_NAME has been cached!"
+else
+ echo "The binary file $SPARK_BINARY_FILE_NAME did not exist, try to
download."
+ wget
http://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/${SPARK_BINARY_FILE_NAME}
-O ${SPARK_BINARY_FILE_PATH}
+fi
+
+# check signature to verify the completeness
+if [[ $(cat $SPARK_BINARY_CHECKSUM_FILE_PATH) == $(shasum -a 512
$SPARK_BINARY_FILE_PATH | awk '{print $1}') ]]; then
Review comment:
The offical spark checksum is generated using this command:
```
gpg --print-md sha512 spark-3.0.1-bin-hadoop2.7.tgz
spark-3.0.1-bin-hadoop2.7.tgz: F4A10BAE C5B8FF18 41F10651 CAC2C4AA 39C162D3
029CA180 A9749149 E6060805 B5B5DDF9 287B4AA3
21434810 172F8CC0 534943AC 005531BB 48B6622F
BE228DDC
```
There are really only two versions: `gpg` and `shasum`
It is also really simple to generate the format you want using:
```
cat spark-3.0.1-bin-hadoop2.7.tgz.sha512 | tr -d " \t\n\r" | awk -F: '{print
$2 " " $1}' > converted.sha512
```
That output is the same as the shasum output. Redirect into a file and use
`shasum -c -a 512 converted.sha512` and you are good to go
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
