This is an automated email from the ASF dual-hosted git repository. ztang pushed a commit to annotated tag release-0.3.0-RC0 in repository https://gitbox.apache.org/repos/asf/submarine.git
commit c4470ec5d36827b3b350e15e29dc60db38cea1cf Author: Zhankun Tang <[email protected]> AuthorDate: Wed Jan 22 14:39:26 2020 +0800 SUBMARINE-349. Support using existing artifacts to build mini-submarine image ### What is this PR for? The hard-coded submarine version(0.3.0-SNAPSHOT) in build_mini-submarine.sh is not convenient. When doing a release, we need the image to package a candidate artifacts like 0.3.0. This JIRA is for extending the build script to package specified artifacts. Setting environment variable submarine_version and release_candidates_path can trigger this code path. It will copy binary tarball with the submarine_version to the local dir and build the image. ### What type of PR is it? Improvement ### What is the Jira issue? https://issues.apache.org/jira/browse/SUBMARINE-349 ### How should this be tested? Put submarine candidate aritifacts to a folder like "~/releases/submarine-release" $ ls $release_candidates_path submarine-dist-0.3.0-hadoop-2.9.tar.gz submarine-dist-0.3.0-src.tar.gz.asc submarine-dist-0.3.0-hadoop-2.9.tar.gz.asc submarine-dist-0.3.0-src.tar.gz.sha512 submarine-dist-0.3.0-hadoop-2.9.tar.gz.sha512 submarine-dist-0.3.0-src.tar.gz export submarine_version=0.3.0 export release_candidates_path=~/releases/submarine-release ./build_mini-submarine.sh docker run -it -h submarine-dev --net=bridge --privileged -P local/mini-submarine:0.3.0 /bin/bash In the container, check the submarine jar version is 0.3.0 ### Questions: * Does the licenses files need an update? No * Is there breaking changes for older versions? No * Does this needs documentation? Yes Author: Zhankun Tang <[email protected]> Closes #158 from tangzhankun/submarine-349 and squashes the following commits: 165421a [Zhankun Tang] Simplify variable default value assignment and fix a typo in document 9a807c0 [Zhankun Tang] Refine the doc 099b966 [Zhankun Tang] Add doc 82990d2 [Zhankun Tang] Support using existing artifacts to build mini-submarine image (cherry picked from commit 15d5c915a4023af6713e0b3d6afd33fb44ad879f) --- dev-support/mini-submarine/Dockerfile | 2 +- dev-support/mini-submarine/README.md | 20 ++++++++++++++++++++ dev-support/mini-submarine/build_mini-submarine.sh | 19 ++++++++++++++++--- .../submarine/run_customized_submarine-all_mnist.sh | 3 ++- .../submarine/run_submarine_mnist_tony.sh | 4 ++-- .../submarine/run_submarine_mnist_tony_rpc.sh | 3 ++- 6 files changed, 43 insertions(+), 8 deletions(-) diff --git a/dev-support/mini-submarine/Dockerfile b/dev-support/mini-submarine/Dockerfile index bdb8f13..3c369ee 100644 --- a/dev-support/mini-submarine/Dockerfile +++ b/dev-support/mini-submarine/Dockerfile @@ -116,7 +116,7 @@ ENV IMAGE_N=${IMAGE_NAME} # Install Submarine ARG SUBMARINE_VERSION= ENV SUBMARINE_VER=${SUBMARINE_VERSION} -ADD submarine-dist-${SUBMARINE_VER}*.tar.gz /opt +ADD submarine-dist-${SUBMARINE_VER}-hadoop*.tar.gz /opt ADD pysubmarine /opt/pysubmarine RUN ln -s /opt/submarine-dist-${SUBMARINE_VER}* /opt/submarine-current ADD submarine /home/yarn/submarine diff --git a/dev-support/mini-submarine/README.md b/dev-support/mini-submarine/README.md index a067bed..bc657bf 100644 --- a/dev-support/mini-submarine/README.md +++ b/dev-support/mini-submarine/README.md @@ -52,6 +52,26 @@ mvn clean install package -DskipTests cd submarine/dev-support/mini-submarine/ ./build_mini-submarine.sh ``` +#### Package An Existing Release Candidates +When doing release, the release manager might needs to package a artifact candidates in this docker image and public the image candidate for a vote. +In this scenario, we can do this: + +Put submarine candidate aritifacts to a folder like "~/releases/submarine-release" +``` +$ ls $release_candidates_path +submarine-dist-0.3.0-hadoop-2.9.tar.gz submarine-dist-0.3.0-src.tar.gz.asc +submarine-dist-0.3.0-hadoop-2.9.tar.gz.asc submarine-dist-0.3.0-src.tar.gz.sha512 +submarine-dist-0.3.0-hadoop-2.9.tar.gz.sha512 submarine-dist-0.3.0-src.tar.gz +``` +``` +export submarine_version=0.3.0 +export release_candidates_path=~/releases/submarine-release +./build_mini-submarine.sh +#docker run -it -h submarine-dev --net=bridge --privileged -P local/mini-submarine:0.3.0 /bin/bash +docker tag local/mini-submarine:0.3.0 apache/mini-submarine:0.3.0:RC0 +docker push apache/mini-submarine:0.3.0:RC0 +``` +In the container, we can verify that the submarine jar version is the expected 0.3.0. Then we can upload this image with a "RC" tag for a vote. ### Run mini-submarine image diff --git a/dev-support/mini-submarine/build_mini-submarine.sh b/dev-support/mini-submarine/build_mini-submarine.sh index 2d40e29..54b0f34 100755 --- a/dev-support/mini-submarine/build_mini-submarine.sh +++ b/dev-support/mini-submarine/build_mini-submarine.sh @@ -13,10 +13,13 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. - +set -e hadoop_v=2.9.2 spark_v=2.4.4 -submarine_v=0.3.0-SNAPSHOT + +submarine_v=${submarine_version:-"0.3.0-SNAPSHOT"} +echo "Using submarine version: $submarine_v" + image_name="local/mini-submarine:${submarine_v}" if [ -L ${BASH_SOURCE-$0} ]; then @@ -55,10 +58,20 @@ download_package "spark-${spark_v}-bin-hadoop2.7.tgz" "http://mirrors.tuna.tsing # download zookeeper download_package "zookeeper-3.4.14.tar.gz" "http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.14"; + if [ ! -d "${SUBMARINE_PROJECT_PATH}/submarine-dist/target" ]; then mkdir "${SUBMARINE_PROJECT_PATH}/submarine-dist/target" fi + submarine_dist_exists=$(find -L "${SUBMARINE_PROJECT_PATH}/submarine-dist/target" -name "submarine-dist-${submarine_v}*.tar.gz") + +# If exists, use the release candidate artifacts to build image +if [[ ! -z "${release_candidates_path}" ]]; then + submarine_dist_exists=${release_candidates_path} + echo "Using release candidates artifacts: ${release_candidates_path}" + cp ${release_candidates_path}/submarine-dist-${submarine_v}-hadoop*.tar.gz ${MINI_PATH} +fi + # Build source code if the package doesn't exist. if [[ -z "${submarine_dist_exists}" ]]; then # update tony code @@ -70,9 +83,9 @@ if [[ -z "${submarine_dist_exists}" ]]; then cd "${SUBMARINE_PROJECT_PATH}" mvn clean package -DskipTests + cp ${SUBMARINE_PROJECT_PATH}/submarine-dist/target/submarine-dist-${submarine_v}*.tar.gz ${MINI_PATH} fi -cp ${SUBMARINE_PROJECT_PATH}/submarine-dist/target/submarine-dist-${submarine_v}*.tar.gz ${MINI_PATH} cp -r ${SUBMARINE_PROJECT_PATH}/submarine-sdk/pysubmarine ${MINI_PATH} cp -r ${SUBMARINE_PROJECT_PATH}/docs/database ${MINI_PATH} diff --git a/dev-support/mini-submarine/submarine/run_customized_submarine-all_mnist.sh b/dev-support/mini-submarine/submarine/run_customized_submarine-all_mnist.sh index f2432d2..6d3c047 100755 --- a/dev-support/mini-submarine/submarine/run_customized_submarine-all_mnist.sh +++ b/dev-support/mini-submarine/submarine/run_customized_submarine-all_mnist.sh @@ -49,7 +49,8 @@ else WORKER_CMD="myvenv.zip/venv/bin/python mnist_distributed.py --steps 2 --data_dir /tmp/data --working_dir /tmp/mode" fi -SUBMARINE_VERSION=0.3.0-SNAPSHOT +SUBMARINE_VERSION=${SUBMARINE_VER:-"0.3.0-SNAPSHOT"} + HADOOP_VERSION=2.9 ${JAVA_CMD} -cp /tmp/submarine-all-${SUBMARINE_VERSION}-hadoop-${HADOOP_VERSION}.jar:/usr/local/hadoop/etc/hadoop \ diff --git a/dev-support/mini-submarine/submarine/run_submarine_mnist_tony.sh b/dev-support/mini-submarine/submarine/run_submarine_mnist_tony.sh index 93d923e..4711536 100755 --- a/dev-support/mini-submarine/submarine/run_submarine_mnist_tony.sh +++ b/dev-support/mini-submarine/submarine/run_submarine_mnist_tony.sh @@ -17,7 +17,7 @@ # Below are configurable variables, please adapt base on your local environment. # Version of submarine jar -SUBMARINE_VERSION=0.3.0-SNAPSHOT +SUBMARINE_VERSION=${SUBMARINE_VER:-"0.3.0-SNAPSHOT"} # Version of affiliated Hadoop version for this Submarine jar. SUBMARINE_HADOOP_VERSION=2.9 @@ -74,7 +74,7 @@ else WORKER_CMD="myvenv.zip/$WORKER_CMD" fi -${JAVA_CMD} -cp $(${HADOOP_HOME}/bin/hadoop classpath --glob):${SUBMARINE_PATH}/submarine-all-${SUBMARINE_VERSION}-hadoop-${SUBMARINE_HADOOP_VERSION}.jar:${HADOOP_CONF_PATH} \ +${JAVA_CMD} -cp $(${HADOOP_COMMON_HOME}/bin/hadoop classpath --glob):${SUBMARINE_PATH}/submarine-all-${SUBMARINE_VERSION}-hadoop-${SUBMARINE_HADOOP_VERSION}.jar:${HADOOP_CONF_PATH} \ org.apache.submarine.client.cli.Cli job run --name tf-job-001 \ --framework tensorflow \ --verbose \ diff --git a/dev-support/mini-submarine/submarine/run_submarine_mnist_tony_rpc.sh b/dev-support/mini-submarine/submarine/run_submarine_mnist_tony_rpc.sh index ed016ae..b55f74a 100644 --- a/dev-support/mini-submarine/submarine/run_submarine_mnist_tony_rpc.sh +++ b/dev-support/mini-submarine/submarine/run_submarine_mnist_tony_rpc.sh @@ -47,7 +47,8 @@ else WORKER_CMD="myvenv.zip/venv/bin/python mnist_distributed.py --steps 2 --data_dir /tmp/data --working_dir /tmp/mode" fi -SUBMARINE_VERSION=0.3.0-SNAPSHOT +SUBMARINE_VERSION=${SUBMARINE_VER:-"0.3.0-SNAPSHOT"} + HADOOP_VERSION=2.9 ${JAVA_CMD} -cp /opt/submarine-dist-${SUBMARINE_VERSION}-hadoop-${HADOOP_VERSION}/submarine-all-${SUBMARINE_VERSION}-hadoop-${HADOOP_VERSION}.jar:/usr/local/hadoop/etc/hadoop:/opt/submarine-dist-${SUBMARINE_VERSION}-hadoop-${HADOOP_VERSION}/conf \ --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
