This is an automated email from the ASF dual-hosted git repository.
zhouquan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/submarine.git
The following commit(s) were added to refs/heads/master by this push:
new 15d5c91 SUBMARINE-349. Support using existing artifacts to build
mini-submarine image
15d5c91 is described below
commit 15d5c915a4023af6713e0b3d6afd33fb44ad879f
Author: Zhankun Tang <[email protected]>
AuthorDate: Wed Jan 22 14:39:26 2020 +0800
SUBMARINE-349. Support using existing artifacts to build mini-submarine
image
### What is this PR for?
The hard-coded submarine version(0.3.0-SNAPSHOT) in build_mini-submarine.sh
is not convenient. When doing a release, we need the image to package a
candidate artifacts like 0.3.0.
This JIRA is for extending the build script to package specified artifacts.
Setting environment variable submarine_version and release_candidates_path can
trigger this code path.
It will copy binary tarball with the submarine_version to the local dir and
build the image.
### What type of PR is it?
Improvement
### What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-349
### How should this be tested?
Put submarine candidate aritifacts to a folder like
"~/releases/submarine-release"
$ ls $release_candidates_path
submarine-dist-0.3.0-hadoop-2.9.tar.gz
submarine-dist-0.3.0-src.tar.gz.asc
submarine-dist-0.3.0-hadoop-2.9.tar.gz.asc
submarine-dist-0.3.0-src.tar.gz.sha512
submarine-dist-0.3.0-hadoop-2.9.tar.gz.sha512
submarine-dist-0.3.0-src.tar.gz
export submarine_version=0.3.0
export release_candidates_path=~/releases/submarine-release
./build_mini-submarine.sh
docker run -it -h submarine-dev --net=bridge --privileged -P
local/mini-submarine:0.3.0 /bin/bash
In the container, check the submarine jar version is 0.3.0
### Questions:
* Does the licenses files need an update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? Yes
Author: Zhankun Tang <[email protected]>
Closes #158 from tangzhankun/submarine-349 and squashes the following
commits:
165421a [Zhankun Tang] Simplify variable default value assignment and fix a
typo in document
9a807c0 [Zhankun Tang] Refine the doc
099b966 [Zhankun Tang] Add doc
82990d2 [Zhankun Tang] Support using existing artifacts to build
mini-submarine image
---
dev-support/mini-submarine/Dockerfile | 2 +-
dev-support/mini-submarine/README.md | 20 ++++++++++++++++++++
dev-support/mini-submarine/build_mini-submarine.sh | 19 ++++++++++++++++---
.../submarine/run_customized_submarine-all_mnist.sh | 3 ++-
.../submarine/run_submarine_mnist_tony.sh | 4 ++--
.../submarine/run_submarine_mnist_tony_rpc.sh | 3 ++-
6 files changed, 43 insertions(+), 8 deletions(-)
diff --git a/dev-support/mini-submarine/Dockerfile
b/dev-support/mini-submarine/Dockerfile
index bdb8f13..3c369ee 100644
--- a/dev-support/mini-submarine/Dockerfile
+++ b/dev-support/mini-submarine/Dockerfile
@@ -116,7 +116,7 @@ ENV IMAGE_N=${IMAGE_NAME}
# Install Submarine
ARG SUBMARINE_VERSION=
ENV SUBMARINE_VER=${SUBMARINE_VERSION}
-ADD submarine-dist-${SUBMARINE_VER}*.tar.gz /opt
+ADD submarine-dist-${SUBMARINE_VER}-hadoop*.tar.gz /opt
ADD pysubmarine /opt/pysubmarine
RUN ln -s /opt/submarine-dist-${SUBMARINE_VER}* /opt/submarine-current
ADD submarine /home/yarn/submarine
diff --git a/dev-support/mini-submarine/README.md
b/dev-support/mini-submarine/README.md
index a067bed..bc657bf 100644
--- a/dev-support/mini-submarine/README.md
+++ b/dev-support/mini-submarine/README.md
@@ -52,6 +52,26 @@ mvn clean install package -DskipTests
cd submarine/dev-support/mini-submarine/
./build_mini-submarine.sh
```
+#### Package An Existing Release Candidates
+When doing release, the release manager might needs to package a artifact
candidates in this docker image and public the image candidate for a vote.
+In this scenario, we can do this:
+
+Put submarine candidate aritifacts to a folder like
"~/releases/submarine-release"
+```
+$ ls $release_candidates_path
+submarine-dist-0.3.0-hadoop-2.9.tar.gz
submarine-dist-0.3.0-src.tar.gz.asc
+submarine-dist-0.3.0-hadoop-2.9.tar.gz.asc
submarine-dist-0.3.0-src.tar.gz.sha512
+submarine-dist-0.3.0-hadoop-2.9.tar.gz.sha512 submarine-dist-0.3.0-src.tar.gz
+```
+```
+export submarine_version=0.3.0
+export release_candidates_path=~/releases/submarine-release
+./build_mini-submarine.sh
+#docker run -it -h submarine-dev --net=bridge --privileged -P
local/mini-submarine:0.3.0 /bin/bash
+docker tag local/mini-submarine:0.3.0 apache/mini-submarine:0.3.0:RC0
+docker push apache/mini-submarine:0.3.0:RC0
+```
+In the container, we can verify that the submarine jar version is the expected
0.3.0. Then we can upload this image with a "RC" tag for a vote.
### Run mini-submarine image
diff --git a/dev-support/mini-submarine/build_mini-submarine.sh
b/dev-support/mini-submarine/build_mini-submarine.sh
index 2d40e29..54b0f34 100755
--- a/dev-support/mini-submarine/build_mini-submarine.sh
+++ b/dev-support/mini-submarine/build_mini-submarine.sh
@@ -13,10 +13,13 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-
+set -e
hadoop_v=2.9.2
spark_v=2.4.4
-submarine_v=0.3.0-SNAPSHOT
+
+submarine_v=${submarine_version:-"0.3.0-SNAPSHOT"}
+echo "Using submarine version: $submarine_v"
+
image_name="local/mini-submarine:${submarine_v}"
if [ -L ${BASH_SOURCE-$0} ]; then
@@ -55,10 +58,20 @@ download_package "spark-${spark_v}-bin-hadoop2.7.tgz"
"http://mirrors.tuna.tsing
# download zookeeper
download_package "zookeeper-3.4.14.tar.gz"
"http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.14";
+
if [ ! -d "${SUBMARINE_PROJECT_PATH}/submarine-dist/target" ]; then
mkdir "${SUBMARINE_PROJECT_PATH}/submarine-dist/target"
fi
+
submarine_dist_exists=$(find -L
"${SUBMARINE_PROJECT_PATH}/submarine-dist/target" -name
"submarine-dist-${submarine_v}*.tar.gz")
+
+# If exists, use the release candidate artifacts to build image
+if [[ ! -z "${release_candidates_path}" ]]; then
+ submarine_dist_exists=${release_candidates_path}
+ echo "Using release candidates artifacts: ${release_candidates_path}"
+ cp ${release_candidates_path}/submarine-dist-${submarine_v}-hadoop*.tar.gz
${MINI_PATH}
+fi
+
# Build source code if the package doesn't exist.
if [[ -z "${submarine_dist_exists}" ]]; then
# update tony code
@@ -70,9 +83,9 @@ if [[ -z "${submarine_dist_exists}" ]]; then
cd "${SUBMARINE_PROJECT_PATH}"
mvn clean package -DskipTests
+ cp
${SUBMARINE_PROJECT_PATH}/submarine-dist/target/submarine-dist-${submarine_v}*.tar.gz
${MINI_PATH}
fi
-cp
${SUBMARINE_PROJECT_PATH}/submarine-dist/target/submarine-dist-${submarine_v}*.tar.gz
${MINI_PATH}
cp -r ${SUBMARINE_PROJECT_PATH}/submarine-sdk/pysubmarine ${MINI_PATH}
cp -r ${SUBMARINE_PROJECT_PATH}/docs/database ${MINI_PATH}
diff --git
a/dev-support/mini-submarine/submarine/run_customized_submarine-all_mnist.sh
b/dev-support/mini-submarine/submarine/run_customized_submarine-all_mnist.sh
index f2432d2..6d3c047 100755
--- a/dev-support/mini-submarine/submarine/run_customized_submarine-all_mnist.sh
+++ b/dev-support/mini-submarine/submarine/run_customized_submarine-all_mnist.sh
@@ -49,7 +49,8 @@ else
WORKER_CMD="myvenv.zip/venv/bin/python mnist_distributed.py --steps 2
--data_dir /tmp/data --working_dir /tmp/mode"
fi
-SUBMARINE_VERSION=0.3.0-SNAPSHOT
+SUBMARINE_VERSION=${SUBMARINE_VER:-"0.3.0-SNAPSHOT"}
+
HADOOP_VERSION=2.9
${JAVA_CMD} -cp
/tmp/submarine-all-${SUBMARINE_VERSION}-hadoop-${HADOOP_VERSION}.jar:/usr/local/hadoop/etc/hadoop
\
diff --git a/dev-support/mini-submarine/submarine/run_submarine_mnist_tony.sh
b/dev-support/mini-submarine/submarine/run_submarine_mnist_tony.sh
index 93d923e..4711536 100755
--- a/dev-support/mini-submarine/submarine/run_submarine_mnist_tony.sh
+++ b/dev-support/mini-submarine/submarine/run_submarine_mnist_tony.sh
@@ -17,7 +17,7 @@
# Below are configurable variables, please adapt base on your local
environment.
# Version of submarine jar
-SUBMARINE_VERSION=0.3.0-SNAPSHOT
+SUBMARINE_VERSION=${SUBMARINE_VER:-"0.3.0-SNAPSHOT"}
# Version of affiliated Hadoop version for this Submarine jar.
SUBMARINE_HADOOP_VERSION=2.9
@@ -74,7 +74,7 @@ else
WORKER_CMD="myvenv.zip/$WORKER_CMD"
fi
-${JAVA_CMD} -cp $(${HADOOP_HOME}/bin/hadoop classpath
--glob):${SUBMARINE_PATH}/submarine-all-${SUBMARINE_VERSION}-hadoop-${SUBMARINE_HADOOP_VERSION}.jar:${HADOOP_CONF_PATH}
\
+${JAVA_CMD} -cp $(${HADOOP_COMMON_HOME}/bin/hadoop classpath
--glob):${SUBMARINE_PATH}/submarine-all-${SUBMARINE_VERSION}-hadoop-${SUBMARINE_HADOOP_VERSION}.jar:${HADOOP_CONF_PATH}
\
org.apache.submarine.client.cli.Cli job run --name tf-job-001 \
--framework tensorflow \
--verbose \
diff --git
a/dev-support/mini-submarine/submarine/run_submarine_mnist_tony_rpc.sh
b/dev-support/mini-submarine/submarine/run_submarine_mnist_tony_rpc.sh
index ed016ae..b55f74a 100644
--- a/dev-support/mini-submarine/submarine/run_submarine_mnist_tony_rpc.sh
+++ b/dev-support/mini-submarine/submarine/run_submarine_mnist_tony_rpc.sh
@@ -47,7 +47,8 @@ else
WORKER_CMD="myvenv.zip/venv/bin/python mnist_distributed.py --steps 2
--data_dir /tmp/data --working_dir /tmp/mode"
fi
-SUBMARINE_VERSION=0.3.0-SNAPSHOT
+SUBMARINE_VERSION=${SUBMARINE_VER:-"0.3.0-SNAPSHOT"}
+
HADOOP_VERSION=2.9
${JAVA_CMD} -cp
/opt/submarine-dist-${SUBMARINE_VERSION}-hadoop-${HADOOP_VERSION}/submarine-all-${SUBMARINE_VERSION}-hadoop-${HADOOP_VERSION}.jar:/usr/local/hadoop/etc/hadoop:/opt/submarine-dist-${SUBMARINE_VERSION}-hadoop-${HADOOP_VERSION}/conf
\
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
