This is an automated email from the ASF dual-hosted git repository.
zhouquan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/submarine.git
The following commit(s) were added to refs/heads/master by this push:
new 1eb3fcb SUBMARINE-331. Update mini-submarine's workbench-ng
1eb3fcb is described below
commit 1eb3fcb376605f49db658e436c123eca3c829d11
Author: Xun Liu <[email protected]>
AuthorDate: Wed Jan 1 20:20:55 2020 +0800
SUBMARINE-331. Update mini-submarine's workbench-ng
### What is this PR for?
Submarine is now ready to use the angularJS version of workbench WEB, So
mini-submarine needs to be updated.
### What type of PR is it?
Improvement
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/SUBMARINE-331
### How should this be tested?
* [CI Pass](https://travis-ci.org/liuxunorg/submarine/builds/631777961)
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Xun Liu <[email protected]>
Closes #140 from liuxunorg/SUBMARINE-331 and squashes the following commits:
4c32e6b [Xun Liu] SUBMARINE-331. Update mini-submarine's workbench-ng
---
conf/submarine-site.xml.template | 2 +-
dev-support/mini-submarine/Dockerfile | 12 ++--
dev-support/mini-submarine/README.md | 65 +++++++++++++---------
dev-support/mini-submarine/build_mini-submarine.sh | 32 +++++++++--
dev-support/mini-submarine/conf/submarine-site.xml | 2 +-
submarine-workbench/workbench-web-ng/package.json | 2 +-
6 files changed, 74 insertions(+), 41 deletions(-)
diff --git a/conf/submarine-site.xml.template b/conf/submarine-site.xml.template
index 42dbb10..86dc941 100755
--- a/conf/submarine-site.xml.template
+++ b/conf/submarine-site.xml.template
@@ -103,7 +103,7 @@
<property>
<name>workbench.web.war</name>
- <value>../workbench-web.war</value>
+ <value>../workbench-web-ng.war</value>
<description>Submarine workbench web war file path.</description>
</property>
diff --git a/dev-support/mini-submarine/Dockerfile
b/dev-support/mini-submarine/Dockerfile
index 1f65149..bdb8f13 100644
--- a/dev-support/mini-submarine/Dockerfile
+++ b/dev-support/mini-submarine/Dockerfile
@@ -22,7 +22,7 @@ RUN apt-get -q update \
ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64/jre
-#INSTALL Docker
+# INSTALL Docker
RUN \
apt-get update && \
apt-get -y install apt-transport-https ca-certificates curl
software-properties-common && \
@@ -91,18 +91,19 @@ RUN \
COPY conf /tmp/hadoop-config
ENV HADOOP_VER=${HADOOP_VERSION}
+
#Install Spark
ARG SPARK_VERSION=
ENV SPARK_VER=${SPARK_VERSION}
ADD spark-${SPARK_VERSION}-bin-hadoop2.7.tgz /opt
ADD spark-defaults-dynamic-allocation.conf
/opt/spark-${SPARK_VERSION}/conf/spark-defaults.conf
-RUN \
- apt-get update && \
- apt-get install -y vim python python-numpy wget zip python3
+RUN apt-get update && apt-get install -y vim python python-numpy wget zip
python3
+
# Add pyspark sample
ADD spark-script /home/yarn/spark-script
RUN chown -R yarn /home/yarn/spark-script && \
chmod +x -R /home/yarn/spark-script
+
# Add distributedShell example
ADD conf/yarn-ds-docker.sh /home/yarn
RUN chown -R yarn /home/yarn/yarn-ds-docker.sh && \
@@ -127,4 +128,5 @@ RUN cd /home/yarn/submarine && \
/home/yarn/submarine/build_python_virtual_env.sh
# Grant read permission for submarine job
-RUN chown yarn /home/yarn/submarine
+RUN chown -R yarn /home/yarn/submarine
+RUN chown -R yarn /opt/*
diff --git a/dev-support/mini-submarine/README.md
b/dev-support/mini-submarine/README.md
index 2d59ce8..74afd0d 100644
--- a/dev-support/mini-submarine/README.md
+++ b/dev-support/mini-submarine/README.md
@@ -18,9 +18,9 @@ This is a docker image built for submarine development and
quick start test.
**Please Note: don't use the image in production environment. It's only for
test purpose.**
-### Start mini-submarine
+## Start mini-submarine
-#### Use the image we provide
+### Use the image we provide
> Tag 0.3.0-SNAPSHOT indicates the version number of hadoop submarine in images
@@ -28,7 +28,7 @@ This is a docker image built for submarine development and
quick start test.
docker pull hadoopsubmarine/mini-submarine:0.3.0-SNAPSHOT
```
-#### Create image by yourself
+### Create image by yourself
> You may need a VPN if your network is limited
@@ -53,10 +53,10 @@ cd submarine/dev-support/mini-submarine/
./build_mini-submarine.sh
```
-#### Run mini-submarine image
+### Run mini-submarine image
```
-docker run -it -h submarine-dev --net=bridge --privileged -P
local/mini-submarine:0.3.0-SNAPSHOT /bin/bash
+docker run -it -h submarine-dev --name mini-submarine --net=bridge
--privileged -P local/mini-submarine:0.3.0-SNAPSHOT /bin/bash
# In the container, use root user to bootstrap hdfs and yarn
/tmp/hadoop-config/bootstrap.sh
@@ -67,7 +67,7 @@ yarn node -list -showDetails
If you pull the image directly, please replace
"local/mini-submarine:0.3.0-SNAPSHOT" with
"hadoopsubmarine/mini-submarine:0.3.0-SNAPSHOT".
-#### You should see info like this:
+### You should see info like this:
```
Total Nodes:1
@@ -87,38 +87,49 @@ hdfs dfs -ls /user
> drwxr-xr-x - yarn supergroup 0 2019-07-22 07:59 /user/yarn
-### Run workbench server
+## Run workbench server
1. Setup mysql mariadb server
+> Because mysql and mariadb use the GPL license, you need to manually execute
the script to install it.
+
```
/tmp/hadoop-config/setup-mysql.sh
```
-2. Start workbench server
+You can execute command `mysql -uroot` login mysql mariadb.
+
+2. Start submarine server
```
su yarn
-/opt/submarine-current/bin/workbench-daemon.sh start getMysqlJar
+/opt/submarine-current/bin/submarine-daemon.sh start getMysqlJar
```
-### Run a sumbarine job
+3. login submarine workbench
+You can use the following command to get the login URL of the submarine
workbench
+```shell
+echo "http://localhost:$(docker inspect --format='{{(index (index
.NetworkSettings.Ports "8080/tcp") 0).HostPort}}' mini-submarine)"
+```
+The URL returned by the command (like to: http://localhost:32819) is opened
through a browser. The username and initial password of the workbench are both
`admin`.
+
+## Run a sumbarine job
-#### Switch to user yarn
+### Switch to user yarn
```
su yarn
```
-#### Run a mnist TF job with submarine + TonY runtime
+### Run a mnist TF job with submarine + TonY runtime
```
-cd && cd submarine && ./run_submarine_mnist_tony.sh
+/home/yarn/submarine/run_submarine_mnist_tony.sh
```
When run_submarine_mnist_tony.sh is executed, mnist data is download from the
url, [google mnist](https://storage.googleapis.com/cvdf-datasets/mnist/), by
default. If the url is unaccessible, you can use parameter "-d" to specify a
customized url.
For example, if you are in mainland China, you can use the following command
```
-cd && cd submarine && ./run_submarine_mnist_tony.sh -d
http://yann.lecun.com/exdb/mnist/
+/home/yarn/submarine/run_submarine_mnist_tony.sh -d
http://yann.lecun.com/exdb/mnist/
```
-#### Run a mnist TF job via submarine server
+### Run a mnist TF job via submarine server
Submarine server is supposed to manage jobs lifecycle. Clients can just submit
job parameters or yaml file to submarine server instead of submitting jobs
@@ -138,7 +149,7 @@ Run the following command to submit a job via submarine
server
./run_submarine_mnist_tony_rpc.sh
```
-#### Try your own submarine program
+### Try your own submarine program
Run container with your source code. You can also use "docker cp" to an
existing running container
@@ -149,24 +160,24 @@ Run container with your source code. You can also use
"docker cp" to an existing
3. Try to run it. Since this is a single node environment, keep in mind that
the workers could have conflicts with each other. For instance, the
mnist_distributed.py example has a workaround to fix the conflicts when two
workers are using same "data_dir" to download data set.
-### Update Submarine Version
+## Update Submarine Version
You can follow the documentation instructions to update your own modified and
compiled submarine package to the submarine container.
-#### Build Submarine
+### Build Submarine
```
cd submarine-project-dir/
mvn clean install package -DskipTests
```
-#### Copy submarine jar to mini-submarine container
+### Copy submarine jar to mini-submarine container
```
docker cp
submarine-all/target/submarine-all-<SUBMARINE_VERSION>-hadoop-<HADOOP_VERSION>.jar
<container-id>:/tmp/
```
-#### Modify environment variables
+### Modify environment variables
```
cd /home/yarn/submarine
@@ -177,18 +188,18 @@ SUBMARINE_VERSION=<submarine-version-number>
HADOOP_VERSION=<hadoop-version-number> # default 2.9
```
-#### Test submarine jar package in container
+### Test submarine jar package in container
```
cd /home/yarn/submarine
./run_customized_submarine-all_mnist.sh
```
-### Debug Submarine
+## Debug Submarine
When using mini-submarine, you can debug submarine client, applicationMaster
and executor for trouble shooting.
-#### <span id="debug">Debug submarine client</span>
+### <span id="debug">Debug submarine client</span>
Run the following command to start mini-submarine.
@@ -217,7 +228,7 @@ For example, we can get some info like this
Then port 32804 can be used for remote debug.
-#### Debug submarine job applicationMaster
+### Debug submarine job applicationMaster
Run the following command to start mini-submarine.
@@ -237,7 +248,7 @@ Add the following configuration in the file
/usr/local/hadoop/etc/hadoop/tony.xm
You can use run_submarine_mnist_tony.sh to submit a job. Port 8001 is used for
AM debugging in mini-submarine.
And the debug port mapping can be obtained using the way as [Debug submarine
client](#debug) shows.
-#### Debug submarine job executor
+### Debug submarine job executor
Run the following command to start mini-submarine.
@@ -265,7 +276,7 @@ submarine job should be like this
You can get the debug port mapping using the way as [Debug submarine
client](#debug) shows.
-### Run a distributedShell job with docker container
+## Run a distributedShell job with docker container
You can also run a distributedShell job in mini-submarine.
@@ -273,7 +284,7 @@ You can also run a distributedShell job in mini-submarine.
cd && ./yarn-ds-docker.sh
```
-### Run a spark job
+## Run a spark job
Spark jobs are supported as well.
diff --git a/dev-support/mini-submarine/build_mini-submarine.sh
b/dev-support/mini-submarine/build_mini-submarine.sh
index d83d1fa..2d40e29 100755
--- a/dev-support/mini-submarine/build_mini-submarine.sh
+++ b/dev-support/mini-submarine/build_mini-submarine.sh
@@ -25,6 +25,7 @@ else
PWD=$(dirname ${BASH_SOURCE-$0})
fi
export MINI_PATH=$(cd "${PWD}">/dev/null; pwd)
+SUBMARINE_PROJECT_PATH=${MINI_PATH}/../..
download_package() {
if [ -f "$1" ]; then
@@ -43,6 +44,10 @@ download_package() {
fi
}
+is_empty_dir(){
+ return `ls -A $1|wc -w`
+}
+
# download hadoop
download_package "hadoop-${hadoop_v}.tar.gz"
"http://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-${hadoop_v}";
# download spark
@@ -50,18 +55,33 @@ download_package "spark-${spark_v}-bin-hadoop2.7.tgz"
"http://mirrors.tuna.tsing
# download zookeeper
download_package "zookeeper-3.4.14.tar.gz"
"http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.14";
-cd ../..
-submarine_dist_exists=$(find -L "submarine-dist/target" -name
"submarine-dist-${submarine_v}*.tar.gz")
+if [ ! -d "${SUBMARINE_PROJECT_PATH}/submarine-dist/target" ]; then
+ mkdir "${SUBMARINE_PROJECT_PATH}/submarine-dist/target"
+fi
+submarine_dist_exists=$(find -L
"${SUBMARINE_PROJECT_PATH}/submarine-dist/target" -name
"submarine-dist-${submarine_v}*.tar.gz")
# Build source code if the package doesn't exist.
if [[ -z "${submarine_dist_exists}" ]]; then
+ # update tony code
+ if is_empty_dir "${SUBMARINE_PROJECT_PATH}/submodules/tony" ]; then
+ git submodule update --init --recursive
+ else
+ git submodule update --recursive
+ fi
+
+ cd "${SUBMARINE_PROJECT_PATH}"
mvn clean package -DskipTests
fi
-cp submarine-dist/target/submarine-dist-${submarine_v}*.tar.gz ${MINI_PATH}
-cp -r submarine-sdk/pysubmarine ${MINI_PATH}
-cp -r docs/database ${MINI_PATH}
-cd ${MINI_PATH}
+cp
${SUBMARINE_PROJECT_PATH}/submarine-dist/target/submarine-dist-${submarine_v}*.tar.gz
${MINI_PATH}
+cp -r ${SUBMARINE_PROJECT_PATH}/submarine-sdk/pysubmarine ${MINI_PATH}
+cp -r ${SUBMARINE_PROJECT_PATH}/docs/database ${MINI_PATH}
# build image
echo "Start building the mini-submarine docker image..."
+cd ${MINI_PATH}
docker build --build-arg HADOOP_VERSION=${hadoop_v} --build-arg
SPARK_VERSION=${spark_v} --build-arg SUBMARINE_VERSION=${submarine_v}
--build-arg IMAGE_NAME=${image_name} -t ${image_name} .
+
+# clean template file
+rm -rf ${MINI_PATH}/database
+rm -rf ${MINI_PATH}/pysubmarine
+rm -rf ${MINI_PATH}/submarine-dist-${submarine_v}*.tar.gz
diff --git a/dev-support/mini-submarine/conf/submarine-site.xml
b/dev-support/mini-submarine/conf/submarine-site.xml
index ac2d555..4c3257f 100644
--- a/dev-support/mini-submarine/conf/submarine-site.xml
+++ b/dev-support/mini-submarine/conf/submarine-site.xml
@@ -103,7 +103,7 @@
<property>
<name>workbench.web.war</name>
- <value>../workbench-web.war</value>
+ <value>../workbench-web-ng.war</value>
<description>Submarine workbench web war file path.</description>
</property>
diff --git a/submarine-workbench/workbench-web-ng/package.json
b/submarine-workbench/workbench-web-ng/package.json
index 8fc2e31..4e16987 100644
--- a/submarine-workbench/workbench-web-ng/package.json
+++ b/submarine-workbench/workbench-web-ng/package.json
@@ -28,7 +28,7 @@
"zone.js": "~0.9.1"
},
"devDependencies": {
- "@angular-devkit/build-angular": "~0.803.9",
+ "@angular-devkit/build-angular": "^0.803.21",
"@angular/cli": "~8.3.9",
"@angular/compiler-cli": "~8.2.9",
"@angular/language-service": "~8.2.9",
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
