This is an automated email from the ASF dual-hosted git repository.
pingsutw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/submarine.git
The following commit(s) were added to refs/heads/master by this push:
new 0c28117 SUBMARINE-682. Improve documentation generally
0c28117 is described below
commit 0c28117a5f3f374c161f56d1de8afe75d1ea0435
Author: ByronHsu <[email protected]>
AuthorDate: Fri Nov 27 10:39:22 2020 +0800
SUBMARINE-682. Improve documentation generally
### What is this PR for?
I do the following revision on documentation.
1. Update some commands from 0.4.0 to 0.5.0.
2. Replace "mvn clean install package" with "mvn clean package" because
"install" will be called automatically before "package". There is no need to
specify it.
3. Remove vue.js part
4. Clarify database documentation
5. Add QuickDevTips.md, recording some useful tips for developers.
### What type of PR is it?
[Documentation]
### What is the Jira issue?
[SUBMARINE-682. Improve documentation
generally](https://issues.apache.org/jira/projects/SUBMARINE/issues/SUBMARINE-682?filter=allopenissues)
Author: ByronHsu <[email protected]>
Author: Xun Liu <[email protected]>
Closes #457 from ByronHsu/SUBMARINE-682 and squashes the following commits:
48f78fe [Xun Liu] Merge branch 'master' into SUBMARINE-682
6239b5e [ByronHsu] improve doc
---
dev-support/cicd/create_release.sh | 5 +--
dev-support/mini-submarine/README.md | 10 ++---
docs/database/README.md | 10 ++++-
docs/development-guide-home.md | 4 +-
docs/development/BuildFromCode.md | 12 ++---
docs/development/QuickDevTips.md | 36 +++++++++++++++
docs/development/README.md | 2 +-
docs/userdocs/k8s/run-experiment-python-sdk.md | 2 +-
docs/userdocs/yarn/YARNRuntimeGuide.md | 12 ++---
docs/workbench/HowToRun.md | 51 +---------------------
submarine-cloud/submarine-operator.md | 4 +-
.../example/tensorflow/deepfm/README.md | 2 +-
.../pysubmarine/example/tensorflow/fm/README.md | 2 +-
13 files changed, 75 insertions(+), 77 deletions(-)
diff --git a/dev-support/cicd/create_release.sh
b/dev-support/cicd/create_release.sh
index f49b24e..fd83ef5 100755
--- a/dev-support/cicd/create_release.sh
+++ b/dev-support/cicd/create_release.sh
@@ -49,13 +49,12 @@ function compile_src_and_bin() {
cd ${WORKING_DIR}/submarine
echo "mvn versions:set -DnewVersion=${RELEASE_VERSION}"
mvn versions:set -DnewVersion="${RELEASE_VERSION}"
- echo "mvn clean install package -DskipTests -Psrc"
- mvn clean install package -DskipTests -Psrc
+ echo "mvn clean package -DskipTests -Psrc"
+ mvn clean package -DskipTests -Psrc
if [[ $? -ne 0 ]]; then
echo "Build failed. ${BUILD_FLAGS:-}"
exit 1
fi
-
}
function make_source_package() {
diff --git a/dev-support/mini-submarine/README.md
b/dev-support/mini-submarine/README.md
index 48af31b..063bbee 100644
--- a/dev-support/mini-submarine/README.md
+++ b/dev-support/mini-submarine/README.md
@@ -38,7 +38,7 @@ git clone https://github.com/apache/submarine.git
2.Build Submarine
```
cd ./submarine
-mvn clean install package -DskipTests
+mvn clean package -DskipTests
```
3.Build image of mini-submarine
@@ -197,7 +197,7 @@ You can follow the documentation instructions to update
your own modified and co
```
cd submarine-project-dir/
-mvn clean install package -DskipTests
+mvn clean package -DskipTests
```
### Copy submarine jar to mini-submarine container
@@ -233,7 +233,7 @@ When using mini-submarine, you can debug submarine client,
applicationMaster and
Run the following command to start mini-submarine.
```
-docker run -it -P -h submarine-dev --net=bridge --expose=8000 --privileged
local/mini-submarine:0.4.0 /bin/bash
+docker run -it -P -h submarine-dev --net=bridge --expose=8000 --privileged
local/mini-submarine:<REPLACE_VERSION> /bin/bash
```
Debug submarine client with the parameter "--debug"
@@ -262,7 +262,7 @@ Then port 32804 can be used for remote debug.
Run the following command to start mini-submarine.
```
-docker run -it -P -h submarine-dev --net=bridge --expose=8001 --privileged
local/mini-submarine:0.4.0 /bin/bash
+docker run -it -P -h submarine-dev --net=bridge --expose=8001 --privileged
local/mini-submarine:<REPLACE_VERSION> /bin/bash
```
Add the following configuration in the file
/usr/local/hadoop/etc/hadoop/tony.xml.
@@ -282,7 +282,7 @@ And the debug port mapping can be obtained using the way as
[Debug submarine cli
Run the following command to start mini-submarine.
```
-docker run -it -P -h submarine-dev --net=bridge --expose=8002 --privileged
local/mini-submarine:0.4.0 /bin/bash
+docker run -it -P -h submarine-dev --net=bridge --expose=8002 --privileged
local/mini-submarine:<REPLACE_VERSION> /bin/bash
```
Add the following configuration in the file
/usr/local/hadoop/etc/hadoop/tony.xml.
diff --git a/docs/database/README.md b/docs/database/README.md
index 3fddaae..bf0e4ad 100644
--- a/docs/database/README.md
+++ b/docs/database/README.md
@@ -32,7 +32,7 @@ Must:
By using the official docker image of submarine database, only one docker
command is required to run submarine database
```bash
-docker run -it -p 3306:3306 -d --name submarine-database -e
MYSQL_ROOT_PASSWORD=password apache/submarine:database-0.4.0
+docker run -it -p 3306:3306 -d --name submarine-database -e
MYSQL_ROOT_PASSWORD=password apache/submarine:database-<REPLACE_VERSION>
```
## Initialize submarine database
It will create users and tables that submarine requires
@@ -83,7 +83,15 @@ If you need to store Chinese character data in mysql, you
need to execute the fo
### Create Submarine Database
+
#### Create development database
+
+Run [mysql docker container](https://hub.docker.com/_/mysql)
+
+```
+docker run --name some-mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql:tag
+```
+
Copy the files, submarine.sql, submarine-data.sql and metastore.sql to the
mysql docker.
```
diff --git a/docs/development-guide-home.md b/docs/development-guide-home.md
index 4972319..6035df7 100644
--- a/docs/development-guide-home.md
+++ b/docs/development-guide-home.md
@@ -21,8 +21,10 @@
[Submarine Workbench Guide](./workbench/README.md)
-## Development
+## Development
[Submarine Project Development Guide](./development/README.md)
[Submarine Project Database Guide](./database/README.md)
+
+[Submarine Quick Development Tips](./development/QuickDevTips.md)
\ No newline at end of file
diff --git a/docs/development/BuildFromCode.md
b/docs/development/BuildFromCode.md
index d62dd56..5ede0c9 100644
--- a/docs/development/BuildFromCode.md
+++ b/docs/development/BuildFromCode.md
@@ -57,37 +57,37 @@ mvn clean org.apache.rat:apache-rat-plugin:check
+ Create binary distribution with default hadoop version
```
-mvn clean install package -DskipTests
+mvn clean package -DskipTests
```
+ Create binary distribution with hadoop-2.9.x version
```
-mvn clean install package -DskipTests -Phadoop-2.9
+mvn clean package -DskipTests -Phadoop-2.9
```
+ Create binary distribution with hadoop-2.10.x version
```
-mvn clean install package -DskipTests -Phadoop-2.10
+mvn clean package -DskipTests -Phadoop-2.10
```
+ Create binary distribution with hadoop-3.1.x version
```
-mvn clean install package -DskipTests -Phadoop-3.1
+mvn clean package -DskipTests -Phadoop-3.1
```
+ Create binary distribution with hadoop-3.2.x version
```
-mvn clean install package -DskipTests -Phadoop-3.2
+mvn clean package -DskipTests -Phadoop-3.2
```
+ Create source code distribution
```
-mvn clean install package -DskipTests -Psrc
+mvn clean package -DskipTests -Psrc
```
## TonY code modification
diff --git a/docs/development/QuickDevTips.md b/docs/development/QuickDevTips.md
new file mode 100644
index 0000000..e614a46
--- /dev/null
+++ b/docs/development/QuickDevTips.md
@@ -0,0 +1,36 @@
+## 1. Introduction
+
+This document describes some useful tips that can accelerate your development
efficiency.
+
+### Run server outside of k8s
+If you are working on submarine-server, you do not need to bundle
submarine-server into docker-image, and restart the helm-chart every time.
+
+You can run each module individually and only need to re-package the
submarine-server, getting rid of other unnecessary process.
+
+1. Run db docker
+
+```
+docker run -it -p 3306:3306 -d --name submarine-database -e
MYSQL_ROOT_PASSWORD=password apache/submarine:database-0.5.0-SNAPSHOT
+```
+
+2. Run k8s
+
+```
+minikube start # or other alternatives, such as kind
+kubectl apply -f ./dev-support/k8s/tfjob/crd.yaml
+kubectl kustomize ./dev-support/k8s/tfjob/operator | kubectl apply -f -
+kubectl apply -f ./dev-support/k8s/pytorchjob/
+export KUBECONFIG=/home/<user_name>/.kube/config # (in ~/.bashrc)
+```
+3. Package server
+
+```
+mvn clean package -DskipTests
+```
+
+4. Start server
+
+cd
submarine-dist/target/submarine-dist-0.5.0-SNAPSHOT-hadoop-2.9/submarine-dist-0.5.0-SNAPSHOT-hadoop-2.9/
+./bin/submarine-daemon.sh start getMysqlJar
+```
+```
\ No newline at end of file
diff --git a/docs/development/README.md b/docs/development/README.md
index bb74be5..f917d77 100644
--- a/docs/development/README.md
+++ b/docs/development/README.md
@@ -104,7 +104,7 @@ By executing the above command, `workbench-web` will
publish the web page to the
+ **Lints and fixes files**
-When you write the `angular, js` file in `workbench-web` through IDEA, because
IDEA can't format these files well, you need to execute the following command
to format the angular and js files to avoid some warnings during the yarn build.
+When you write the `Angular, js` file in `workbench-web` through IDEA, because
IDEA can't format these files well, you need to execute the following command
to format the angular and js files to avoid some warnings during the yarn build.
```
yarn run lint
diff --git a/docs/userdocs/k8s/run-experiment-python-sdk.md
b/docs/userdocs/k8s/run-experiment-python-sdk.md
index e7c8c9b..c28fdc6 100644
--- a/docs/userdocs/k8s/run-experiment-python-sdk.md
+++ b/docs/userdocs/k8s/run-experiment-python-sdk.md
@@ -43,7 +43,7 @@ python3 virtualenv-16.0.0/virtualenv.py venv
Starting from 0.4.0, Submarine provides Python SDK. Please change it to a
proper version needed.
```bash
-pip install submarine-sdk==0.4.0
+pip install submarine-sdk==<REPLACE_VERSION>
```
**Install SDK from source code**
diff --git a/docs/userdocs/yarn/YARNRuntimeGuide.md
b/docs/userdocs/yarn/YARNRuntimeGuide.md
index 2b00988..ac93095 100644
--- a/docs/userdocs/yarn/YARNRuntimeGuide.md
+++ b/docs/userdocs/yarn/YARNRuntimeGuide.md
@@ -60,7 +60,7 @@ Get mnist_distributed.py from
https://github.com/linkedin/TonY/tree/master/tony-
```
-SUBMARINE_VERSION=0.4.0
+SUBMARINE_VERSION=<REPLACE_VERSION>
SUBMARINE_HADOOP_VERSION=3.1
CLASSPATH=$(hadoop classpath
--glob):path-to/submarine-all-${SUBMARINE_VERSION}-hadoop-${SUBMARINE_HADOOP_VERSION}.jar
\
java org.apache.submarine.client.cli.Cli job run --name tf-job-001 \
@@ -94,7 +94,7 @@ You should then be able to see links and status of the jobs
from command line:
### With Docker
```
-SUBMARINE_VERSION=0.4.0
+SUBMARINE_VERSION=<REPLACE_VERSION>
SUBMARINE_HADOOP_VERSION=3.1
CLASSPATH=$(hadoop classpath
--glob):path-to/submarine-all-${SUBMARINE_VERSION}-hadoop-${SUBMARINE_HADOOP_VERSION}.jar
\
java org.apache.submarine.client.cli.Cli job run --name tf-job-001 \
@@ -155,7 +155,7 @@ Get mnist_distributed.py from
https://github.com/linkedin/TonY/tree/master/tony-
```
-SUBMARINE_VERSION=0.4.0
+SUBMARINE_VERSION=<REPLACE_VERSION>
SUBMARINE_HADOOP_VERSION=3.1
CLASSPATH=$(hadoop classpath
--glob):path-to/submarine-all-${SUBMARINE_VERSION}-hadoop-${SUBMARINE_HADOOP_VERSION}.jar
\
java org.apache.submarine.client.cli.Cli job run --name PyTorch-job-001 \
@@ -188,7 +188,7 @@ You should then be able to see links and status of the jobs
from command line:
### With Docker
```
-SUBMARINE_VERSION=0.4.0
+SUBMARINE_VERSION=<REPLACE_VERSION>
SUBMARINE_HADOOP_VERSION=3.1
CLASSPATH=$(hadoop classpath
--glob):path-to/submarine-all-${SUBMARINE_VERSION}-hadoop-${SUBMARINE_HADOOP_VERSION}.jar
\
java org.apache.submarine.client.cli.Cli job run --name PyTorch-job-001 \
@@ -240,7 +240,7 @@ Get image_classification.py from this
[link](https://github.com/apache/submarine
```
-SUBMARINE_VERSION=0.4.0
+SUBMARINE_VERSION=<REPLACE_VERSION>
SUBMARINE_HADOOP_VERSION=3.1
CLASSPATH=$(hadoop classpath
--glob):path-to/submarine-all-${SUBMARINE_VERSION}-hadoop-${SUBMARINE_HADOOP_VERSION}.jar
\
java org.apache.submarine.client.cli.Cli job run --name MXNet-job-001 \
@@ -282,7 +282,7 @@ You should then be able to see links and status of the jobs
from command line:
### With Docker
You could refer to this [sample
Dockerfile](project/github/submarine/docs/userdocs/yarn/docker/mxnet/cifar10/Dockerfile.cifar10.mx_1.5.1)
for building your own Docker image.
```
-SUBMARINE_VERSION=0.4.0
+SUBMARINE_VERSION=<REPLACE_VERSION>
SUBMARINE_HADOOP_VERSION=3.1
CLASSPATH=$(hadoop classpath
--glob):path-to/submarine-all-${SUBMARINE_VERSION}-hadoop-${SUBMARINE_HADOOP_VERSION}.jar
\
java org.apache.submarine.client.cli.Cli job run --name MXNet-job-001 \
diff --git a/docs/workbench/HowToRun.md b/docs/workbench/HowToRun.md
index 4d05741..4ef2cb7 100644
--- a/docs/workbench/HowToRun.md
+++ b/docs/workbench/HowToRun.md
@@ -28,57 +28,10 @@ By using the official images of Submarine, only a few
docker commands are requir
### Launch the Submarine Workbench(Angular)
* It should be noted that since Submarine Workbench depends on the Submarine
database, so you need to run the docker container of the Submarine database
first.
```
-docker run -it -p 3306:3306 -d --name submarine-database -e
MYSQL_ROOT_PASSWORD=password apache/submarine:database-0.4.0
-docker run -it -p 8080:8080 -d --link=submarine-database:submarine-database
--name submarine-server apache/submarine:server-0.4.0
+docker run -it -p 3306:3306 -d --name submarine-database -e
MYSQL_ROOT_PASSWORD=password apache/submarine:database-<REPLACE_VERSION>
+docker run -it -p 8080:8080 -d --link=submarine-database:submarine-database
--name submarine-server apache/submarine:server-<REPLACE_VERSION>
```
* The login page of Submarine Workbench will be shown in
```http://127.0.0.1:8080```.
-
-### Switch from version Angular to version Vue
-* Step1: Launch submarine-database and submarine-server containers
-```
-docker run -it -p 3306:3306 -d --name submarine-database -e
MYSQL_ROOT_PASSWORD=password apache/submarine:database-0.4.0
-docker run -it -p 8080:8080 -d --link=submarine-database:submarine-database
--name submarine-server apache/submarine:server-0.4.0
-```
-* Step2: Compile Submarine in your host (not in the container)
-```
-cd ./submarine
-mvn clean install package -DskipTests
-```
-* Step3: Copy workbench-web.war into the submarine-server container
-```
-cd submarine-workbench/workbench-web/target
-docker cp workbench-web.war
submarine-server:/opt/submarine-dist-0.4.0-hadoop-2.9
-```
-* Step4: Enter the submarine-server container
-```
-docker exec -it submarine-server bash
-```
-* Step5: Modify the value of the configuration **workbench.web.war** in
conf/submarine-site.xml from "../submarine-workbench-web-ng.war" to
"../submarine-workbench-web.war".
-
-* Step6: Restart the Submarine Server
-```
-./bin/submarine-daemon.sh restart
-```
-* Step7: Launch the submarine-server container
-```
-docker start submarine-server
-```
-* Step8: Open a new **incognito window(not a tab)** and check
```http://127.0.0.1:8080```
-### Switch from version Vue to version Angular
-* Step1: Enter the submarine-server container
-```
-docker exec -it submarine-server bash
-```
-* Step2: Modify the value of the configuration **workbench.web.war** in
conf/submarine-site.xml from "../workbench-web.war" to
"../submarine-workbench-web-ng.war".
-* Step3: Restart the Submarine Server
-```
-./bin/submarine-daemon.sh restart
-```
-* Step4: Launch the submarine-server container
-```
-docker start submarine-server
-```
-* Step5: Open a **new incognito window(not a tab)** and check
```http://127.0.0.1:8080```
### Check the data in the submarine-database
* Step1: Enter the submarine-database container
```
diff --git a/submarine-cloud/submarine-operator.md
b/submarine-cloud/submarine-operator.md
index 63815b5..df0971f 100644
--- a/submarine-cloud/submarine-operator.md
+++ b/submarine-cloud/submarine-operator.md
@@ -165,6 +165,6 @@ kubectl describe std test1
```
kind load docker-image --name=submarine busybox:1.28.4
-kind load docker-image --name=submarine apache/submarine:server-0.4.0
-kind load docker-image --name=submarine apache/submarine:database-0.4.0
+kind load docker-image --name=submarine
apache/submarine:server-<REPLACE_VERSION>
+kind load docker-image --name=submarine
apache/submarine:database-<REPLACE_VERSION>
```
diff --git a/submarine-sdk/pysubmarine/example/tensorflow/deepfm/README.md
b/submarine-sdk/pysubmarine/example/tensorflow/deepfm/README.md
index a17e201..70da78f 100644
--- a/submarine-sdk/pysubmarine/example/tensorflow/deepfm/README.md
+++ b/submarine-sdk/pysubmarine/example/tensorflow/deepfm/README.md
@@ -38,7 +38,7 @@ python run_deepfm.py -conf=deepfm.json -task_type evaluate
3. Submit Job
```
-SUBMARINE_VERSION=0.4.0
+SUBMARINE_VERSION=<REPLACE_VERSION>
SUBMARINE_HADOOP_VERSION=2.9
java -cp $(${HADOOP_COMMON_HOME}/bin/hadoop classpath
--glob):submarine-all-${SUBMARINE_VERSION}-hadoop-${SUBMARINE_HADOOP_VERSION}.jar:${HADOOP_CONF_PATH}
\
diff --git a/submarine-sdk/pysubmarine/example/tensorflow/fm/README.md
b/submarine-sdk/pysubmarine/example/tensorflow/fm/README.md
index 0aca2bf..1286f67 100644
--- a/submarine-sdk/pysubmarine/example/tensorflow/fm/README.md
+++ b/submarine-sdk/pysubmarine/example/tensorflow/fm/README.md
@@ -30,7 +30,7 @@ python run_fm.py -conf=fm.json -task_type evaluate
3. Submit Job
```
-SUBMARINE_VERSION=0.4.0
+SUBMARINE_VERSION=<REPLACE_VERSION>
SUBMARINE_HADOOP_VERSION=2.9
java -cp $(${HADOOP_COMMON_HOME}/bin/hadoop classpath
--glob):submarine-all-${SUBMARINE_VERSION}-hadoop-${SUBMARINE_HADOOP_VERSION}.jar:${HADOOP_CONF_PATH}
\
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
