This is an automated email from the ASF dual-hosted git repository.
liuxun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/submarine.git
The following commit(s) were added to refs/heads/master by this push:
new 126e518 SUBMARINE-670. Improve Submarine User Docs (K8s)
126e518 is described below
commit 126e51811f78503d260758c7e5d2a9b1dd2653e8
Author: Wangda Tan <[email protected]>
AuthorDate: Mon Nov 2 16:31:35 2020 -0800
SUBMARINE-670. Improve Submarine User Docs (K8s)
### What is this PR for?
A few sentences describing the overall goals of the pull request's commits.
First time? Check out the contributing guide -
https://submarine.apache.org/contribution/contributions.html
### What type of PR is it?
Documentation
### Todos
* [ ] - Task
### What is the Jira issue?
* Open an issue on Jira https://issues.apache.org/jira/browse/SUBMARINE-670
Author: Wangda Tan <[email protected]>
Closes #444 from wangdatan/SUBMARINE-670 and squashes the following commits:
ccdad88 [Wangda Tan] more changes
139c820 [Wangda Tan] SUBMARINE-670
---
docs/development/BuildFromCode.md | 27 ++++++++++-
docs/user-guide-home.md | 4 +-
docs/userdocs/k8s/README.md | 55 ++++-----------------
docs/userdocs/k8s/helm.md | 23 ---------
docs/userdocs/k8s/notebook.md | 4 ++
.../{README.md => run-experiment-python-sdk.md} | 44 +----------------
...template.md => run-experiment-template-rest.md} | 2 +-
...xperiment.md => run-pytorch-experiment-rest.md} | 2 +-
...riment.md => run-tensorflow-experiment-rest.md} | 2 +-
docs/userdocs/k8s/run-tensorflow-experiment-ui.md | 56 ++++++++++++++++++++++
10 files changed, 100 insertions(+), 119 deletions(-)
diff --git a/docs/development/BuildFromCode.md
b/docs/development/BuildFromCode.md
index 783c5f5..d62dd56 100644
--- a/docs/development/BuildFromCode.md
+++ b/docs/development/BuildFromCode.md
@@ -23,7 +23,30 @@
## Quick Start
-### Building the code
+### Build Your Custom Submarine Docker Images
+
+Submarine provides default Docker image in the release artifacts, sometimes
you would like to do some modifications on the images. You can rebuild Docker
image after you make changes.
+
+> Note that you need to make sure the images built above can be accessed in k8s
+> Usually this needs a rename and push to a proper Docker registry.
+
+```bash
+mvn clean package -DskipTests
+```
+
+Build submarine server image:
+
+```bash
+./dev-support/docker-images/submarine/build.sh
+```
+
+Build submarine database image:
+
+```bash
+./dev-support/docker-images/database/build.sh
+```
+
+### Building source code / binary distribution
+ Checking releases for licenses
@@ -70,4 +93,4 @@ mvn clean install package -DskipTests -Psrc
## TonY code modification
If it is needed to make modifications to TonY project, please make a PR
-to [Tony repository](https://github.com/linkedin/TonY).
+to [Tony repository](https://github.com/linkedin/TonY).
\ No newline at end of file
diff --git a/docs/user-guide-home.md b/docs/user-guide-home.md
index 9e272eb..fc575d8 100644
--- a/docs/user-guide-home.md
+++ b/docs/user-guide-home.md
@@ -18,9 +18,7 @@ This is index of user-document of Submarine.
## Build From Code
-[Build From Code Guide](./development/BuildFromCode.md)
-
-FIXME: Where's build guide for K8s? How can we make it clear?
+Follow the guidee [Build From Code Guide](./development/BuildFromCode.md) to
build source code / binary distribution of Submarine, and rebuild Docker images.
## Use Submarine on YARN
diff --git a/docs/userdocs/k8s/README.md b/docs/userdocs/k8s/README.md
index b5d4ab9..63f5326 100644
--- a/docs/userdocs/k8s/README.md
+++ b/docs/userdocs/k8s/README.md
@@ -24,8 +24,6 @@ Submarine can run on K8s >= 1.14, supports features like GPU
isolation.
We have validated Submarine on the following versions:
-[FIXME]: is it accurate?
-
| K8s Version | Support? |
| ------------- |:-------------:|
| 1.13.x (or earlier) | X |
@@ -45,60 +43,25 @@ From our experiences, Docker Desktop is an easier choice.
### Install Submarine Use Helm Charts
After you have an up-and-running K8s, you can follow [Submarine Helm Charts
Guide](helm.md) to deploy Submarine services on K8s cluster in minutes.
-## Model Training (Experiment) on K8s
-
-### Model Training With Submarine Python SDK
-
-Submarine Python SDK can runs on any machine and it will talk to Submarine
Server via REST API. So you can install Submarine Python SDK on your laptop, a
gateway machine, your favorite IDE (like PyCharm/Jupyter, etc.).
-
-#### Prepare Python Environment to run Submarine SDK
-
-First of all
-
-Submarine SDK requires Python3.7+.
-It's better to use a new Python environment created by `Anoconda` or Python
`virtualenv` to try this to avoid trouble to existing Python environment.
-A sample Python virtual env can be setup like this:
-```bash
-wget
https://files.pythonhosted.org/packages/33/bc/fa0b5347139cd9564f0d44ebd2b147ac97c36b2403943dbee8a25fd74012/virtualenv-16.0.0.tar.gz
-tar xf virtualenv-16.0.0.tar.gz
+## Using Jupyter notebooks in Submarine
-# Make sure to install using Python 3
-python3 virtualenv-16.0.0/virtualenv.py venv
-. venv/bin/activate
-```
+Please refer to [Notebook guide](notebook.md) if you want to use jupyter
notebook in Submarine.
-#### Install Submarine SDK
-
-**Install SDK from pypi.org (recommended)**
-
-Starting from 0.4.0, Submarine provides Python SDK. Please change it to a
proper version needed.
-
-```bash
-pip install submarine-sdk==0.4.0
-```
-
-**Install SDK from source code**
-
-Please first clone code from github or go to
`http://submarine.apache.org/download.html` to download released source code.
+## Model Training (Experiment) on K8s
-```bash
-git clone https://github.com/apache/submarine.git
-git checkout <correct release tag/branch>
-cd submarine/submarine-sdk/pysubmarine
-pip install .
-```
+### Model Training with Submarine UI
-#### Run with Submarine Python SDK
+You can now submit Submarine model training on the UI, please refer to [Submit
Experiment from Submarine UI](./run-tensorflow-experiment-ui.md)
-Assuming you've installed submarine on K8s and forward the service to
localhost, now you can open a Python shell, Jupyter notebook or any tools with
Submarine SDK installed.
+### Model Training With Submarine Python SDK
-Follow [SDK experiment
example](../../../submarine-sdk/pysubmarine/example/submarine_experiment_sdk.ipynb)
to try the SDK.
+You can also follow the guidance to run Submarine model training using Python
SDK, please refer to [Submit Experiment using Python
SDK](./run-experiment-python-sdk.md)
### Model Training With Submarine REST API
Alternatively, we support use REST API to submit, list, delete experiments
(model training)
-- [Run model training using Tensorflow](run-tensorflow-experiment.md)
-- [Run model training using PyTorch](run-pytorch-experiment.md)
+- [Run model training using Tensorflow](run-tensorflow-experiment-rest.md)
+- [Run model training using PyTorch](run-pytorch-experiment-rest.md)
- [Experiment API Reference](api/experiment.md)
diff --git a/docs/userdocs/k8s/helm.md b/docs/userdocs/k8s/helm.md
index acf795b..0ad7405 100644
--- a/docs/userdocs/k8s/helm.md
+++ b/docs/userdocs/k8s/helm.md
@@ -115,26 +115,3 @@ curl http://127.0.0.1:8080/api/v1/experiment/ping
```bash
helm delete submarine
```
-
-### Create Your Custom Submarine Images (Optional)
-Sometimes we'd like to do some modifications on the images.
-After that, you need to rebuild submarine images:
-> Note that you need to make sure the images built above can be accessed in k8s
-> Usually this needs a rename and push to a proper Docker registry.
-
-```bash
-mvn clean package -DskipTests
-```
-
-Build submarine server image:
-```bash
-./dev-support/docker-images/submarine/build.sh
-```
-
-Build submarine database image:
-```bash
-./dev-support/docker-images/database/build.sh
-```
-
-### Using Jupyter notebooks in Submarine
-Please refer to [Notebook guide](notebook.md) if you want to use jupyter
notebook in Submarine.
diff --git a/docs/userdocs/k8s/notebook.md b/docs/userdocs/k8s/notebook.md
index 1a0ea92..fa9a614 100644
--- a/docs/userdocs/k8s/notebook.md
+++ b/docs/userdocs/k8s/notebook.md
@@ -91,3 +91,7 @@ experiment_spec = ExperimentSpec(meta=experiment_meta,
experiment =
submarine_client.create_experiment(experiment_spec=experiment_spec)
```
+
+You can create a new notebook, paste the above code and run it. Or, you can
find the notebook `submarine_experiment_sdk.ipynb` inside the launched notebook
session. You can open it, try it out.
+
+After experiment submitted to Submarine server, you can find the experiment
jobs on the UI.
\ No newline at end of file
diff --git a/docs/userdocs/k8s/README.md
b/docs/userdocs/k8s/run-experiment-python-sdk.md
similarity index 61%
copy from docs/userdocs/k8s/README.md
copy to docs/userdocs/k8s/run-experiment-python-sdk.md
index b5d4ab9..e7c8c9b 100644
--- a/docs/userdocs/k8s/README.md
+++ b/docs/userdocs/k8s/run-experiment-python-sdk.md
@@ -17,38 +17,6 @@ specific language governing permissions and limitations
under the License.
-->
-# Submarine on K8s
-Submarine for K8s supports distributed TensorFlow and PyTorch.
-
-Submarine can run on K8s >= 1.14, supports features like GPU isolation.
-
-We have validated Submarine on the following versions:
-
-[FIXME]: is it accurate?
-
-| K8s Version | Support? |
-| ------------- |:-------------:|
-| 1.13.x (or earlier) | X |
-| 1.14.x | √ |
-| 1.15.x | √ |
-| 1.16.x | √ |
-| 1.17.x | To be verified |
-| 1.17.x | To be verified |
-
-## Install Submarine
-
-### Setup Kubernetes
-Submarine can be deployed on any K8s environment if version matches. If you
don't have a running K8s, you can set up a K8s using [Docker
Desktop](https://www.docker.com/products/docker-desktop),
[MiniKube](https://kubernetes.io/docs/tasks/tools/install-minikube/), or [kind,
Kubernetes-in-Docker](https://kind.sigs.k8s.io/).
-
-From our experiences, Docker Desktop is an easier choice.
-
-### Install Submarine Use Helm Charts
-After you have an up-and-running K8s, you can follow [Submarine Helm Charts
Guide](helm.md) to deploy Submarine services on K8s cluster in minutes.
-
-## Model Training (Experiment) on K8s
-
-### Model Training With Submarine Python SDK
-
Submarine Python SDK can runs on any machine and it will talk to Submarine
Server via REST API. So you can install Submarine Python SDK on your laptop, a
gateway machine, your favorite IDE (like PyCharm/Jupyter, etc.).
#### Prepare Python Environment to run Submarine SDK
@@ -58,6 +26,7 @@ First of all
Submarine SDK requires Python3.7+.
It's better to use a new Python environment created by `Anoconda` or Python
`virtualenv` to try this to avoid trouble to existing Python environment.
A sample Python virtual env can be setup like this:
+
```bash
wget
https://files.pythonhosted.org/packages/33/bc/fa0b5347139cd9564f0d44ebd2b147ac97c36b2403943dbee8a25fd74012/virtualenv-16.0.0.tar.gz
tar xf virtualenv-16.0.0.tar.gz
@@ -92,13 +61,4 @@ pip install .
Assuming you've installed submarine on K8s and forward the service to
localhost, now you can open a Python shell, Jupyter notebook or any tools with
Submarine SDK installed.
-Follow [SDK experiment
example](../../../submarine-sdk/pysubmarine/example/submarine_experiment_sdk.ipynb)
to try the SDK.
-
-### Model Training With Submarine REST API
-
-Alternatively, we support use REST API to submit, list, delete experiments
(model training)
-
-- [Run model training using Tensorflow](run-tensorflow-experiment.md)
-- [Run model training using PyTorch](run-pytorch-experiment.md)
-- [Experiment API Reference](api/experiment.md)
-
+Follow [SDK experiment
example](../../../submarine-sdk/pysubmarine/example/submarine_experiment_sdk.ipynb)
to try the SDK.
\ No newline at end of file
diff --git a/docs/userdocs/k8s/run-experiment-template.md
b/docs/userdocs/k8s/run-experiment-template-rest.md
similarity index 97%
rename from docs/userdocs/k8s/run-experiment-template.md
rename to docs/userdocs/k8s/run-experiment-template-rest.md
index dcae977..dd62f9b 100644
--- a/docs/userdocs/k8s/run-experiment-template.md
+++ b/docs/userdocs/k8s/run-experiment-template-rest.md
@@ -17,7 +17,7 @@ specific language governing permissions and limitations
under the License.
-->
-# Experiment Template API Reference
+# Experiment Template REST API Reference
## Experiment Template Spec
The experiment is represented in [JSON](https://www.json.org) or
[YAML](https://yaml.org) format.
diff --git a/docs/userdocs/k8s/run-pytorch-experiment.md
b/docs/userdocs/k8s/run-pytorch-experiment-rest.md
similarity index 98%
rename from docs/userdocs/k8s/run-pytorch-experiment.md
rename to docs/userdocs/k8s/run-pytorch-experiment-rest.md
index a1b6f9c..3fcac79 100644
--- a/docs/userdocs/k8s/run-pytorch-experiment.md
+++ b/docs/userdocs/k8s/run-pytorch-experiment-rest.md
@@ -17,7 +17,7 @@ specific language governing permissions and limitations
under the License.
-->
-# Run PyTorch Experiment Guide
+# Run PyTorch Experiment Guide (REST)
## Experiment Spec
The experiment is represented in [JSON](https://www.json.org) or
[YAML](https://yaml.org) format.
diff --git a/docs/userdocs/k8s/run-tensorflow-experiment.md
b/docs/userdocs/k8s/run-tensorflow-experiment-rest.md
similarity index 98%
rename from docs/userdocs/k8s/run-tensorflow-experiment.md
rename to docs/userdocs/k8s/run-tensorflow-experiment-rest.md
index cba89f6..5145dc0 100644
--- a/docs/userdocs/k8s/run-tensorflow-experiment.md
+++ b/docs/userdocs/k8s/run-tensorflow-experiment-rest.md
@@ -17,7 +17,7 @@ specific language governing permissions and limitations
under the License.
-->
-# Run TensorFlow Experiment Guide
+# Run TensorFlow Experiment Guide (REST)
## Experiment Spec
The experiment is represented in [JSON](https://www.json.org) or
[YAML](https://yaml.org) format.
diff --git a/docs/userdocs/k8s/run-tensorflow-experiment-ui.md
b/docs/userdocs/k8s/run-tensorflow-experiment-ui.md
new file mode 100644
index 0000000..f68a878
--- /dev/null
+++ b/docs/userdocs/k8s/run-tensorflow-experiment-ui.md
@@ -0,0 +1,56 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Run Tensorflow Experiment on Submarine UI
+
+## Steps to run Tensorflow Experiment
+
+- Click `+ New Experiment` on the "Experiment" page.
+
+- Click `Define your experiment`
+
+- Put a name to experiment, like "minst-example"
+
+- Command: `python /var/tf_mnist/mnist_with_summaries.py --log_dir=/train/log
--learning_rate=0.01 --batch_size=150`
+
+- Image you can put; `apache/submarine:tf-mnist-with-summaries-1.0`
+
+- Click `Next Step`
+
+- Choose `Distributed Tensorflow`
+
+- Click `Add new spec` twice to add two new specs (roles)
+
+- One is Worker, another one is PS, leave rest of the parameters unchanged
+
+- Click next step, you can review your parameters before submitting the job:
+
+
+
+ It should look like below:
+
+ | Name | mnist-example-111
| | |
+ | --------------------- |
------------------------------------------------------------ | ---- | ---- |
+ | Command | python /var/tf_mnist/mnist_with_summaries.py
--log_dir=/train/log --learning_rate=0.01 --batch_size=150 | | |
+ | Image | apache/submarine:tf-mnist-with-summaries-1.0
| | |
+ | Environment Variables |
| | |
+ | Ps | cpu=1,nvidia.com/gpu=0,memory=1024M
| | |
+ | Worker | cpu=1,nvidia.com/gpu=0,memory=1024M
| | |
+
+- Click `Submit` it will be submitted, you can see the new example running in
the `Experiment` list, you can get logs, etc. directly from the UI
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
