This is an automated email from the ASF dual-hosted git repository.
liuxun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/submarine.git
The following commit(s) were added to refs/heads/master by this push:
new 3c8c173 SUBMARINE-809. Initialize model management SDK
3c8c173 is described below
commit 3c8c1738e9411f0ea3394410b56cc09d1492b378
Author: ByronHsu <[email protected]>
AuthorDate: Mon May 3 19:46:10 2021 +0800
SUBMARINE-809. Initialize model management SDK
### What is this PR for?
1. Document how to develop model management sdk with the help of
telepresence
2. Create a model management client in SDK
3. Write the corresponding test for model management
### What type of PR is it?
[Feature]
### Todos
- [ ] - Install submarine CI/CD by helm
- [ ] - Improve test case
### What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-809
### How should this be tested?
### Screenshots (if appropriate)
### Questions:
* Do the license files need updating? No
* Are there breaking changes for older versions? No
* Does this need new documentation? No
Author: ByronHsu <[email protected]>
Author: kobe860219 <[email protected]>
Signed-off-by: Liu Xun <[email protected]>
Closes #574 from ByronHsu/SUBMARINE-809 and squashes the following commits:
e0cce04 [ByronHsu] import mapping from abc
f129fab [ByronHsu] revert validation.py
c03e29c [ByronHsu] remove duplicate pip installation
44ebb36 [ByronHsu] temporarily remove py style check
d55b954 [ByronHsu] remove python 3.5 test
62f40dd [ByronHsu] resolve conflict
f7cda7f [ByronHsu] fix grammer
5dea7ca [ByronHsu] add dev doc
fb88908 [ByronHsu] test fine with telepresence
7604e49 [ByronHsu] initialize model management package in sdk
af43eef [kobe860219] SUBMARINE-773. Support MLflow model registry
348dd23 [kobe860219] Model registry done
97547d4 [kobe860219] minio server done
4da9366 [kobe860219] SUBMARINE-773. Support MLflow model registry
---
.github/workflows/python.yml | 5 +-
.gitignore | 2 +
submarine-sdk/pysubmarine/README.md | 2 +-
submarine-sdk/pysubmarine/setup.py | 3 ++
submarine-sdk/pysubmarine/submarine/__init__.py | 1 +
.../pysubmarine/submarine/models/__init__.py | 16 ++++++
.../pysubmarine/submarine/models/client.py | 51 ++++++++++++++++++
.../pysubmarine/submarine/models/constant.py | 21 ++++++++
.../pysubmarine/submarine/models/utils.py | 16 ++++++
submarine-sdk/pysubmarine/submarine/utils/env.py | 2 +-
submarine-sdk/pysubmarine/tests/models/pytorch.py | 29 ++++++++++
.../pysubmarine/tests/models/test_model.py | 62 ++++++++++++++++++++++
.../submarine-sdk/pysubmarine/development.md | 50 ++++++++++++++++-
13 files changed, 253 insertions(+), 7 deletions(-)
diff --git a/.github/workflows/python.yml b/.github/workflows/python.yml
index ddf90eb..283754b 100644
--- a/.github/workflows/python.yml
+++ b/.github/workflows/python.yml
@@ -9,8 +9,8 @@ jobs:
timeout-minutes: 120
strategy:
matrix:
- python-version: [3.5,3.6,3.7]
- tf-version: [1.14.0,1.15.0]
+ python-version: [3.6, 3.7]
+ tf-version: [1.14.0, 1.15.0]
fail-fast: false
env:
PYTHON: ${{ matrix.python }}
@@ -29,7 +29,6 @@ jobs:
pip install -r
./submarine-sdk/pysubmarine/github-actions/test-requirements.txt
pip install -r
./submarine-sdk/pysubmarine/github-actions/lint-requirements.txt
- name: Check python sdk code style
- if: matrix.python-version == '3.6'
run: |
./submarine-sdk/pysubmarine/github-actions/lint.sh
- name: Run unit test
diff --git a/.gitignore b/.gitignore
index 7629275..1bd36f5 100644
--- a/.gitignore
+++ b/.gitignore
@@ -101,3 +101,5 @@ submarine-cloud-v2/charts/*
# jupyter notebook checkpoints
.ipynb_checkpoints
+# redundant logging files
+*.log
\ No newline at end of file
diff --git a/submarine-sdk/pysubmarine/README.md
b/submarine-sdk/pysubmarine/README.md
index 173bc64..bc9dffb 100644
--- a/submarine-sdk/pysubmarine/README.md
+++ b/submarine-sdk/pysubmarine/README.md
@@ -49,4 +49,4 @@ Makes it easy to run distributed or non-distributed
TensorFlow, PyTorch experime
## Development
-See [Python
Development](https://github.com/apache/submarine/blob/master/website/docs/userDocs/submarine-sdk/pysubmarine/development.md)
in the documentation subproject.
+See [Python
Development](https://github.com/apache/submarine/tree/master/website/docs/userDocs/submarine-sdk/pysubmarine/development.md)
in the documentation subproject.
diff --git a/submarine-sdk/pysubmarine/setup.py
b/submarine-sdk/pysubmarine/setup.py
index d73b86a..ad16512 100644
--- a/submarine-sdk/pysubmarine/setup.py
+++ b/submarine-sdk/pysubmarine/setup.py
@@ -40,6 +40,9 @@ setup(
'python-dateutil >= 2.5.3',
'pyarrow==0.17.0',
'torch>=1.5.0',
+ 'torchvision>=0.6.0',
+ 'mlflow>=1.15.0',
+ 'boto3>=1.17.58 '
],
classifiers=[
'Intended Audience :: Developers',
diff --git a/submarine-sdk/pysubmarine/submarine/__init__.py
b/submarine-sdk/pysubmarine/submarine/__init__.py
index a23de5b..66536a1 100644
--- a/submarine-sdk/pysubmarine/submarine/__init__.py
+++ b/submarine-sdk/pysubmarine/submarine/__init__.py
@@ -16,6 +16,7 @@
import submarine.tracking as tracking
import submarine.tracking.fluent
from submarine.experiment.api.experiment_client import ExperimentClient
+from submarine.models.client import ModelsClient
log_param = submarine.tracking.fluent.log_param
log_metric = submarine.tracking.fluent.log_metric
diff --git a/submarine-sdk/pysubmarine/submarine/models/__init__.py
b/submarine-sdk/pysubmarine/submarine/models/__init__.py
new file mode 100644
index 0000000..273b024
--- /dev/null
+++ b/submarine-sdk/pysubmarine/submarine/models/__init__.py
@@ -0,0 +1,16 @@
+"""
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+"""
diff --git a/submarine-sdk/pysubmarine/submarine/models/client.py
b/submarine-sdk/pysubmarine/submarine/models/client.py
new file mode 100644
index 0000000..7214384
--- /dev/null
+++ b/submarine-sdk/pysubmarine/submarine/models/client.py
@@ -0,0 +1,51 @@
+"""
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+"""
+import os
+
+import mlflow
+from mlflow.tracking import MlflowClient
+
+from .constant import (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY,
+ MLFLOW_S3_ENDPOINT_URL, MLFLOW_TRACKING_URI)
+
+
+class ModelsClient():
+
+ def __init__(self):
+ """
+ Set up mlflow server connection, including: s3 endpoint, aws, tracking
server
+ """
+ os.environ["MLFLOW_S3_ENDPOINT_URL"] = MLFLOW_S3_ENDPOINT_URL
+ os.environ["AWS_ACCESS_KEY_ID"] = AWS_ACCESS_KEY_ID
+ os.environ["AWS_SECRET_ACCESS_KEY"] = AWS_SECRET_ACCESS_KEY
+ os.environ["MLFLOW_TRACKING_URI"] = MLFLOW_TRACKING_URI
+ self._client = MlflowClient()
+
+ def log_model(self, name, checkpoint):
+ mlflow.pytorch.log_model(registered_model_name=name,
+ pytorch_model=checkpoint,
+ artifact_path="pytorch-model")
+
+ def load_model(self, name, version):
+ model = mlflow.pyfunc.load_model(model_uri=f"models:/{name}/{version}")
+ return model
+
+ def update_model(self, name, new_name):
+ self._client.rename_registered_model(name=name, new_name=new_name)
+
+ def delete_model(self, name, version):
+ self._client.delete_model_version(name=name, version=version)
diff --git a/submarine-sdk/pysubmarine/submarine/models/constant.py
b/submarine-sdk/pysubmarine/submarine/models/constant.py
new file mode 100644
index 0000000..d0d7843
--- /dev/null
+++ b/submarine-sdk/pysubmarine/submarine/models/constant.py
@@ -0,0 +1,21 @@
+"""
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+"""
+
+MLFLOW_S3_ENDPOINT_URL = "http://10.96.0.4:9000";
+AWS_ACCESS_KEY_ID = "submarine_minio"
+AWS_SECRET_ACCESS_KEY = "submarine_minio"
+MLFLOW_TRACKING_URI = "http://10.96.0.3:5000";
diff --git a/submarine-sdk/pysubmarine/submarine/models/utils.py
b/submarine-sdk/pysubmarine/submarine/models/utils.py
new file mode 100644
index 0000000..273b024
--- /dev/null
+++ b/submarine-sdk/pysubmarine/submarine/models/utils.py
@@ -0,0 +1,16 @@
+"""
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+"""
diff --git a/submarine-sdk/pysubmarine/submarine/utils/env.py
b/submarine-sdk/pysubmarine/submarine/utils/env.py
index 56d133e..f339650 100644
--- a/submarine-sdk/pysubmarine/submarine/utils/env.py
+++ b/submarine-sdk/pysubmarine/submarine/utils/env.py
@@ -16,7 +16,7 @@
import copy
import json
import os
-from collections import Mapping
+from collections.abc import Mapping
def get_env(variable_name):
diff --git a/submarine-sdk/pysubmarine/tests/models/pytorch.py
b/submarine-sdk/pysubmarine/tests/models/pytorch.py
new file mode 100644
index 0000000..a543252
--- /dev/null
+++ b/submarine-sdk/pysubmarine/tests/models/pytorch.py
@@ -0,0 +1,29 @@
+"""
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+"""
+import numpy as np
+import torch
+
+
+class LinearNNModel(torch.nn.Module):
+
+ def __init__(self):
+ super(LinearNNModel, self).__init__()
+ self.linear = torch.nn.Linear(1, 1) # One in and one out
+
+ def forward(self, x):
+ y_pred = self.linear(x)
+ return y_pred
diff --git a/submarine-sdk/pysubmarine/tests/models/test_model.py
b/submarine-sdk/pysubmarine/tests/models/test_model.py
new file mode 100644
index 0000000..30fb254
--- /dev/null
+++ b/submarine-sdk/pysubmarine/tests/models/test_model.py
@@ -0,0 +1,62 @@
+"""
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+"""
+
+import numpy as np
+import pytest
+
+from pytorch import LinearNNModel
+from submarine import ModelsClient
+
+
+# Temporarily skip these tests after the following is solved:
+# TODO: Setup cluster by helm in CI/CD to enable mlflow server connection
+# TODO: Set an cooldown time between each test case
[email protected](reason="no way of currently testing this")
+class TestSubmarineModelsClient():
+
+ def setUp(self):
+ pass
+
+ def tearDown(self):
+ pass
+
+ def test_log_model(self):
+ client = ModelsClient()
+ model = LinearNNModel()
+ name = "simple-nn-model"
+ client.log_model(name, model)
+
+ def test_update_model(self):
+ client = ModelsClient()
+ name = "simple-nn-model"
+ new_name = "new-simple-nn-model"
+ client.update_model(name, new_name)
+
+ def test_load_model(self):
+ client = ModelsClient()
+ name = "simple-nn-model"
+ version = "1"
+ model = client.load_model(name, version)
+ x = np.float32([[1.0], [2.0]])
+ y = model.predict(x)
+ assert y.shape[0] == 2
+ assert y.shape[1] == 1
+
+ def test_delete_model(self):
+ client = ModelsClient()
+ name = "simple-nn-model"
+ client.delete_model(name, '1')
diff --git a/website/docs/userDocs/submarine-sdk/pysubmarine/development.md
b/website/docs/userDocs/submarine-sdk/pysubmarine/development.md
index c011f57..8975ce1 100644
--- a/website/docs/userDocs/submarine-sdk/pysubmarine/development.md
+++ b/website/docs/userDocs/submarine-sdk/pysubmarine/development.md
@@ -17,9 +17,12 @@ title: Python SDK Development
-->
This page provides general Python development guidelines and source build
instructions
+
### Prerequisites
+
This is required for developing & testing changes, we recommend installing
pysubmarine
in its own conda environment by running the following
+
```bash
conda create --name submarine-dev python=3.6
conda activate submarine-dev
@@ -31,73 +34,116 @@ pip install -r test-requirements.txt
# Installs pysubmarine from current checkout
pip install ./submarine-sdk/pysubmarine
```
+
### PySubmarine Docker
+
We also use docker to provide build environments for CI, development,
generate python sdk from swagger.
+
```bash
./run-pysubmarine-ci.sh
```
+
The script does the following things:
+
- Start an interactive bash session
- Mount submarine directory to /workspace and set it as home
- Switch user to be the same user that calls the `run-pysubmarine-ci.sh`
+
### Coding Style
+
- Use [yapf](https://github.com/google/yapf) to format Python code
- yapf style is configured in `.style.yapf` file
- To autoformat code
+
```bash
./submarine-sdk/pysubmarine/github-actions/auto-format.sh
```
+
- Verify linter pass before submitting a pull request by running:
+
```bash
./submarine-sdk/pysubmarine/github-actions/lint.sh
```
+
### Unit Testing
+
We are using [pytest](https://docs.pytest.org/en/latest/) to develop our unit
test suite.
After building the project (see below) you can run its unit tests like so:
+
```bash
cd submarine-sdk/pysubmarine
```
+
- Run unit test
+
```shell script
pytest --cov=submarine -vs -m "not e2e"
```
+
- Run integration test
+
```shell script
pytest --cov=submarine -vs -m "e2e"
```
-> Before run this command in local, you should make sure the submarine server
is running.
+
+> Before run this command in local, you should make sure the submarine server
is running.
### Generate python SDK from swagger
+
We use [open-api
generator](https://openapi-generator.tech/docs/installation/#jar)
to generate pysubmarine client API that used to communicate with submarine
server.
If change below files, please run `./dev-support/pysubmarine/gen-sdk.sh`
to generate latest version of SDK.
+
-
[Bootstrap.java](https://github.com/apache/submarine/blob/master/submarine-server/server-core/src/main/java/org/apache/submarine/server/Bootstrap.java)
-
[ExperimentRestApi.java](https://github.com/apache/submarine/blob/master/submarine-server/server-core/src/main/java/org/apache/submarine/server/rest/ExperimentRestApi.java)
+### Model Management Model Development
+
+For local development, we can access cluster's service easily thanks to
[telepresence](https://www.telepresence.io/).
+To elaborate, we can develop the sdk in local but can reach out to mlflow
server by proxy.
+
+1. Install telepresence follow [the
instruction](https://www.telepresence.io/reference/install).
+2. Start proxy pod
+
+```
+telepresence --new-deployment submarine-dev
+```
+
+3. You can develop as if in the cluster.
+
### Upload package to PyPi
+
For Apache Submarine committer and PMCs to do a new release.
+
1. Change the version from 0.x.x-SNAPSHOT to 0.x.x
-in
[setup.py](https://github.com/apache/submarine/blob/master/submarine-sdk/pysubmarine/setup.py)
+ in
[setup.py](https://github.com/apache/submarine/blob/master/submarine-sdk/pysubmarine/setup.py)
2. Install Python packages
+
```bash
cd submarine-sdk/pysubmarine
pip install -r github-actions/pypi-requirements.txt
```
+
3. Compiling Your Package
It will create `build`, `dist`, and `project.egg.info`
in your local directory
+
```bash
python setup.py bdist_wheel
```
+
4. Upload python package to TestPyPI for testing
+
```bash
python -m twine upload --repository testpypi dist/*
```
+
5. Upload python package to PyPi
+
```bash
python -m twine upload --repository-url https://upload.pypi.org/legacy/ dist/*
```
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
