This is an automated email from the ASF dual-hosted git repository.
cdmikechen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/submarine.git
The following commit(s) were added to refs/heads/master by this push:
new 6103f2a9 SUBMARINE-1363. Git-Sync container can not pull specified
branch or main codes
6103f2a9 is described below
commit 6103f2a9e60f0cd165e2ec43b1e5f92180fabf7e
Author: cdmikechen <cdmikec...@apache.org>
AuthorDate: Fri Jun 23 08:01:46 2023 +0800
SUBMARINE-1363. Git-Sync container can not pull specified branch or main
codes
### What is this PR for?
Make submarine support for pulling git with branch, username and password.
In the meantime, some python CI issues have been discovered while changing
codes. This has now been fixed as well.
### What type of PR is it?
Bug Fix
### Todos
* [x] - Submarine Server
* [x] - e2e test
* [x] - Submarine Workbench
* [x] - Submarine API Documents
* [x] - Submarine SDK
* [x] - Fix some CI error in python workflow
### What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-1363
### How should this be tested?
cicd
### Screenshots (if appropriate)
<img width="1029" alt="image"
src="https://github.com/apache/submarine/assets/12069428/a2f5bc33-293b-4bf4-b242-06890a5933ec";>
### Questions:
* Do the license files need updating? No
* Are there breaking changes for older versions? Yes
* Does this need new documentation? Yes
Author: cdmikechen <cdmikec...@apache.org>
Signed-off-by: cdmikechen <cdmikec...@apache.org>
Closes #1064 from cdmikechen/SUBMARINE-1363 and squashes the following
commits:
0632373a [cdmikechen] remove df command
82ddd84d [cdmikechen] replace step
c5e09617 [cdmikechen] clear some mvn cache
8327d96f [cdmikechen] remove images
b2ee2369 [cdmikechen] check storage
99a5229e [cdmikechen] change apis
c3773e09 [cdmikechen] fix in integration
780b3d4b [cdmikechen] remove --no-cache-dir
108d4511 [cdmikechen] fix tensorflow-estimator version
a4e950d5 [cdmikechen] fix sed
c60375b1 [cdmikechen] test pipeline
2b471918 [cdmikechen] replace name
d103d5ae [cdmikechen] fix python ci jdk error
579b4d06 [cdmikechen] fix python error
b0eb7cc8 [cdmikechen] Regenerate openapi.json
bc669926 [cdmikechen] Regenerate the api for the sdk
515938c0 [cdmikechen] add GitCodeSpec in front
cc2145ab [cdmikechen] fix construction
ed6861ee [cdmikechen] fix git url
05ce4144 [cdmikechen] fix npe
3896c494 [cdmikechen] backend change
---
.github/scripts/build-image-locally-v3.sh | 2 +
.github/workflows/python.yml | 51 +-
dev-support/pysubmarine/openapi.json | 574 +++++++++++----------
.../example/submarine_experiment_sdk.ipynb | 6 +-
submarine-sdk/pysubmarine/setup.py | 2 +-
.../pysubmarine/submarine/client/__init__.py | 1 +
.../submarine/client/models/__init__.py | 1 +
.../submarine/client/models/code_spec.py | 34 +-
.../client/models/experiment_task_spec.py | 54 +-
.../{notebook_pod_spec.py => git_code_spec.py} | 164 +++---
.../submarine/client/models/notebook_pod_spec.py | 54 +-
.../pysubmarine/tests/cli/test_experiment.py | 3 +-
.../tests/experiment/test_experiment_client.py | 3 +-
.../apache/submarine/server/api/spec/CodeSpec.java | 18 +-
.../submarine/server/api/spec/ExperimentSpec.java | 2 +-
.../server/api/spec/code/GitCodeSpec.java | 92 ++++
.../codelocalizer/AbstractCodeLocalizer.java | 34 +-
.../codelocalizer/DummyCodeLocalizer.java | 3 +-
.../experiment/codelocalizer/GitCodeLocalizer.java | 87 ++--
.../codelocalizer/HTTPGitCodeLocalizer.java | 5 +-
.../codelocalizer/SSHGitCodeLocalizer.java | 5 +-
.../submitter/k8s/parser/ExperimentSpecParser.java | 3 +-
.../K8sResourceUtils.java} | 22 +-
.../pytorch_job_req_http_git_code_localizer.json | 4 +-
.../pytorch_job_req_ssh_git_code_localizer.json | 4 +-
.../tf-mnist-with-http-git-code-localizer-req.json | 4 +-
.../tf-mnist-with-ssh-git-code-localizer-req.json | 4 +-
.../src/app/interfaces/experiment-spec.ts | 8 +
.../experiment-customized-form.component.html | 39 +-
.../experiment-customized-form.component.ts | 24 +-
website/docs/api/experiment-template.md | 21 +-
website/docs/api/experiment.md | 16 +-
website/docs/gettingStarted/notebook.md | 2 +-
33 files changed, 801 insertions(+), 545 deletions(-)
diff --git a/.github/scripts/build-image-locally-v3.sh
b/.github/scripts/build-image-locally-v3.sh
index 00fdb770..e8525469 100755
--- a/.github/scripts/build-image-locally-v3.sh
+++ b/.github/scripts/build-image-locally-v3.sh
@@ -33,5 +33,7 @@ do
echo "Execute ./dev-support/docker-images/${FOLDER_LIST[i]}/build.sh"
./dev-support/docker-images/"${FOLDER_LIST[i]}"/build.sh
kind load docker-image "${IMAGE_LIST[i]}"
+ # remove docker image to reduce current space
+ docker rmi "${IMAGE_LIST[i]}"
done
diff --git a/.github/workflows/python.yml b/.github/workflows/python.yml
index c0727ee4..11c2a7b2 100644
--- a/.github/workflows/python.yml
+++ b/.github/workflows/python.yml
@@ -73,8 +73,9 @@ jobs:
- name: Install pysubmarine with tf2 and pytorch
if: startsWith(matrix.tf-version, '2.')
run: |
+ sed -i "s/tensorflow>=2.6.0,<2.10.0/tensorflow~=${{
matrix.tf-version }}/" ./submarine-sdk/pysubmarine/setup.py
+ sed -i
"s/tensorflow-estimator>=2.6.0,<2.10.0/tensorflow-estimator~=${{
matrix.tf-version }}/" ./submarine-sdk/pysubmarine/setup.py
pip install --no-cache-dir -e
./submarine-sdk/pysubmarine/.[tf2,pytorch]
- pip install --no-cache-dir 'tensorflow~=${{ matrix.tf-version }}'
- name: List installed packages
run: pip list && pipdeptree
- name: Run unit test
@@ -109,10 +110,31 @@ jobs:
- uses: actions/checkout@v3
with:
fetch-depth: 50
- - name: Set up JDK 1.8
+ - name: Setup python environment
+ uses: actions/setup-python@v4
+ with:
+ python-version: ${{ matrix.python-version }}
+ - name: Install tf1 dependencies
+ if: ${{ matrix.tf-version == '1.15.0' }}
+ run: |
+ pip install --upgrade pip setuptools wheel
+ pip install --no-cache-dir -e
./submarine-sdk/pysubmarine/.[tf,pytorch]
+ pip install -r
./submarine-sdk/pysubmarine/github-actions/test-requirements.txt
+ - name: Install tf2 dependencies
+ if: startsWith(matrix.tf-version, '2.')
+ run: |
+ pip install --upgrade pip setuptools wheel
+ sed -i "s/tensorflow>=2.6.0,<2.10.0/tensorflow~=${{
matrix.tf-version }}/" ./submarine-sdk/pysubmarine/setup.py
+ sed -i
"s/tensorflow-estimator>=2.6.0,<2.10.0/tensorflow-estimator~=${{
matrix.tf-version }}/" ./submarine-sdk/pysubmarine/setup.py
+ pip install --no-cache-dir -e
./submarine-sdk/pysubmarine/.[tf2,pytorch]
+ pip install -r
./submarine-sdk/pysubmarine/github-actions/test-requirements.txt
+ pip cache purge
+ - name: List installed packages
+ run: pip list
+ - name: Set up JDK 11
uses: actions/setup-java@v1
with:
- java-version: "1.8"
+ java-version: "11"
- name: Set up Maven 3.6.3
uses: stCarolas/setup-maven@v4
with:
@@ -145,28 +167,11 @@ jobs:
restore-keys: |
${{ runner.os }}-maven-
- name: Build Image locally
- run: .github/scripts/build-image-locally-v3.sh
+ run:
+ .github/scripts/build-image-locally-v3.sh
+ mvn clean
- name: Start submarine
run: bash ./.github/scripts/start-submarine.sh
- - name: Setup python environment
- uses: actions/setup-python@v4
- with:
- python-version: ${{ matrix.python-version }}
- - name: Install tf1 dependencies
- if: ${{ matrix.tf-version == '1.15.0' }}
- run: |
- pip install --upgrade pip setuptools wheel
- pip install --no-cache-dir -e
./submarine-sdk/pysubmarine/.[tf,pytorch]
- pip install -r
./submarine-sdk/pysubmarine/github-actions/test-requirements.txt
- - name: Install tf2 dependencies
- if: startsWith(matrix.tf-version, '2.')
- run: |
- pip install --upgrade pip setuptools wheel
- pip install --no-cache-dir -e
./submarine-sdk/pysubmarine/.[tf2,pytorch]
- pip install --no-cache-dir 'tensorflow~=${{ matrix.tf-version }}'
- pip install -r
./submarine-sdk/pysubmarine/github-actions/test-requirements.txt
- - name: List installed packages
- run: pip list
- name: Set CLI config for e2e pytest
run: |
submarine config set connection.hostname localhost
diff --git a/dev-support/pysubmarine/openapi.json
b/dev-support/pysubmarine/openapi.json
index f04cf32c..0a13aafd 100644
--- a/dev-support/pysubmarine/openapi.json
+++ b/dev-support/pysubmarine/openapi.json
@@ -174,33 +174,68 @@
}
}
},
- "/v1/experiment/{name}" : {
- "post" : {
+ "/v1/experiment/logs/{id}" : {
+ "get" : {
"tags" : [ "experiment" ],
- "summary" : "use experiment template to create an experiment",
- "operationId" : "SubmitExperimentTemplate",
+ "summary" : "Log experiment by id",
+ "operationId" : "getLog",
"parameters" : [ {
- "name" : "name",
+ "name" : "id",
"in" : "path",
"required" : true,
"schema" : {
"type" : "string"
}
} ],
- "requestBody" : {
- "content" : {
- "application/yaml" : {
- "schema" : {
- "$ref" : "#/components/schemas/ExperimentTemplateSubmit"
+ "responses" : {
+ "default" : {
+ "description" : "successful operation",
+ "content" : {
+ "application/json; charset=utf-8" : {
+ "schema" : {
+ "$ref" : "#/components/schemas/JsonResponse"
+ }
}
- },
- "application/json" : {
- "schema" : {
- "$ref" : "#/components/schemas/ExperimentTemplateSubmit"
+ }
+ },
+ "404" : {
+ "description" : "Experiment not found"
+ }
+ }
+ }
+ },
+ "/v1/experiment/ping" : {
+ "get" : {
+ "tags" : [ "experiment" ],
+ "summary" : "Ping submarine server",
+ "description" : "Return the Pong message for test the connectivity",
+ "operationId" : "ping",
+ "responses" : {
+ "200" : {
+ "description" : "successful operation",
+ "content" : {
+ "application/json; charset=utf-8" : {
+ "schema" : {
+ "type" : "string"
+ }
}
}
}
- },
+ }
+ }
+ },
+ "/v1/experiment/logs" : {
+ "get" : {
+ "tags" : [ "experiment" ],
+ "summary" : "List experiment's log",
+ "operationId" : "listLog",
+ "parameters" : [ {
+ "name" : "status",
+ "in" : "query",
+ "schema" : {
+ "type" : "string"
+ }
+ } ],
"responses" : {
"default" : {
"description" : "successful operation",
@@ -272,6 +307,47 @@
}
}
},
+ "/v1/experiment/{name}" : {
+ "post" : {
+ "tags" : [ "experiment" ],
+ "summary" : "use experiment template to create an experiment",
+ "operationId" : "SubmitExperimentTemplate",
+ "parameters" : [ {
+ "name" : "name",
+ "in" : "path",
+ "required" : true,
+ "schema" : {
+ "type" : "string"
+ }
+ } ],
+ "requestBody" : {
+ "content" : {
+ "application/yaml" : {
+ "schema" : {
+ "$ref" : "#/components/schemas/ExperimentTemplateSubmit"
+ }
+ },
+ "application/json" : {
+ "schema" : {
+ "$ref" : "#/components/schemas/ExperimentTemplateSubmit"
+ }
+ }
+ }
+ },
+ "responses" : {
+ "default" : {
+ "description" : "successful operation",
+ "content" : {
+ "application/json; charset=utf-8" : {
+ "schema" : {
+ "$ref" : "#/components/schemas/JsonResponse"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
"/v1/experiment/{id}" : {
"get" : {
"tags" : [ "experiment" ],
@@ -402,19 +478,11 @@
}
}
},
- "/v1/experiment/logs/{id}" : {
+ "/v1/experiment/tensorboard" : {
"get" : {
"tags" : [ "experiment" ],
- "summary" : "Log experiment by id",
- "operationId" : "getLog",
- "parameters" : [ {
- "name" : "id",
- "in" : "path",
- "required" : true,
- "schema" : {
- "type" : "string"
- }
- } ],
+ "summary" : "Get tensorboard's information",
+ "operationId" : "getTensorboardInfo",
"responses" : {
"default" : {
"description" : "successful operation",
@@ -427,7 +495,7 @@
}
},
"404" : {
- "description" : "Experiment not found"
+ "description" : "Tensorboard not found"
}
}
}
@@ -454,74 +522,6 @@
}
}
},
- "/v1/experiment/tensorboard" : {
- "get" : {
- "tags" : [ "experiment" ],
- "summary" : "Get tensorboard's information",
- "operationId" : "getTensorboardInfo",
- "responses" : {
- "default" : {
- "description" : "successful operation",
- "content" : {
- "application/json; charset=utf-8" : {
- "schema" : {
- "$ref" : "#/components/schemas/JsonResponse"
- }
- }
- }
- },
- "404" : {
- "description" : "Tensorboard not found"
- }
- }
- }
- },
- "/v1/experiment/logs" : {
- "get" : {
- "tags" : [ "experiment" ],
- "summary" : "List experiment's log",
- "operationId" : "listLog",
- "parameters" : [ {
- "name" : "status",
- "in" : "query",
- "schema" : {
- "type" : "string"
- }
- } ],
- "responses" : {
- "default" : {
- "description" : "successful operation",
- "content" : {
- "application/json; charset=utf-8" : {
- "schema" : {
- "$ref" : "#/components/schemas/JsonResponse"
- }
- }
- }
- }
- }
- }
- },
- "/v1/experiment/ping" : {
- "get" : {
- "tags" : [ "experiment" ],
- "summary" : "Ping submarine server",
- "description" : "Return the Pong message for test the connectivity",
- "operationId" : "ping",
- "responses" : {
- "200" : {
- "description" : "successful operation",
- "content" : {
- "application/json; charset=utf-8" : {
- "schema" : {
- "type" : "string"
- }
- }
- }
- }
- }
- }
- },
"/v1/template" : {
"get" : {
"tags" : [ "experimentTemplates" ],
@@ -679,6 +679,26 @@
}
}
},
+ "/v1/model-version/ping" : {
+ "get" : {
+ "tags" : [ "model-version" ],
+ "summary" : "Ping submarine server",
+ "description" : "Return the Pong message for test the connectivity",
+ "operationId" : "ping_1",
+ "responses" : {
+ "200" : {
+ "description" : "successful operation",
+ "content" : {
+ "application/json; charset=utf-8" : {
+ "schema" : {
+ "type" : "string"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
"/v1/model-version" : {
"post" : {
"tags" : [ "model-version" ],
@@ -748,31 +768,17 @@
}
}
},
- "/v1/model-version/tag" : {
- "post" : {
+ "/v1/model-version/{name}" : {
+ "get" : {
"tags" : [ "model-version" ],
- "summary" : "Create a model version tag instance",
- "operationId" : "createModelVersionTag",
+ "summary" : "List model versions",
+ "operationId" : "listModelVersions",
"parameters" : [ {
"name" : "name",
- "in" : "query",
- "schema" : {
- "type" : "string",
- "default" : ""
- }
- }, {
- "name" : "version",
- "in" : "query",
- "schema" : {
- "type" : "string",
- "default" : ""
- }
- }, {
- "name" : "tag",
- "in" : "query",
+ "in" : "path",
+ "required" : true,
"schema" : {
- "type" : "string",
- "default" : ""
+ "type" : "string"
}
} ],
"responses" : {
@@ -787,31 +793,27 @@
}
}
}
- },
- "delete" : {
+ }
+ },
+ "/v1/model-version/{name}/{version}" : {
+ "get" : {
"tags" : [ "model-version" ],
- "summary" : "Delete a model version tag instance",
- "operationId" : "deleteModelVersionTag",
+ "summary" : "Get detailed info about the model version",
+ "operationId" : "getModelVersion",
"parameters" : [ {
"name" : "name",
- "in" : "query",
+ "in" : "path",
+ "required" : true,
"schema" : {
- "type" : "string",
- "default" : ""
+ "type" : "string"
}
}, {
"name" : "version",
- "in" : "query",
- "schema" : {
- "type" : "string",
- "default" : ""
- }
- }, {
- "name" : "tag",
- "in" : "query",
+ "in" : "path",
+ "required" : true,
"schema" : {
- "type" : "string",
- "default" : ""
+ "type" : "integer",
+ "format" : "int32"
}
} ],
"responses" : {
@@ -824,15 +826,16 @@
}
}
}
+ },
+ "404" : {
+ "description" : "ModelVersionEntity not found"
}
}
- }
- },
- "/v1/model-version/{name}/{version}" : {
- "get" : {
+ },
+ "delete" : {
"tags" : [ "model-version" ],
- "summary" : "Get detailed info about the model version",
- "operationId" : "getModelVersion",
+ "summary" : "Delete the model version",
+ "operationId" : "deleteModelVersion",
"parameters" : [ {
"name" : "name",
"in" : "path",
@@ -864,25 +867,33 @@
"description" : "ModelVersionEntity not found"
}
}
- },
- "delete" : {
+ }
+ },
+ "/v1/model-version/tag" : {
+ "post" : {
"tags" : [ "model-version" ],
- "summary" : "Delete the model version",
- "operationId" : "deleteModelVersion",
+ "summary" : "Create a model version tag instance",
+ "operationId" : "createModelVersionTag",
"parameters" : [ {
"name" : "name",
- "in" : "path",
- "required" : true,
+ "in" : "query",
"schema" : {
- "type" : "string"
+ "type" : "string",
+ "default" : ""
}
}, {
"name" : "version",
- "in" : "path",
- "required" : true,
+ "in" : "query",
"schema" : {
- "type" : "integer",
- "format" : "int32"
+ "type" : "string",
+ "default" : ""
+ }
+ }, {
+ "name" : "tag",
+ "in" : "query",
+ "schema" : {
+ "type" : "string",
+ "default" : ""
}
} ],
"responses" : {
@@ -895,24 +906,33 @@
}
}
}
- },
- "404" : {
- "description" : "ModelVersionEntity not found"
}
}
- }
- },
- "/v1/model-version/{name}" : {
- "get" : {
+ },
+ "delete" : {
"tags" : [ "model-version" ],
- "summary" : "List model versions",
- "operationId" : "listModelVersions",
+ "summary" : "Delete a model version tag instance",
+ "operationId" : "deleteModelVersionTag",
"parameters" : [ {
"name" : "name",
- "in" : "path",
- "required" : true,
+ "in" : "query",
"schema" : {
- "type" : "string"
+ "type" : "string",
+ "default" : ""
+ }
+ }, {
+ "name" : "version",
+ "in" : "query",
+ "schema" : {
+ "type" : "string",
+ "default" : ""
+ }
+ }, {
+ "name" : "tag",
+ "in" : "query",
+ "schema" : {
+ "type" : "string",
+ "default" : ""
}
} ],
"responses" : {
@@ -929,12 +949,12 @@
}
}
},
- "/v1/model-version/ping" : {
+ "/v1/notebook/ping" : {
"get" : {
- "tags" : [ "model-version" ],
+ "tags" : [ "notebook" ],
"summary" : "Ping submarine server",
"description" : "Return the Pong message for test the connectivity",
- "operationId" : "ping_1",
+ "operationId" : "ping_2",
"responses" : {
"200" : {
"description" : "successful operation",
@@ -1064,12 +1084,12 @@
}
}
},
- "/v1/notebook/ping" : {
+ "/v1/registered-model/ping" : {
"get" : {
- "tags" : [ "notebook" ],
+ "tags" : [ "registered-model" ],
"summary" : "Ping submarine server",
"description" : "Return the Pong message for test the connectivity",
- "operationId" : "ping_2",
+ "operationId" : "ping_3",
"responses" : {
"200" : {
"description" : "successful operation",
@@ -1134,72 +1154,6 @@
}
}
},
- "/v1/registered-model/tag" : {
- "post" : {
- "tags" : [ "registered-model" ],
- "summary" : "Create a registered model tag instance",
- "operationId" : "createRegisteredModelTag",
- "parameters" : [ {
- "name" : "name",
- "in" : "query",
- "schema" : {
- "type" : "string",
- "default" : ""
- }
- }, {
- "name" : "tag",
- "in" : "query",
- "schema" : {
- "type" : "string",
- "default" : ""
- }
- } ],
- "responses" : {
- "default" : {
- "description" : "successful operation",
- "content" : {
- "application/json; charset=utf-8" : {
- "schema" : {
- "$ref" : "#/components/schemas/JsonResponse"
- }
- }
- }
- }
- }
- },
- "delete" : {
- "tags" : [ "registered-model" ],
- "summary" : "Delete a registered model tag instance",
- "operationId" : "deleteRegisteredModelTag",
- "parameters" : [ {
- "name" : "name",
- "in" : "query",
- "schema" : {
- "type" : "string",
- "default" : ""
- }
- }, {
- "name" : "tag",
- "in" : "query",
- "schema" : {
- "type" : "string",
- "default" : ""
- }
- } ],
- "responses" : {
- "default" : {
- "description" : "successful operation",
- "content" : {
- "application/json; charset=utf-8" : {
- "schema" : {
- "$ref" : "#/components/schemas/JsonResponse"
- }
- }
- }
- }
- }
- }
- },
"/v1/registered-model/{name}" : {
"get" : {
"tags" : [ "registered-model" ],
@@ -1301,12 +1255,78 @@
}
}
},
- "/v1/registered-model/ping" : {
- "get" : {
+ "/v1/registered-model/tag" : {
+ "post" : {
+ "tags" : [ "registered-model" ],
+ "summary" : "Create a registered model tag instance",
+ "operationId" : "createRegisteredModelTag",
+ "parameters" : [ {
+ "name" : "name",
+ "in" : "query",
+ "schema" : {
+ "type" : "string",
+ "default" : ""
+ }
+ }, {
+ "name" : "tag",
+ "in" : "query",
+ "schema" : {
+ "type" : "string",
+ "default" : ""
+ }
+ } ],
+ "responses" : {
+ "default" : {
+ "description" : "successful operation",
+ "content" : {
+ "application/json; charset=utf-8" : {
+ "schema" : {
+ "$ref" : "#/components/schemas/JsonResponse"
+ }
+ }
+ }
+ }
+ }
+ },
+ "delete" : {
"tags" : [ "registered-model" ],
+ "summary" : "Delete a registered model tag instance",
+ "operationId" : "deleteRegisteredModelTag",
+ "parameters" : [ {
+ "name" : "name",
+ "in" : "query",
+ "schema" : {
+ "type" : "string",
+ "default" : ""
+ }
+ }, {
+ "name" : "tag",
+ "in" : "query",
+ "schema" : {
+ "type" : "string",
+ "default" : ""
+ }
+ } ],
+ "responses" : {
+ "default" : {
+ "description" : "successful operation",
+ "content" : {
+ "application/json; charset=utf-8" : {
+ "schema" : {
+ "$ref" : "#/components/schemas/JsonResponse"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "/v1/serve/ping" : {
+ "get" : {
+ "tags" : [ "serve" ],
"summary" : "Ping submarine server",
"description" : "Return the Pong message for test the connectivity",
- "operationId" : "ping_3",
+ "operationId" : "ping_4",
"responses" : {
"200" : {
"description" : "successful operation",
@@ -1382,26 +1402,6 @@
}
}
}
- },
- "/v1/serve/ping" : {
- "get" : {
- "tags" : [ "serve" ],
- "summary" : "Ping submarine server",
- "description" : "Return the Pong message for test the connectivity",
- "operationId" : "ping_4",
- "responses" : {
- "200" : {
- "description" : "successful operation",
- "content" : {
- "application/json; charset=utf-8" : {
- "schema" : {
- "type" : "string"
- }
- }
- }
- }
- }
- }
}
},
"components" : {
@@ -1473,28 +1473,14 @@
}
}
},
- "ExperimentTemplateSubmit" : {
- "type" : "object",
- "properties" : {
- "name" : {
- "type" : "string"
- },
- "params" : {
- "type" : "object",
- "additionalProperties" : {
- "type" : "string"
- }
- }
- }
- },
"CodeSpec" : {
"type" : "object",
"properties" : {
"syncMode" : {
"type" : "string"
},
- "url" : {
- "type" : "string"
+ "git" : {
+ "$ref" : "#/components/schemas/GitCodeSpec"
}
}
},
@@ -1581,14 +1567,48 @@
"type" : "string"
}
},
+ "memory" : {
+ "type" : "string"
+ },
"cpu" : {
"type" : "string"
},
"gpu" : {
"type" : "string"
+ }
+ }
+ },
+ "GitCodeSpec" : {
+ "type" : "object",
+ "properties" : {
+ "url" : {
+ "type" : "string"
},
- "memory" : {
+ "branch" : {
+ "type" : "string"
+ },
+ "username" : {
+ "type" : "string"
+ },
+ "password" : {
+ "type" : "string"
+ },
+ "trustCerts" : {
+ "type" : "boolean"
+ }
+ }
+ },
+ "ExperimentTemplateSubmit" : {
+ "type" : "object",
+ "properties" : {
+ "name" : {
"type" : "string"
+ },
+ "params" : {
+ "type" : "object",
+ "additionalProperties" : {
+ "type" : "string"
+ }
}
}
},
@@ -1721,13 +1741,13 @@
"resources" : {
"type" : "string"
},
- "cpu" : {
+ "memory" : {
"type" : "string"
},
- "gpu" : {
+ "cpu" : {
"type" : "string"
},
- "memory" : {
+ "gpu" : {
"type" : "string"
}
}
diff --git a/submarine-sdk/pysubmarine/example/submarine_experiment_sdk.ipynb
b/submarine-sdk/pysubmarine/example/submarine_experiment_sdk.ipynb
index 63012a41..13b2214d 100644
--- a/submarine-sdk/pysubmarine/example/submarine_experiment_sdk.ipynb
+++ b/submarine-sdk/pysubmarine/example/submarine_experiment_sdk.ipynb
@@ -95,7 +95,7 @@
"\n",
"worker_spec = ExperimentTaskSpec(resources=\"cpu=1,memory=1024M\",
replicas=1)\n",
"ps_spec = ExperimentTaskSpec(resources=\"cpu=1,memory=1024M\",
replicas=1)\n",
- "code_spec = CodeSpec(sync_mode=\"git\",
url=\"https://github.com/apache/submarine.git\";)\n",
+ "code_spec = CodeSpec(sync_mode=\"git\",
GitCodeSpec=(url=\"https://github.com/apache/submarine.git\";))\n",
"\n",
"experiment_spec = ExperimentSpec(\n",
" meta=experiment_meta,\n",
@@ -159,7 +159,7 @@
" 'envVars': None,\n",
" 'resourceMap': {'memory': '1024M', 'cpu': '1'}}},\n",
" 'code': {'syncMode': 'git',\n",
- " 'url': 'https://github.com/apache/submarine.git'}}}"
+ " 'git': {'url': 'https://github.com/apache/submarine.git'}}}}"
]
},
"execution_count": 5,
@@ -520,7 +520,7 @@
" 'envVars': None,\n",
" 'resourceMap': {'memory': '1024M', 'cpu': '1'}}},\n",
" 'code': {'syncMode': 'git',\n",
- " 'url': 'https://github.com/apache/submarine.git'}}}"
+ " 'git': {'url': 'https://github.com/apache/submarine.git'}}}}"
]
},
"execution_count": 10,
diff --git a/submarine-sdk/pysubmarine/setup.py
b/submarine-sdk/pysubmarine/setup.py
index 506d616d..d66520c1 100644
--- a/submarine-sdk/pysubmarine/setup.py
+++ b/submarine-sdk/pysubmarine/setup.py
@@ -52,7 +52,7 @@ setup(
"keras>=2.6.0",
"protobuf>=3.9.2,<3.20",
"tensorflow-addons==0.17.0",
- "tensorflow-estimator>=2.9.0rc0,<2.10.0",
+ "tensorflow-estimator>=2.6.0,<2.10.0",
"tf_slim==1.1.0",
# todo(cdmikechen): Based on SUBMARINE-1372, typeguard has
recently been upgraded to version 3.0,
# which will restrict some python syntax and
types more tightly.
diff --git a/submarine-sdk/pysubmarine/submarine/client/__init__.py
b/submarine-sdk/pysubmarine/submarine/client/__init__.py
index ae9d7b53..08034ec8 100644
--- a/submarine-sdk/pysubmarine/submarine/client/__init__.py
+++ b/submarine-sdk/pysubmarine/submarine/client/__init__.py
@@ -64,6 +64,7 @@ from submarine.client.models.experiment_template_param_spec
import (
)
from submarine.client.models.experiment_template_spec import
ExperimentTemplateSpec
from submarine.client.models.experiment_template_submit import
ExperimentTemplateSubmit
+from submarine.client.models.git_code_spec import GitCodeSpec
from submarine.client.models.json_response import JsonResponse
from submarine.client.models.kernel_spec import KernelSpec
from submarine.client.models.model_version_entity import ModelVersionEntity
diff --git a/submarine-sdk/pysubmarine/submarine/client/models/__init__.py
b/submarine-sdk/pysubmarine/submarine/client/models/__init__.py
index 1cb407d9..84995761 100644
--- a/submarine-sdk/pysubmarine/submarine/client/models/__init__.py
+++ b/submarine-sdk/pysubmarine/submarine/client/models/__init__.py
@@ -40,6 +40,7 @@ from submarine.client.models.experiment_template_param_spec
import (
)
from submarine.client.models.experiment_template_spec import
ExperimentTemplateSpec
from submarine.client.models.experiment_template_submit import
ExperimentTemplateSubmit
+from submarine.client.models.git_code_spec import GitCodeSpec
from submarine.client.models.json_response import JsonResponse
from submarine.client.models.kernel_spec import KernelSpec
from submarine.client.models.model_version_entity import ModelVersionEntity
diff --git a/submarine-sdk/pysubmarine/submarine/client/models/code_spec.py
b/submarine-sdk/pysubmarine/submarine/client/models/code_spec.py
index 2cd8e0ab..499850a5 100644
--- a/submarine-sdk/pysubmarine/submarine/client/models/code_spec.py
+++ b/submarine-sdk/pysubmarine/submarine/client/models/code_spec.py
@@ -48,24 +48,24 @@ class CodeSpec(object):
attribute_map (dict): The key is attribute name
and the value is json key in definition.
"""
- openapi_types = {'sync_mode': 'str', 'url': 'str'}
+ openapi_types = {'sync_mode': 'str', 'git': 'GitCodeSpec'}
- attribute_map = {'sync_mode': 'syncMode', 'url': 'url'}
+ attribute_map = {'sync_mode': 'syncMode', 'git': 'git'}
- def __init__(self, sync_mode=None, url=None,
local_vars_configuration=None): # noqa: E501
+ def __init__(self, sync_mode=None, git=None,
local_vars_configuration=None): # noqa: E501
"""CodeSpec - a model defined in OpenAPI""" # noqa: E501
if local_vars_configuration is None:
local_vars_configuration = Configuration()
self.local_vars_configuration = local_vars_configuration
self._sync_mode = None
- self._url = None
+ self._git = None
self.discriminator = None
if sync_mode is not None:
self.sync_mode = sync_mode
- if url is not None:
- self.url = url
+ if git is not None:
+ self.git = git
@property
def sync_mode(self):
@@ -89,25 +89,25 @@ class CodeSpec(object):
self._sync_mode = sync_mode
@property
- def url(self):
- """Gets the url of this CodeSpec. # noqa: E501
+ def git(self):
+ """Gets the git of this CodeSpec. # noqa: E501
- :return: The url of this CodeSpec. # noqa: E501
- :rtype: str
+ :return: The git of this CodeSpec. # noqa: E501
+ :rtype: GitCodeSpec
"""
- return self._url
+ return self._git
- @url.setter
- def url(self, url):
- """Sets the url of this CodeSpec.
+ @git.setter
+ def git(self, git):
+ """Sets the git of this CodeSpec.
- :param url: The url of this CodeSpec. # noqa: E501
- :type: str
+ :param git: The git of this CodeSpec. # noqa: E501
+ :type: GitCodeSpec
"""
- self._url = url
+ self._git = git
def to_dict(self):
"""Returns the model properties as a dict"""
diff --git
a/submarine-sdk/pysubmarine/submarine/client/models/experiment_task_spec.py
b/submarine-sdk/pysubmarine/submarine/client/models/experiment_task_spec.py
index 70f896d4..274e433e 100644
--- a/submarine-sdk/pysubmarine/submarine/client/models/experiment_task_spec.py
+++ b/submarine-sdk/pysubmarine/submarine/client/models/experiment_task_spec.py
@@ -55,9 +55,9 @@ class ExperimentTaskSpec(object):
'image': 'str',
'cmd': 'str',
'env_vars': 'dict(str, str)',
+ 'memory': 'str',
'cpu': 'str',
'gpu': 'str',
- 'memory': 'str',
}
attribute_map = {
@@ -67,9 +67,9 @@ class ExperimentTaskSpec(object):
'image': 'image',
'cmd': 'cmd',
'env_vars': 'envVars',
+ 'memory': 'memory',
'cpu': 'cpu',
'gpu': 'gpu',
- 'memory': 'memory',
}
def __init__(
@@ -80,9 +80,9 @@ class ExperimentTaskSpec(object):
image=None,
cmd=None,
env_vars=None,
+ memory=None,
cpu=None,
gpu=None,
- memory=None,
local_vars_configuration=None,
): # noqa: E501
"""ExperimentTaskSpec - a model defined in OpenAPI""" # noqa: E501
@@ -96,9 +96,9 @@ class ExperimentTaskSpec(object):
self._image = None
self._cmd = None
self._env_vars = None
+ self._memory = None
self._cpu = None
self._gpu = None
- self._memory = None
self.discriminator = None
if replicas is not None:
@@ -113,12 +113,12 @@ class ExperimentTaskSpec(object):
self.cmd = cmd
if env_vars is not None:
self.env_vars = env_vars
+ if memory is not None:
+ self.memory = memory
if cpu is not None:
self.cpu = cpu
if gpu is not None:
self.gpu = gpu
- if memory is not None:
- self.memory = memory
@property
def replicas(self):
@@ -246,6 +246,27 @@ class ExperimentTaskSpec(object):
self._env_vars = env_vars
+ @property
+ def memory(self):
+ """Gets the memory of this ExperimentTaskSpec. # noqa: E501
+
+
+ :return: The memory of this ExperimentTaskSpec. # noqa: E501
+ :rtype: str
+ """
+ return self._memory
+
+ @memory.setter
+ def memory(self, memory):
+ """Sets the memory of this ExperimentTaskSpec.
+
+
+ :param memory: The memory of this ExperimentTaskSpec. # noqa: E501
+ :type: str
+ """
+
+ self._memory = memory
+
@property
def cpu(self):
"""Gets the cpu of this ExperimentTaskSpec. # noqa: E501
@@ -288,27 +309,6 @@ class ExperimentTaskSpec(object):
self._gpu = gpu
- @property
- def memory(self):
- """Gets the memory of this ExperimentTaskSpec. # noqa: E501
-
-
- :return: The memory of this ExperimentTaskSpec. # noqa: E501
- :rtype: str
- """
- return self._memory
-
- @memory.setter
- def memory(self, memory):
- """Sets the memory of this ExperimentTaskSpec.
-
-
- :param memory: The memory of this ExperimentTaskSpec. # noqa: E501
- :type: str
- """
-
- self._memory = memory
-
def to_dict(self):
"""Returns the model properties as a dict"""
result = {}
diff --git
a/submarine-sdk/pysubmarine/submarine/client/models/notebook_pod_spec.py
b/submarine-sdk/pysubmarine/submarine/client/models/git_code_spec.py
similarity index 56%
copy from submarine-sdk/pysubmarine/submarine/client/models/notebook_pod_spec.py
copy to submarine-sdk/pysubmarine/submarine/client/models/git_code_spec.py
index cf1855f4..91b84c0b 100644
--- a/submarine-sdk/pysubmarine/submarine/client/models/notebook_pod_spec.py
+++ b/submarine-sdk/pysubmarine/submarine/client/models/git_code_spec.py
@@ -34,7 +34,7 @@ import six
from submarine.client.configuration import Configuration
-class NotebookPodSpec(object):
+class GitCodeSpec(object):
"""NOTE: This class is auto generated by OpenAPI Generator.
Ref: https://openapi-generator.tech
@@ -49,151 +49,157 @@ class NotebookPodSpec(object):
and the value is json key in definition.
"""
openapi_types = {
- 'env_vars': 'dict(str, str)',
- 'resources': 'str',
- 'cpu': 'str',
- 'gpu': 'str',
- 'memory': 'str',
+ 'url': 'str',
+ 'branch': 'str',
+ 'username': 'str',
+ 'password': 'str',
+ 'trust_certs': 'bool',
}
attribute_map = {
- 'env_vars': 'envVars',
- 'resources': 'resources',
- 'cpu': 'cpu',
- 'gpu': 'gpu',
- 'memory': 'memory',
+ 'url': 'url',
+ 'branch': 'branch',
+ 'username': 'username',
+ 'password': 'password',
+ 'trust_certs': 'trustCerts',
}
def __init__(
- self, env_vars=None, resources=None, cpu=None, gpu=None, memory=None,
local_vars_configuration=None
+ self,
+ url=None,
+ branch=None,
+ username=None,
+ password=None,
+ trust_certs=None,
+ local_vars_configuration=None,
): # noqa: E501
- """NotebookPodSpec - a model defined in OpenAPI""" # noqa: E501
+ """GitCodeSpec - a model defined in OpenAPI""" # noqa: E501
if local_vars_configuration is None:
local_vars_configuration = Configuration()
self.local_vars_configuration = local_vars_configuration
- self._env_vars = None
- self._resources = None
- self._cpu = None
- self._gpu = None
- self._memory = None
+ self._url = None
+ self._branch = None
+ self._username = None
+ self._password = None
+ self._trust_certs = None
self.discriminator = None
- if env_vars is not None:
- self.env_vars = env_vars
- if resources is not None:
- self.resources = resources
- if cpu is not None:
- self.cpu = cpu
- if gpu is not None:
- self.gpu = gpu
- if memory is not None:
- self.memory = memory
+ if url is not None:
+ self.url = url
+ if branch is not None:
+ self.branch = branch
+ if username is not None:
+ self.username = username
+ if password is not None:
+ self.password = password
+ if trust_certs is not None:
+ self.trust_certs = trust_certs
@property
- def env_vars(self):
- """Gets the env_vars of this NotebookPodSpec. # noqa: E501
+ def url(self):
+ """Gets the url of this GitCodeSpec. # noqa: E501
- :return: The env_vars of this NotebookPodSpec. # noqa: E501
- :rtype: dict(str, str)
+ :return: The url of this GitCodeSpec. # noqa: E501
+ :rtype: str
"""
- return self._env_vars
+ return self._url
- @env_vars.setter
- def env_vars(self, env_vars):
- """Sets the env_vars of this NotebookPodSpec.
+ @url.setter
+ def url(self, url):
+ """Sets the url of this GitCodeSpec.
- :param env_vars: The env_vars of this NotebookPodSpec. # noqa: E501
- :type: dict(str, str)
+ :param url: The url of this GitCodeSpec. # noqa: E501
+ :type: str
"""
- self._env_vars = env_vars
+ self._url = url
@property
- def resources(self):
- """Gets the resources of this NotebookPodSpec. # noqa: E501
+ def branch(self):
+ """Gets the branch of this GitCodeSpec. # noqa: E501
- :return: The resources of this NotebookPodSpec. # noqa: E501
+ :return: The branch of this GitCodeSpec. # noqa: E501
:rtype: str
"""
- return self._resources
+ return self._branch
- @resources.setter
- def resources(self, resources):
- """Sets the resources of this NotebookPodSpec.
+ @branch.setter
+ def branch(self, branch):
+ """Sets the branch of this GitCodeSpec.
- :param resources: The resources of this NotebookPodSpec. # noqa: E501
+ :param branch: The branch of this GitCodeSpec. # noqa: E501
:type: str
"""
- self._resources = resources
+ self._branch = branch
@property
- def cpu(self):
- """Gets the cpu of this NotebookPodSpec. # noqa: E501
+ def username(self):
+ """Gets the username of this GitCodeSpec. # noqa: E501
- :return: The cpu of this NotebookPodSpec. # noqa: E501
+ :return: The username of this GitCodeSpec. # noqa: E501
:rtype: str
"""
- return self._cpu
+ return self._username
- @cpu.setter
- def cpu(self, cpu):
- """Sets the cpu of this NotebookPodSpec.
+ @username.setter
+ def username(self, username):
+ """Sets the username of this GitCodeSpec.
- :param cpu: The cpu of this NotebookPodSpec. # noqa: E501
+ :param username: The username of this GitCodeSpec. # noqa: E501
:type: str
"""
- self._cpu = cpu
+ self._username = username
@property
- def gpu(self):
- """Gets the gpu of this NotebookPodSpec. # noqa: E501
+ def password(self):
+ """Gets the password of this GitCodeSpec. # noqa: E501
- :return: The gpu of this NotebookPodSpec. # noqa: E501
+ :return: The password of this GitCodeSpec. # noqa: E501
:rtype: str
"""
- return self._gpu
+ return self._password
- @gpu.setter
- def gpu(self, gpu):
- """Sets the gpu of this NotebookPodSpec.
+ @password.setter
+ def password(self, password):
+ """Sets the password of this GitCodeSpec.
- :param gpu: The gpu of this NotebookPodSpec. # noqa: E501
+ :param password: The password of this GitCodeSpec. # noqa: E501
:type: str
"""
- self._gpu = gpu
+ self._password = password
@property
- def memory(self):
- """Gets the memory of this NotebookPodSpec. # noqa: E501
+ def trust_certs(self):
+ """Gets the trust_certs of this GitCodeSpec. # noqa: E501
- :return: The memory of this NotebookPodSpec. # noqa: E501
- :rtype: str
+ :return: The trust_certs of this GitCodeSpec. # noqa: E501
+ :rtype: bool
"""
- return self._memory
+ return self._trust_certs
- @memory.setter
- def memory(self, memory):
- """Sets the memory of this NotebookPodSpec.
+ @trust_certs.setter
+ def trust_certs(self, trust_certs):
+ """Sets the trust_certs of this GitCodeSpec.
- :param memory: The memory of this NotebookPodSpec. # noqa: E501
- :type: str
+ :param trust_certs: The trust_certs of this GitCodeSpec. # noqa: E501
+ :type: bool
"""
- self._memory = memory
+ self._trust_certs = trust_certs
def to_dict(self):
"""Returns the model properties as a dict"""
@@ -227,14 +233,14 @@ class NotebookPodSpec(object):
def __eq__(self, other):
"""Returns true if both objects are equal"""
- if not isinstance(other, NotebookPodSpec):
+ if not isinstance(other, GitCodeSpec):
return False
return self.to_dict() == other.to_dict()
def __ne__(self, other):
"""Returns true if both objects are not equal"""
- if not isinstance(other, NotebookPodSpec):
+ if not isinstance(other, GitCodeSpec):
return True
return self.to_dict() != other.to_dict()
diff --git
a/submarine-sdk/pysubmarine/submarine/client/models/notebook_pod_spec.py
b/submarine-sdk/pysubmarine/submarine/client/models/notebook_pod_spec.py
index cf1855f4..2e0969b8 100644
--- a/submarine-sdk/pysubmarine/submarine/client/models/notebook_pod_spec.py
+++ b/submarine-sdk/pysubmarine/submarine/client/models/notebook_pod_spec.py
@@ -51,21 +51,21 @@ class NotebookPodSpec(object):
openapi_types = {
'env_vars': 'dict(str, str)',
'resources': 'str',
+ 'memory': 'str',
'cpu': 'str',
'gpu': 'str',
- 'memory': 'str',
}
attribute_map = {
'env_vars': 'envVars',
'resources': 'resources',
+ 'memory': 'memory',
'cpu': 'cpu',
'gpu': 'gpu',
- 'memory': 'memory',
}
def __init__(
- self, env_vars=None, resources=None, cpu=None, gpu=None, memory=None,
local_vars_configuration=None
+ self, env_vars=None, resources=None, memory=None, cpu=None, gpu=None,
local_vars_configuration=None
): # noqa: E501
"""NotebookPodSpec - a model defined in OpenAPI""" # noqa: E501
if local_vars_configuration is None:
@@ -74,21 +74,21 @@ class NotebookPodSpec(object):
self._env_vars = None
self._resources = None
+ self._memory = None
self._cpu = None
self._gpu = None
- self._memory = None
self.discriminator = None
if env_vars is not None:
self.env_vars = env_vars
if resources is not None:
self.resources = resources
+ if memory is not None:
+ self.memory = memory
if cpu is not None:
self.cpu = cpu
if gpu is not None:
self.gpu = gpu
- if memory is not None:
- self.memory = memory
@property
def env_vars(self):
@@ -132,6 +132,27 @@ class NotebookPodSpec(object):
self._resources = resources
+ @property
+ def memory(self):
+ """Gets the memory of this NotebookPodSpec. # noqa: E501
+
+
+ :return: The memory of this NotebookPodSpec. # noqa: E501
+ :rtype: str
+ """
+ return self._memory
+
+ @memory.setter
+ def memory(self, memory):
+ """Sets the memory of this NotebookPodSpec.
+
+
+ :param memory: The memory of this NotebookPodSpec. # noqa: E501
+ :type: str
+ """
+
+ self._memory = memory
+
@property
def cpu(self):
"""Gets the cpu of this NotebookPodSpec. # noqa: E501
@@ -174,27 +195,6 @@ class NotebookPodSpec(object):
self._gpu = gpu
- @property
- def memory(self):
- """Gets the memory of this NotebookPodSpec. # noqa: E501
-
-
- :return: The memory of this NotebookPodSpec. # noqa: E501
- :rtype: str
- """
- return self._memory
-
- @memory.setter
- def memory(self, memory):
- """Sets the memory of this NotebookPodSpec.
-
-
- :param memory: The memory of this NotebookPodSpec. # noqa: E501
- :type: str
- """
-
- self._memory = memory
-
def to_dict(self):
"""Returns the model properties as a dict"""
result = {}
diff --git a/submarine-sdk/pysubmarine/tests/cli/test_experiment.py
b/submarine-sdk/pysubmarine/tests/cli/test_experiment.py
index b46bd83e..bd0e3c23 100644
--- a/submarine-sdk/pysubmarine/tests/cli/test_experiment.py
+++ b/submarine-sdk/pysubmarine/tests/cli/test_experiment.py
@@ -18,6 +18,7 @@ from click.testing import CliRunner
import submarine
from submarine.cli import main
+from submarine.client import GitCodeSpec
from submarine.client.models.code_spec import CodeSpec
from submarine.client.models.environment_spec import EnvironmentSpec
from submarine.client.models.experiment_meta import ExperimentMeta
@@ -56,7 +57,7 @@ def test_all_experiment_e2e():
worker_spec = ExperimentTaskSpec(resources="cpu=1,memory=1024M",
replicas=1)
ps_spec = ExperimentTaskSpec(resources="cpu=1,memory=1024M", replicas=1)
- code_spec = CodeSpec(sync_mode="git",
url="https://github.com/apache/submarine.git";)
+ code_spec = CodeSpec(sync_mode="git",
git=GitCodeSpec(url="https://github.com/apache/submarine.git";))
experiment_spec = ExperimentSpec(
meta=experiment_meta,
diff --git
a/submarine-sdk/pysubmarine/tests/experiment/test_experiment_client.py
b/submarine-sdk/pysubmarine/tests/experiment/test_experiment_client.py
index c7c47693..adbd0fff 100644
--- a/submarine-sdk/pysubmarine/tests/experiment/test_experiment_client.py
+++ b/submarine-sdk/pysubmarine/tests/experiment/test_experiment_client.py
@@ -16,6 +16,7 @@
import pytest
import submarine
+from submarine.client import GitCodeSpec
from submarine.client.models.code_spec import CodeSpec
from submarine.client.models.environment_spec import EnvironmentSpec
from submarine.client.models.experiment_meta import ExperimentMeta
@@ -38,7 +39,7 @@ def test_experiment_e2e():
worker_spec = ExperimentTaskSpec(resources="cpu=1,memory=1024M",
replicas=1)
ps_spec = ExperimentTaskSpec(resources="cpu=1,memory=1024M", replicas=1)
- code_spec = CodeSpec(sync_mode="git",
url="https://github.com/apache/submarine.git";)
+ code_spec = CodeSpec(sync_mode="git",
git=GitCodeSpec(url="https://github.com/apache/submarine.git";))
experiment_spec = ExperimentSpec(
meta=experiment_meta,
diff --git
a/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/CodeSpec.java
b/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/CodeSpec.java
index 71585e1a..e7841088 100644
---
a/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/CodeSpec.java
+++
b/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/CodeSpec.java
@@ -19,12 +19,14 @@
package org.apache.submarine.server.api.spec;
+import org.apache.submarine.server.api.spec.code.GitCodeSpec;
+
public class CodeSpec {
-
+
private String syncMode;
-
- private String url;
-
+
+ private GitCodeSpec git;
+
public String getSyncMode() {
return syncMode;
}
@@ -33,11 +35,11 @@ public class CodeSpec {
this.syncMode = syncMode;
}
- public String getUrl() {
- return url;
+ public GitCodeSpec getGit() {
+ return git;
}
- public void setUrl(String url) {
- this.url = url;
+ public void setGit(GitCodeSpec git) {
+ this.git = git;
}
}
diff --git
a/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/ExperimentSpec.java
b/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/ExperimentSpec.java
index 8c3024d6..99a36d3b 100644
---
a/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/ExperimentSpec.java
+++
b/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/ExperimentSpec.java
@@ -64,7 +64,7 @@ public class ExperimentSpec {
public void setCode(CodeSpec code) {
this.code = code;
}
-
+
public Map<String, String> getExperimentHandlerSpec() {
return experimentHandlerSpec;
}
diff --git
a/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/code/GitCodeSpec.java
b/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/code/GitCodeSpec.java
new file mode 100644
index 00000000..34dcfe6d
--- /dev/null
+++
b/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/spec/code/GitCodeSpec.java
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.submarine.server.api.spec.code;
+
+/**
+ * Describes some of the main variable properties of git code
+ */
+public class GitCodeSpec {
+
+ /**
+ * Default git branch.
+ * The new git branch has changed from master to main
+ */
+ public static final String DEFAULT_BRANCH = "main";
+
+ private String url;
+
+ private String branch = DEFAULT_BRANCH;
+
+ private String username;
+
+ private String password;
+
+ /**
+ * Whether the git-sync should trust a self-signed certificate
+ */
+ private Boolean trustCerts;
+
+ public GitCodeSpec() {
+ }
+
+ public GitCodeSpec(String url) {
+ this.url = url;
+ }
+
+ public void setUrl(String url) {
+ this.url = url;
+ }
+
+ public String getUrl() {
+ return url;
+ }
+
+ public String getBranch() {
+ return branch == null || branch.isBlank() ? DEFAULT_BRANCH : branch;
+ }
+
+ public void setBranch(String branch) {
+ this.branch = branch;
+ }
+
+ public String getUsername() {
+ return username;
+ }
+
+ public void setUsername(String username) {
+ this.username = username;
+ }
+
+ public String getPassword() {
+ return password;
+ }
+
+ public void setPassword(String password) {
+ this.password = password;
+ }
+
+ public Boolean getTrustCerts() {
+ return trustCerts;
+ }
+
+ public void setTrustCerts(Boolean trustCerts) {
+ this.trustCerts = trustCerts;
+ }
+}
diff --git
a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/AbstractCodeLocalizer.java
b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/AbstractCodeLocalizer.java
index ff25f27c..841cf3c0 100644
---
a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/AbstractCodeLocalizer.java
+++
b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/AbstractCodeLocalizer.java
@@ -24,6 +24,7 @@ import
org.apache.submarine.server.api.exception.InvalidSpecException;
import io.kubernetes.client.openapi.models.V1EmptyDirVolumeSource;
import io.kubernetes.client.openapi.models.V1PodSpec;
import io.kubernetes.client.openapi.models.V1Volume;
+import org.apache.submarine.server.api.spec.CodeSpec;
public abstract class AbstractCodeLocalizer implements CodeLocalizer {
@@ -31,18 +32,6 @@ public abstract class AbstractCodeLocalizer implements
CodeLocalizer {
public static final String CODE_LOCALIZER_MOUNT_NAME = "code-dir";
public static final String CODE_LOCALIZER_INIT_CONTAINER_NAME =
"code-localizer";
public static final String CODE_LOCALIZER_PATH_ENV_VAR = "CODE_PATH";
- private String url;
-
- public AbstractCodeLocalizer(String url) {
- this.url = url;
- }
-
- /**
- * @return the url
- */
- public String getUrl() {
- return url;
- }
@Override
public void localize(V1PodSpec podSpec) {
@@ -52,12 +41,13 @@ public abstract class AbstractCodeLocalizer implements
CodeLocalizer {
podSpec.addVolumesItem(volume);
}
- public static CodeLocalizer getCodeLocalizer(String syncMode, String url)
+ public static CodeLocalizer getCodeLocalizer(CodeSpec codeSpec)
throws InvalidSpecException {
- if (syncMode.equals(CodeLocalizerModes.GIT.getMode())) {
- return GitCodeLocalizer.getGitCodeLocalizer(url);
+ CodeLocalizerModes syncMode =
CodeLocalizerModes.valueOfSyncMode(codeSpec.getSyncMode());
+ if (syncMode.equals(CodeLocalizerModes.GIT)) {
+ return GitCodeLocalizer.getGitCodeLocalizer(codeSpec.getGit());
} else {
- return new DummyCodeLocalizer(url);
+ return new DummyCodeLocalizer();
}
}
@@ -74,5 +64,17 @@ public abstract class AbstractCodeLocalizer implements
CodeLocalizer {
public String getMode() {
return this.mode;
}
+
+ /**
+ * Get CodeLocalizerModes by code key
+ */
+ public static CodeLocalizerModes valueOfSyncMode(String key) {
+ for (CodeLocalizerModes clm : values()) {
+ if (clm.mode.equals(key)) {
+ return clm;
+ }
+ }
+ return GIT;
+ }
}
}
diff --git
a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/DummyCodeLocalizer.java
b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/DummyCodeLocalizer.java
index 4f08f27f..e7fbdb44 100644
---
a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/DummyCodeLocalizer.java
+++
b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/DummyCodeLocalizer.java
@@ -23,8 +23,7 @@ import io.kubernetes.client.openapi.models.V1PodSpec;
public class DummyCodeLocalizer extends AbstractCodeLocalizer {
- public DummyCodeLocalizer(String url) {
- super(url);
+ public DummyCodeLocalizer() {
}
@Override
diff --git
a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/GitCodeLocalizer.java
b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/GitCodeLocalizer.java
index bded2c3a..0d818457 100644
---
a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/GitCodeLocalizer.java
+++
b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/GitCodeLocalizer.java
@@ -24,19 +24,31 @@ import java.net.URISyntaxException;
import java.util.ArrayList;
import java.util.List;
-import org.apache.submarine.server.api.exception.InvalidSpecException;
-
import io.kubernetes.client.openapi.models.V1Container;
import io.kubernetes.client.openapi.models.V1EnvVar;
import io.kubernetes.client.openapi.models.V1PodSpec;
import io.kubernetes.client.openapi.models.V1VolumeMount;
+import org.apache.submarine.server.api.exception.InvalidSpecException;
+
+import org.apache.submarine.server.api.spec.code.GitCodeSpec;
+import org.apache.submarine.server.submitter.k8s.util.K8sResourceUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
public abstract class GitCodeLocalizer extends AbstractCodeLocalizer {
+ private static final Logger LOG =
LoggerFactory.getLogger(GitCodeLocalizer.class);
+
public static final String GIT_SYNC_IMAGE =
"apache/submarine:git-sync-3.1.6";
- public GitCodeLocalizer(String url) {
- super(url);
+ private final GitCodeSpec codeSpec;
+
+ public GitCodeLocalizer(GitCodeSpec codeSpec) {
+ this.codeSpec = codeSpec;
+ }
+
+ public GitCodeSpec getCodeSpec() {
+ return codeSpec;
}
public void localize(V1PodSpec podSpec) {
@@ -45,27 +57,36 @@ public abstract class GitCodeLocalizer extends
AbstractCodeLocalizer {
container.setName(CODE_LOCALIZER_INIT_CONTAINER_NAME);
container.setImage(GIT_SYNC_IMAGE);
- V1EnvVar repoEnv = new V1EnvVar();
- repoEnv.setName("GIT_SYNC_REPO");
- repoEnv.setValue(this.getUrl());
-
- V1EnvVar rootEnv = new V1EnvVar();
- rootEnv.setName("GIT_SYNC_ROOT");
- rootEnv.setValue(CODE_LOCALIZER_PATH);
-
- V1EnvVar destEnv = new V1EnvVar();
- destEnv.setName("GIT_SYNC_DEST");
- destEnv.setValue("current");
+ // Add some default git sync envs
+ // The current git environment variables supported by git-syn can be
referred to:
+ // https://github.com/kubernetes/git-sync/blob/v3.1.6/cmd/git-sync/main.go
+ List<V1EnvVar> gitSyncEnvVars = new ArrayList<V1EnvVar>();
- V1EnvVar oneTimeEnv = new V1EnvVar();
- oneTimeEnv.setName("GIT_SYNC_ONE_TIME");
- oneTimeEnv.setValue("true");
+ gitSyncEnvVars.add(K8sResourceUtils.createEnvVar("GIT_SYNC_REPO",
getCodeSpec().getUrl()));
+ gitSyncEnvVars.add(K8sResourceUtils.createEnvVar("GIT_SYNC_ROOT",
CODE_LOCALIZER_PATH));
+ // Our scenario is usually to download the latest code once and execute it,
+ // so we set depth to 1 to prevent the code base from getting too large
+ gitSyncEnvVars.add(K8sResourceUtils.createEnvVar("GIT_SYNC_DEPTH", "1"));
+ gitSyncEnvVars.add(K8sResourceUtils.createEnvVar("GIT_SYNC_DEST",
"current"));
+ // Download first and then exit
+ gitSyncEnvVars.add(K8sResourceUtils.createEnvVar("GIT_SYNC_ONE_TIME",
"true"));
+ // branch
+ gitSyncEnvVars.add(K8sResourceUtils.createEnvVar("GIT_SYNC_BRANCH",
getCodeSpec().getBranch()));
+
+ // Add some optional git sync envs
+ // username
+ if (getCodeSpec().getUsername() != null) {
+ gitSyncEnvVars.add(K8sResourceUtils.createEnvVar("GIT_SYNC_USERNAME",
getCodeSpec().getUsername()));
+ }
+ // password
+ if (getCodeSpec().getPassword() != null) {
+ gitSyncEnvVars.add(K8sResourceUtils.createEnvVar("GIT_SYNC_PASSWORD",
getCodeSpec().getPassword()));
+ }
+ // accept a self-signed certificate host
+ if (getCodeSpec().getTrustCerts() != null &&
getCodeSpec().getTrustCerts()) {
+ gitSyncEnvVars.add(K8sResourceUtils.createEnvVar("GIT_SSL_NO_VERIFY",
"true"));
+ }
- List<V1EnvVar> gitSyncEnvVars = new ArrayList<V1EnvVar>();
- gitSyncEnvVars.add(repoEnv);
- gitSyncEnvVars.add(rootEnv);
- gitSyncEnvVars.add(destEnv);
- gitSyncEnvVars.add(oneTimeEnv);
container.setEnv(gitSyncEnvVars);
V1VolumeMount mount = new V1VolumeMount();
@@ -82,18 +103,24 @@ public abstract class GitCodeLocalizer extends
AbstractCodeLocalizer {
super.localize(podSpec);
}
- public static CodeLocalizer getGitCodeLocalizer(String url)
+ /**
+ * Currently, we mainly support https and ssh.
+ * By default, we use https.
+ */
+ public static CodeLocalizer getGitCodeLocalizer(GitCodeSpec gitCodeSpec)
throws InvalidSpecException {
+ String url = gitCodeSpec.getUrl();
try {
URI uriParser = new URI(url);
String scheme = uriParser.getScheme();
- if (scheme.equals(GitCodeLocalizerModes.HTTP.getMode())) {
- return new HTTPGitCodeLocalizer(url);
- } else if (scheme.equals(GitCodeLocalizerModes.SSH.getMode())) {
- return new SSHGitCodeLocalizer(url);
+ if (scheme.equals(GitCodeLocalizerModes.SSH.getMode())) {
+ return new SSHGitCodeLocalizer(gitCodeSpec);
+ } else if (scheme.equals(GitCodeLocalizerModes.HTTPS.getMode())) {
+ return new HTTPGitCodeLocalizer(gitCodeSpec);
} else {
- return new DummyCodeLocalizer(url);
+ LOG.debug("Unknown url scheme, use https as default localizer.");
+ return new HTTPGitCodeLocalizer(gitCodeSpec);
}
} catch (URISyntaxException e) {
throw new InvalidSpecException(
@@ -103,7 +130,7 @@ public abstract class GitCodeLocalizer extends
AbstractCodeLocalizer {
public enum GitCodeLocalizerModes {
- HTTP("https"), SSH("ssh");
+ HTTPS("https"), SSH("ssh");
private final String mode;
diff --git
a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/HTTPGitCodeLocalizer.java
b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/HTTPGitCodeLocalizer.java
index 472135f2..765f7b44 100644
---
a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/HTTPGitCodeLocalizer.java
+++
b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/HTTPGitCodeLocalizer.java
@@ -20,11 +20,12 @@
package org.apache.submarine.server.submitter.k8s.experiment.codelocalizer;
import io.kubernetes.client.openapi.models.V1PodSpec;
+import org.apache.submarine.server.api.spec.code.GitCodeSpec;
public class HTTPGitCodeLocalizer extends GitCodeLocalizer {
- public HTTPGitCodeLocalizer(String url) {
- super(url);
+ public HTTPGitCodeLocalizer(GitCodeSpec codeSpec) {
+ super(codeSpec);
}
@Override
diff --git
a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/SSHGitCodeLocalizer.java
b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/SSHGitCodeLocalizer.java
index d49ef3e3..7460375d 100644
---
a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/SSHGitCodeLocalizer.java
+++
b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/SSHGitCodeLocalizer.java
@@ -26,6 +26,7 @@ import io.kubernetes.client.openapi.models.V1EnvVar;
import io.kubernetes.client.openapi.models.V1PodSpec;
import io.kubernetes.client.openapi.models.V1SecurityContext;
import io.kubernetes.client.openapi.models.V1VolumeMount;
+import org.apache.submarine.server.api.spec.code.GitCodeSpec;
public class SSHGitCodeLocalizer extends GitCodeLocalizer {
@@ -37,8 +38,8 @@ public class SSHGitCodeLocalizer extends GitCodeLocalizer {
public static final String GIT_SYNC_SSH_NAME = "GIT_SYNC_SSH";
public static final String GIT_SYNC_SSH_VALUE = "true";
- public SSHGitCodeLocalizer(String url) {
- super(url);
+ public SSHGitCodeLocalizer(GitCodeSpec codeSpec) {
+ super(codeSpec);
}
@Override
diff --git
a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/ExperimentSpecParser.java
b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/ExperimentSpecParser.java
index 368ae564..7fd468b3 100644
---
a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/ExperimentSpecParser.java
+++
b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/ExperimentSpecParser.java
@@ -120,8 +120,7 @@ public class ExperimentSpecParser {
*/
if (experimentSpec.getCode() != null) {
CodeLocalizer localizer = AbstractCodeLocalizer.getCodeLocalizer(
- experimentSpec.getCode().getSyncMode(),
- experimentSpec.getCode().getUrl());
+ experimentSpec.getCode());
localizer.localize(podSpec);
if (podSpec.getInitContainers() != null
diff --git
a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/HTTPGitCodeLocalizer.java
b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/util/K8sResourceUtils.java
similarity index 63%
copy from
submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/HTTPGitCodeLocalizer.java
copy to
submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/util/K8sResourceUtils.java
index 472135f2..68b14f68 100644
---
a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/experiment/codelocalizer/HTTPGitCodeLocalizer.java
+++
b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/util/K8sResourceUtils.java
@@ -17,19 +17,21 @@
* under the License.
*/
-package org.apache.submarine.server.submitter.k8s.experiment.codelocalizer;
+package org.apache.submarine.server.submitter.k8s.util;
-import io.kubernetes.client.openapi.models.V1PodSpec;
+import io.kubernetes.client.openapi.models.V1EnvVar;
+import io.kubernetes.client.openapi.models.V1EnvVarBuilder;
-public class HTTPGitCodeLocalizer extends GitCodeLocalizer {
-
- public HTTPGitCodeLocalizer(String url) {
- super(url);
- }
+/**
+ * Provides some common processing methods for k8s resources
+ */
+public class K8sResourceUtils {
- @Override
- public void localize(V1PodSpec podSpec) {
- super.localize(podSpec);
+ /**
+ * Create {@link V1EnvVar} with key and value
+ */
+ public static V1EnvVar createEnvVar(String key, String value) {
+ return new V1EnvVarBuilder().withName(key).withValue(value).build();
}
}
diff --git
a/submarine-server/server-submitter/submitter-k8s/src/test/resources/pytorch_job_req_http_git_code_localizer.json
b/submarine-server/server-submitter/submitter-k8s/src/test/resources/pytorch_job_req_http_git_code_localizer.json
index 408fea0f..af19faee 100644
---
a/submarine-server/server-submitter/submitter-k8s/src/test/resources/pytorch_job_req_http_git_code_localizer.json
+++
b/submarine-server/server-submitter/submitter-k8s/src/test/resources/pytorch_job_req_http_git_code_localizer.json
@@ -25,6 +25,8 @@
},
"code": {
"syncMode": "git",
- "url" : "https://github.com/apache/submarine.git";
+ "git": {
+ "url" : "https://github.com/apache/submarine.git";
+ }
}
}
diff --git
a/submarine-server/server-submitter/submitter-k8s/src/test/resources/pytorch_job_req_ssh_git_code_localizer.json
b/submarine-server/server-submitter/submitter-k8s/src/test/resources/pytorch_job_req_ssh_git_code_localizer.json
index 9afdec4a..a4d6bde0 100644
---
a/submarine-server/server-submitter/submitter-k8s/src/test/resources/pytorch_job_req_ssh_git_code_localizer.json
+++
b/submarine-server/server-submitter/submitter-k8s/src/test/resources/pytorch_job_req_ssh_git_code_localizer.json
@@ -25,6 +25,8 @@
},
"code": {
"syncMode": "git",
- "url" : "ssh://g...@github.com/apache/submarine.git"
+ "git": {
+ "url" : "ssh://g...@github.com/apache/submarine.git"
+ }
}
}
diff --git
a/submarine-test/test-k8s/src/test/resources/tensorflow/tf-mnist-with-http-git-code-localizer-req.json
b/submarine-test/test-k8s/src/test/resources/tensorflow/tf-mnist-with-http-git-code-localizer-req.json
index a47ed17e..fdb08ef7 100644
---
a/submarine-test/test-k8s/src/test/resources/tensorflow/tf-mnist-with-http-git-code-localizer-req.json
+++
b/submarine-test/test-k8s/src/test/resources/tensorflow/tf-mnist-with-http-git-code-localizer-req.json
@@ -23,6 +23,8 @@
},
"code": {
"syncMode": "git",
- "url" : "https://github.com/apache/submarine.git";
+ "git": {
+ "url" : "https://github.com/apache/submarine.git";
+ }
}
}
diff --git
a/submarine-test/test-k8s/src/test/resources/tensorflow/tf-mnist-with-ssh-git-code-localizer-req.json
b/submarine-test/test-k8s/src/test/resources/tensorflow/tf-mnist-with-ssh-git-code-localizer-req.json
index c759f887..69e4142d 100644
---
a/submarine-test/test-k8s/src/test/resources/tensorflow/tf-mnist-with-ssh-git-code-localizer-req.json
+++
b/submarine-test/test-k8s/src/test/resources/tensorflow/tf-mnist-with-ssh-git-code-localizer-req.json
@@ -23,6 +23,8 @@
},
"code": {
"syncMode": "git",
- "url" : "ssh://g...@github.com/apache/submarine.git"
+ "git": {
+ "url" : "ssh://g...@github.com/apache/submarine.git"
+ }
}
}
diff --git
a/submarine-workbench/workbench-web/src/app/interfaces/experiment-spec.ts
b/submarine-workbench/workbench-web/src/app/interfaces/experiment-spec.ts
index 9b2d65c8..5ebf08d1 100644
--- a/submarine-workbench/workbench-web/src/app/interfaces/experiment-spec.ts
+++ b/submarine-workbench/workbench-web/src/app/interfaces/experiment-spec.ts
@@ -46,7 +46,15 @@ export interface Specs {
export interface CodeSpec {
syncMode: string;
+ git: GitCodeSpec;
+}
+
+// Git code spec
+export interface GitCodeSpec {
url: string;
+ branch?: string; // default branch should be main
+ username?: string;
+ password?: string;
}
export interface ExperimentSpec {
diff --git
a/submarine-workbench/workbench-web/src/app/pages/workbench/experiment/experiment-home/experiment-form/experiment-customized-form/experiment-customized-form.component.html
b/submarine-workbench/workbench-web/src/app/pages/workbench/experiment/experiment-home/experiment-form/experiment-customized-form/experiment-customized-form.component.html
index 946be87f..f78a3a01 100644
---
a/submarine-workbench/workbench-web/src/app/pages/workbench/experiment/experiment-home/experiment-form/experiment-customized-form/experiment-customized-form.component.html
+++
b/submarine-workbench/workbench-web/src/app/pages/workbench/experiment/experiment-home/experiment-form/experiment-customized-form/experiment-customized-form.component.html
@@ -27,7 +27,7 @@
<div>
<form [formGroup]="experiment">
<div [ngSwitch]="step" style="margin-top: 30px">
- <div *ngSwitchCase="0" id="firstStep">
+ <div *ngSwitchCase="0" id="firstStep" style="max-height: 450px;
overflow-y: auto" >
<div class="single-field-group">
<label for="experimentName">
<span class="red-star">*</span>
@@ -133,6 +133,43 @@
<i nz-icon nz-tooltip nzTitle="{{'Copy git repo to /code/current
in container.' | translate}}" nzType="info-circle"></i>
</ng-template>
</div>
+ <div *ngIf="ADVANCED" class="single-field-group">
+ <label for="gitBranch">
+ {{'Git branch' | translate}}
+ </label>
+ <input
+ nz-input
+ type="text"
+ name="gitBranch"
+ id="gitBranch"
+ placeholder="main"
+ formControlName="gitBranch"
+ />
+ </div>
+ <div *ngIf="ADVANCED" class="single-field-group">
+ <label for="gitUsername">
+ {{'Git username' | translate}}
+ </label>
+ <input
+ nz-input
+ type="text"
+ name="gitUsername"
+ id="gitUsername"
+ formControlName="gitUsername"
+ />
+ </div>
+ <div *ngIf="ADVANCED" class="single-field-group">
+ <label for="gitPassword">
+ {{'Git password' | translate}}
+ </label>
+ <input
+ nz-input
+ name="gitPassword"
+ type="password"
+ formControlName="gitPassword"
+ id="gitPassword"
+ />
+ </div>
<div *ngIf="ADVANCED">
<ul formArrayName="envs" class="list-container">
<ng-container *ngFor="let env of envs.controls; index as i">
diff --git
a/submarine-workbench/workbench-web/src/app/pages/workbench/experiment/experiment-home/experiment-form/experiment-customized-form/experiment-customized-form.component.ts
b/submarine-workbench/workbench-web/src/app/pages/workbench/experiment/experiment-home/experiment-form/experiment-customized-form/experiment-customized-form.component.ts
index 2fd3da1a..bbacaf60 100644
---
a/submarine-workbench/workbench-web/src/app/pages/workbench/experiment/experiment-home/experiment-form/experiment-customized-form/experiment-customized-form.component.ts
+++
b/submarine-workbench/workbench-web/src/app/pages/workbench/experiment/experiment-home/experiment-form/experiment-customized-form/experiment-customized-form.component.ts
@@ -74,7 +74,6 @@ export class ExperimentCustomizedFormComponent implements
OnInit, OnDestroy {
framework = 'Tensorflow';
currentSpecPage = 1;
-
// About update
@Input() targetId: string = null;
@Input() targetSpec: ExperimentSpec = null;
@@ -95,7 +94,10 @@ export class ExperimentCustomizedFormComponent implements
OnInit, OnDestroy {
image: new FormControl(this.defaultImage, [Validators.required]),
envs: new FormArray([],
[this.experimentValidatorService.nameValidatorFactory('key')]),
specs: new FormArray([],
[this.experimentValidatorService.nameValidatorFactory('name')]),
- gitRepo: new FormControl(null, [])
+ gitRepo: new FormControl(null, []),
+ gitBranch: new FormControl(null, []),
+ gitUsername: new FormControl(null, []),
+ gitPassword: new FormControl(null, [])
});
this.experimentService.fetchExperimentList().subscribe(
@@ -189,6 +191,15 @@ export class ExperimentCustomizedFormComponent implements
OnInit, OnDestroy {
get gitRepo() {
return this.experiment.get('gitRepo');
}
+ get gitBranch() {
+ return this.experiment.get('gitBranch');
+ }
+ get gitGitUsername() {
+ return this.experiment.get('gitUsername');
+ }
+ get gitGitPassword() {
+ return this.experiment.get('gitPassword');
+ }
/**
* Reset properties in parent component when the form is about to closed
@@ -367,7 +378,12 @@ export class ExperimentCustomizedFormComponent implements
OnInit, OnDestroy {
const code: CodeSpec = {
syncMode: 'git',
- url: this.gitRepo.value
+ git: {
+ url: this.gitRepo.value,
+ branch: this.gitBranch.value,
+ username: this.gitGitUsername.value,
+ password: this.gitGitPassword.value
+ }
};
const newExperimentSpec: ExperimentSpec = {
@@ -376,7 +392,7 @@ export class ExperimentCustomizedFormComponent implements
OnInit, OnDestroy {
spec: specs
};
- if (code.url !== null) {
+ if (code.git.url !== null) {
newExperimentSpec.code = code;
}
return newExperimentSpec;
diff --git a/website/docs/api/experiment-template.md
b/website/docs/api/experiment-template.md
index ccb8d89b..d0217d34 100644
--- a/website/docs/api/experiment-template.md
+++ b/website/docs/api/experiment-template.md
@@ -131,10 +131,23 @@ See more details in [environment api](environment.md).
#### **CodeSpec**
-| Field Name | Type | Description |
-| ---------- | ------ | ----------------------- |
-| syncMode | String | sync mode of code spec. |
-| url | String | url of code spec. |
+| Field Name | Type | Description |
+| ---------- | ----------- | ----------------------- |
+| syncMode | String | sync mode of code spec. |
+| url | GitCodeSpec | git code spec. |
+
+#### **GitCodeSpec**
+
+Pull variables supported by the git code.
+
+| Field Name | Type | Description |
+| ---------- | ------- | ---------------------------------------- |
+| url | String | url of git repo. |
+| branch | String | the branch of git repo, default is main. |
+| username | String | the username of git repo. |
+| password | String | the password of git repo. |
+| trustCerts | Boolean | accept a self-signed certificate host. |
+
### Example
<details>
diff --git a/website/docs/api/experiment.md b/website/docs/api/experiment.md
index 00f1ac86..090ced53 100644
--- a/website/docs/api/experiment.md
+++ b/website/docs/api/experiment.md
@@ -77,7 +77,19 @@ Currently only support pulling from github. HDFS, NFS and s3
are in development
| Field Name | Type | Description |
Required |
| ---------- | ----------------------------- | ----------------------- |
:------: |
| syncMode | String \(git\|hdfs\|nfs\|s3\) | sync mode of code spec. | o
|
-| url | String | url of code spec. | o
|
+| url | GitCodeSpec | git code spec. | x
|
+
+#### **GitCodeSpec**
+
+Pull variables supported by the git code.
+
+| Field Name | Type | Description | Required |
+| ---------- | ------- | ---------------------------------------- | :------: |
+| url | String | url of git repo. | o |
+| branch | String | the branch of git repo, default is main. | x |
+| username | String | the username of git repo. | x |
+| password | String | the password of git repo. | x |
+| trustCerts | Boolean | accept a self-signed certificate host. | x |
### Example
<details>
@@ -381,7 +393,7 @@ curl -X GET
http://127.0.0.1:32080/api/v1/experiment/experiment-1647574374688-00
| meta | ExperimentMeta | body | Meta data of the
experiment template. | o |
| environment | EnvironmentSpec | body | Environment of the
experiment template. | o |
| spec | Map<String, ExperimentTaskSpec> | body | Spec of pods.
| o |
-| code | CodeSpec | body | TODO
| x |
+| code | CodeSpec | body | Experiment codespec.
| x |
### Example
<details>
diff --git a/website/docs/gettingStarted/notebook.md
b/website/docs/gettingStarted/notebook.md
index 091b8291..03f5dbda 100644
--- a/website/docs/gettingStarted/notebook.md
+++ b/website/docs/gettingStarted/notebook.md
@@ -94,7 +94,7 @@ worker_spec =
ExperimentTaskSpec(resources='cpu=1,memory=1024M',
replicas=1)
ps_spec = ExperimentTaskSpec(resources='cpu=1,memory=1024M',
replicas=1)
-code_spec = CodeSpec(sync_mode='git',
url='https://github.com/apache/submarine.git')
+code_spec = CodeSpec(sync_mode="git",
git=GitCodeSpec(url="https://github.com/apache/submarine.git";))
experiment_spec = ExperimentSpec(meta=experiment_meta,
environment=environment,
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
