[jira] [Assigned] (SUBMARINE-670) Improvements of Submarine user docs
[ https://issues.apache.org/jira/browse/SUBMARINE-670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan reassigned SUBMARINE-670: Assignee: Wangda Tan > Improvements of Submarine user docs > --- > > Key: SUBMARINE-670 > URL: https://issues.apache.org/jira/browse/SUBMARINE-670 > Project: Apache Submarine > Issue Type: Task > Components: Doc >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > > Found the following issues during test: > * Moved build Docker image from helm chart session to the build from code. > * Moved Submarine Python SDK tutorial from README.md of k8s, to a separate > file. > * Added run experiment from UI doc. > Updated contents of k8s/README.md, and make it points to all correct links. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-670) Improvements of Submarine user docs
[ https://issues.apache.org/jira/browse/SUBMARINE-670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17225033#comment-17225033 ] Wangda Tan commented on SUBMARINE-670: -- Submitted PR: https://github.com/apache/submarine/pull/444 > Improvements of Submarine user docs > --- > > Key: SUBMARINE-670 > URL: https://issues.apache.org/jira/browse/SUBMARINE-670 > Project: Apache Submarine > Issue Type: Task > Components: Doc >Reporter: Wangda Tan >Priority: Major > > Found the following issues during test: > * Moved build Docker image from helm chart session to the build from code. > * Moved Submarine Python SDK tutorial from README.md of k8s, to a separate > file. > * Added run experiment from UI doc. > Updated contents of k8s/README.md, and make it points to all correct links. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Created] (SUBMARINE-670) Improvements of Submarine user docs
Wangda Tan created SUBMARINE-670: Summary: Improvements of Submarine user docs Key: SUBMARINE-670 URL: https://issues.apache.org/jira/browse/SUBMARINE-670 Project: Apache Submarine Issue Type: Task Components: Doc Reporter: Wangda Tan Found the following issues during test: * Moved build Docker image from helm chart session to the build from code. * Moved Submarine Python SDK tutorial from README.md of k8s, to a separate file. * Added run experiment from UI doc. Updated contents of k8s/README.md, and make it points to all correct links. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-477) [Umbrella] Support Jupyter notebook
[ https://issues.apache.org/jira/browse/SUBMARINE-477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199806#comment-17199806 ] Wangda Tan commented on SUBMARINE-477: -- [~lowc1012], I haven't check the code yet, one question is, how notebook connect to Submarine server (via Submarine SDK)? If we ask users to enter URL for submarine server, that may not be a good user experience. Is it possible to set submarine server URL when notebook session is launched by Submarine server? So the Submarine SDK can directly connect to server. Thoughts on this? > [Umbrella] Support Jupyter notebook > --- > > Key: SUBMARINE-477 > URL: https://issues.apache.org/jira/browse/SUBMARINE-477 > Project: Apache Submarine > Issue Type: New Feature > Components: Workbench Notebook >Reporter: Liu Xun >Assignee: Ryan Lo >Priority: Major > > Python code writing by running jupyter container in submarine -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-548) [Umbrella] Predefined Experiment
[ https://issues.apache.org/jira/browse/SUBMARINE-548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170536#comment-17170536 ] Wangda Tan commented on SUBMARINE-548: -- [~pingsutw], Something similar to that, please feel free to add whatever required parameters (such as author). > [Umbrella] Predefined Experiment > > > Key: SUBMARINE-548 > URL: https://issues.apache.org/jira/browse/SUBMARINE-548 > Project: Apache Submarine > Issue Type: New Feature > Components: experiment template >Reporter: JohnTing >Assignee: JohnTing >Priority: Major > Fix For: 0.5.0 > > > Predefined-experiment features > * [API] Define Experiment API for pre-defined template > * [SDK] Add Python SDK to support pre-defined experiment > * [UI] Allow Run pre-defined experiment > * [API] Define Swagger API for pre-defined template submission > * [API] Define Swagger API for pre-defined template registration/delete, etc. > * [Sever] Support submit pre-defined template, and translate it to actual job > [https://github.com/apache/submarine/blob/master/docs/design/experiment-implementation.md#support-predefined-experiment-templates] > [https://cwiki.apache.org/confluence/display/SUBMARINE/Roadmap] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Comment Edited] (SUBMARINE-548) [Umbrella] Predefined Experiment
[
https://issues.apache.org/jira/browse/SUBMARINE-548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17168293#comment-17168293
]
Wangda Tan edited comment on SUBMARINE-548 at 7/30/20, 11:22 PM:
-
[~jotjohnting], thanks for working on this, I just reviewed
[https://github.com/apache/submarine/pull/351]
I think we missed some part in the design:
The design doc:
[https://github.com/apache/submarine/blob/master/docs/design/experiment-implementation.md#predefined-experiment-template-api-to-run-experiment]
defined the spec of how to submit a pre-defined template, which will be
sufficient for submission from CLI/REST/UI. However, it is not enough to
*register/define* a pre-defined template.
The differences between register and submission a pre-defined template are:
* *Register* an experiment-template requires information of how Submarine can
run the experiment, for example, it needs to include: resources required for
worker; environment (docker image, conda kernel); commandline options for
workers/ps, etc.
* In contrast, *submit* an experiment-template only requires filling
required/optional parameters.
So to register a pre-defined template, we need to *not only* include
ExperimentTemplate, but also, we need to tell how Submarine can run it.
*So the predefined template registration should include the following:*
*1) A template of Experiment yaml, for example, if we take an experiment
example from our* doc:
[https://github.com/apache/submarine/blob/master/docs/userdocs/k8s/run-tensorflow-experiment.md]
{code:java}
meta:
name: "tf-mnist-yaml"
namespace: "default"
framework: "TensorFlow"
cmd: "python /var/tf_mnist/mnist_with_summaries.py --log_dir=/train/log
--learning_rate=0.01 --batch_size=150"
envVars:
ENV_1: "ENV1"
environment:
image: "gcr.io/kubeflow-ci/tf-mnist-with-summaries:1.0"
spec:
Ps:
replicas: 1
resources: "cpu=1,memory=1024M"
Worker:
replicas: 1
resources: "cpu=1,memory=1024M" {code}
We can create a template of the YAML (with placeholders) using syntax like:
{code:java}
meta:
name: {{name}}
namespace: "default"
framework: "TensorFlow"
cmd: "python /var/tf_mnist/mnist_with_summaries.py --input {{input}}
--log_dir=/train/log --learning_rate={{training.learning_rate}}
--batch_size={{training.batch_size}}"
envVars:
ENV_1: "ENV1"
environment:
image: "gcr.io/kubeflow-ci/tf-mnist-with-summaries:1.0"
spec:
Ps:
replicas: 1
resources: "cpu=1,memory=1024M"
Worker:
replicas: 1
resources: "cpu=1,memory=1024M" {code}
The above template defined 3 variables (placeholders):
* name
* input
* training.learning_rate.
* training.batch_size
(The above YAML placeholder is based on [https://stackoverflow.com/a/41620747)]
*2) A list of parameters (Similar to ExperimentTemplate)*
*So I think we need the following object:*
*a. RegisterExperimentTemplateSpec*
{code:java}
{
template_name: Name of the template
experiment_spec: the spec for experiment with placeholders.
parameters:
List of parameters definition
} {code}
*a. SubmissionExperimentTemplateSpec*
{code:java}
{
experiment_name: Name of the running experiment
template_name: Name of the template
parameters:
List of parameters (with values)
} {code}
Does this make sense? cc: [~pingsutw], [~ztang] for suggestions.
And if we agree with the proposal, we need to update our experiment spec design
accordingly:
[https://github.com/apache/submarine/blob/master/docs/design/experiment-implementation.md#predefined-experiment-template-api-to-run-experiment]
was (Author: wangda):
[~jotjohnting], thanks for working on this, I just reviewed
[https://github.com/apache/submarine/pull/351]
I think we missed some part in the design:
The design doc:
[https://github.com/apache/submarine/blob/master/docs/design/experiment-implementation.md#predefined-experiment-template-api-to-run-experiment]
defined the spec of how to submit a pre-defined template, which will be
sufficient for submission from CLI/REST/UI. However, it is not enough to
*register/define* a pre-defined template.
The differences between register and submission a pre-defined template are:
* *Register* an experiment-template requires information of how Submarine can
run the experiment, for example, it needs to include: resources required for
worker; environment (docker image, conda kernel); commandline options for
workers/ps, etc.
* In contrast, *submit* an experiment-template only requires filling
required/optional parameters.
So to register a pre-defined template, we need to *not only* include
ExperimentTemplate, but also, we need to tell how Submarine can run it.
*So the predefined template registration should include the following:*
*1) A template of Experiment yaml, for example, if we take an experiment
example from our* doc:
[jira] [Commented] (SUBMARINE-548) [Umbrella] Predefined Experiment
[
https://issues.apache.org/jira/browse/SUBMARINE-548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17168293#comment-17168293
]
Wangda Tan commented on SUBMARINE-548:
--
[~jotjohnting], thanks for working on this, I just reviewed
[https://github.com/apache/submarine/pull/351]
I think we missed some part in the design:
The design doc:
[https://github.com/apache/submarine/blob/master/docs/design/experiment-implementation.md#predefined-experiment-template-api-to-run-experiment]
defined the spec of how to submit a pre-defined template, which will be
sufficient for submission from CLI/REST/UI. However, it is not enough to
*register/define* a pre-defined template.
The differences between register and submission a pre-defined template are:
* *Register* an experiment-template requires information of how Submarine can
run the experiment, for example, it needs to include: resources required for
worker; environment (docker image, conda kernel); commandline options for
workers/ps, etc.
* In contrast, *submit* an experiment-template only requires filling
required/optional parameters.
So to register a pre-defined template, we need to *not only* include
ExperimentTemplate, but also, we need to tell how Submarine can run it.
*So the predefined template registration should include the following:*
*1) A template of Experiment yaml, for example, if we take an experiment
example from our* doc:
[https://github.com/apache/submarine/blob/master/docs/userdocs/k8s/run-tensorflow-experiment.md]
{code:java}
meta:
name: "tf-mnist-yaml"
namespace: "default"
framework: "TensorFlow"
cmd: "python /var/tf_mnist/mnist_with_summaries.py --log_dir=/train/log
--learning_rate=0.01 --batch_size=150"
envVars:
ENV_1: "ENV1"
environment:
image: "gcr.io/kubeflow-ci/tf-mnist-with-summaries:1.0"
spec:
Ps:
replicas: 1
resources: "cpu=1,memory=1024M"
Worker:
replicas: 1
resources: "cpu=1,memory=1024M" {code}
We can create a template of the YAML (with placeholders) using syntax like:
{code:java}
meta:
name: {{name}}
namespace: "default"
framework: "TensorFlow"
cmd: "python /var/tf_mnist/mnist_with_summaries.py --input {{input}}
--log_dir=/train/log --learning_rate={{training.learning_rate}}
--batch_size={{training.batch_size}}"
envVars:
ENV_1: "ENV1"
environment:
image: "gcr.io/kubeflow-ci/tf-mnist-with-summaries:1.0"
spec:
Ps:
replicas: 1
resources: "cpu=1,memory=1024M"
Worker:
replicas: 1
resources: "cpu=1,memory=1024M" {code}
The above template defined 3 variables (placeholders):
* name
* input
* training.learning_rate.
* training.batch_size
(The above YAML placeholder is based on [https://stackoverflow.com/a/41620747)]
*2) A list of parameters (Similar to ExperimentTemplate)*
*So I think we need the following object:*
*a. RegisterExperimentTemplateSpec*
{code:java}
{
template_name: Name of the template
experiment_spec: the spec for experiment with placeholders.
parameters:
List of parameters definition
} {code}
*a. SubmissionExperimentTemplateSpec*
{code:java}
{
experiment_name: Name of the running experiment
template_name: Name of the template
parameters:
List of parameters (with values)
} {code}
Does this make sense? cc: [~pingsutw], [~ztang] for suggestions.
> [Umbrella] Predefined Experiment
>
>
> Key: SUBMARINE-548
> URL: https://issues.apache.org/jira/browse/SUBMARINE-548
> Project: Apache Submarine
> Issue Type: New Feature
> Components: experiment template
>Reporter: JohnTing
>Assignee: JohnTing
>Priority: Major
> Fix For: 0.5.0
>
>
> Predefined-experiment features
> * [API] Define Experiment API for pre-defined template
> * [SDK] Add Python SDK to support pre-defined experiment
> * [UI] Allow Run pre-defined experiment
> * [API] Define Swagger API for pre-defined template submission
> * [API] Define Swagger API for pre-defined template registration/delete, etc.
> * [Sever] Support submit pre-defined template, and translate it to actual job
> [https://github.com/apache/submarine/blob/master/docs/design/experiment-implementation.md#support-predefined-experiment-templates]
> [https://cwiki.apache.org/confluence/display/SUBMARINE/Roadmap]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-566) [WEB] Create a new experiment through UI
[ https://issues.apache.org/jira/browse/SUBMARINE-566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-566: - Status: Patch Available (was: Open) > [WEB] Create a new experiment through UI > > > Key: SUBMARINE-566 > URL: https://issues.apache.org/jira/browse/SUBMARINE-566 > Project: Apache Submarine > Issue Type: Sub-task > Components: Backend Server >Reporter: TsungJui >Assignee: TsungJui >Priority: Major > Labels: pull-request-available > > Implement the real functionality of creating a new experiment through UI > components. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Created] (SUBMARINE-571) Environment management UI
Wangda Tan created SUBMARINE-571: Summary: Environment management UI Key: SUBMARINE-571 URL: https://issues.apache.org/jira/browse/SUBMARINE-571 Project: Apache Submarine Issue Type: New Feature Components: environment, Workbench Web Reporter: Wangda Tan We need UI to management Environment. Umbrella of Environment backend/API implementation can be found at: SUBMARINE-507 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-570) Support run experiment/notebook with synced code
[
https://issues.apache.org/jira/browse/SUBMARINE-570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated SUBMARINE-570:
-
Target Version: 0.5.0
> Support run experiment/notebook with synced code
>
>
> Key: SUBMARINE-570
> URL: https://issues.apache.org/jira/browse/SUBMARINE-570
> Project: Apache Submarine
> Issue Type: New Feature
> Components: code-sync
>Reporter: Wangda Tan
>Priority: Major
>
> In Submarine Experiment, there're needs to sync up code from Git/S3 to run
> experiment.
> For example, Experiment can be defined to:
> {code:java}
> experiment: #Or notebook, model-serving
>name: "abc",
>environment: "team-default-ml-env"
>// Other fields
>code:
>sync_mode: git
>url: "https://foo.com/training-job.git"{code}
> And once the experiment is launched, code from
> [https://foo.com/training-job.git] will be synced under /code directory of
> the launched container.
> Different sync_mode can be supported, it can be git, s3, hdfs, nfs, etc.
> Similarily, notebook can sync code when a notebook session is launched, this
> avoids user manual checkout code post notebook session launch.
> More design can be found at the following design docs:
> -
> [https://github.com/apache/submarine/blob/master/docs/design/storage-implementation.md#store-code-for-experimentnotebookmodel-serving]
> -
> [https://github.com/apache/submarine/blob/master/docs/design/experiment-implementation.md#api-of-experiment]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Created] (SUBMARINE-570) Support run experiment/notebook with synced code
Wangda Tan created SUBMARINE-570:
Summary: Support run experiment/notebook with synced code
Key: SUBMARINE-570
URL: https://issues.apache.org/jira/browse/SUBMARINE-570
Project: Apache Submarine
Issue Type: New Feature
Components: code-sync
Reporter: Wangda Tan
In Submarine Experiment, there're needs to sync up code from Git/S3 to run
experiment.
For example, Experiment can be defined to:
{code:java}
experiment: #Or notebook, model-serving
name: "abc",
environment: "team-default-ml-env"
// Other fields
code:
sync_mode: git
url: "https://foo.com/training-job.git"{code}
And once the experiment is launched, code from
[https://foo.com/training-job.git] will be synced under /code directory of the
launched container.
Different sync_mode can be supported, it can be git, s3, hdfs, nfs, etc.
Similarily, notebook can sync code when a notebook session is launched, this
avoids user manual checkout code post notebook session launch.
More design can be found at the following design docs:
-
[https://github.com/apache/submarine/blob/master/docs/design/storage-implementation.md#store-code-for-experimentnotebookmodel-serving]
-
[https://github.com/apache/submarine/blob/master/docs/design/experiment-implementation.md#api-of-experiment]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-507) [Umbrella] Submarine Environment Management
[ https://issues.apache.org/jira/browse/SUBMARINE-507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-507: - Target Version: 0.5.0 > [Umbrella] Submarine Environment Management > --- > > Key: SUBMARINE-507 > URL: https://issues.apache.org/jira/browse/SUBMARINE-507 > Project: Apache Submarine > Issue Type: New Feature >Reporter: Manikandan R >Assignee: Manikandan R >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > > Scope of this JIRA is to support environment management. It includes the > following: > 1. Create Environment > 2. Update Environment > 3. Delete Environment > 4. List Environments > In addition, this JIRA should also ensures that environments has been > persisted like experiments so that it can used for later use. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-568) [WEB] Fix a display bug in experiment info page
[ https://issues.apache.org/jira/browse/SUBMARINE-568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-568: - Parent Issue: SUBMARINE-569 (was: SUBMARINE-98) > [WEB] Fix a display bug in experiment info page > --- > > Key: SUBMARINE-568 > URL: https://issues.apache.org/jira/browse/SUBMARINE-568 > Project: Apache Submarine > Issue Type: Sub-task > Components: Workbench Web >Reporter: Wen-Chien,Ruan >Assignee: Wen-Chien,Ruan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > > # When reload experiment info page, the experimentID at top of page will miss. > # The presenting of experiment output page, not friendly for users. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-568) [WEB] Fix a display bug in experiment info page
[ https://issues.apache.org/jira/browse/SUBMARINE-568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-568: - Fix Version/s: (was: 0.5.0) > [WEB] Fix a display bug in experiment info page > --- > > Key: SUBMARINE-568 > URL: https://issues.apache.org/jira/browse/SUBMARINE-568 > Project: Apache Submarine > Issue Type: Sub-task > Components: Workbench Web >Reporter: Wen-Chien,Ruan >Assignee: Wen-Chien,Ruan >Priority: Major > Labels: pull-request-available > > # When reload experiment info page, the experimentID at top of page will miss. > # The presenting of experiment output page, not friendly for users. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-568) [WEB] Fix a display bug in experiment info page
[ https://issues.apache.org/jira/browse/SUBMARINE-568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-568: - Target Version: 0.5.0 > [WEB] Fix a display bug in experiment info page > --- > > Key: SUBMARINE-568 > URL: https://issues.apache.org/jira/browse/SUBMARINE-568 > Project: Apache Submarine > Issue Type: Sub-task > Components: Workbench Web >Reporter: Wen-Chien,Ruan >Assignee: Wen-Chien,Ruan >Priority: Major > Labels: pull-request-available > > # When reload experiment info page, the experimentID at top of page will miss. > # The presenting of experiment output page, not friendly for users. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-554) [WEB] UI for Submarine experiment creation
[ https://issues.apache.org/jira/browse/SUBMARINE-554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-554: - Fix Version/s: 0.5.0 > [WEB] UI for Submarine experiment creation > -- > > Key: SUBMARINE-554 > URL: https://issues.apache.org/jira/browse/SUBMARINE-554 > Project: Apache Submarine > Issue Type: Sub-task > Components: Workbench Web >Reporter: TsungJui >Assignee: TsungJui >Priority: Minor > Labels: pull-request-available > Fix For: 0.5.0 > > > In the experiment page, update the + New experiment UI. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-554) [WEB] UI for Submarine experiment creation
[ https://issues.apache.org/jira/browse/SUBMARINE-554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-554: - Parent Issue: SUBMARINE-569 (was: SUBMARINE-98) > [WEB] UI for Submarine experiment creation > -- > > Key: SUBMARINE-554 > URL: https://issues.apache.org/jira/browse/SUBMARINE-554 > Project: Apache Submarine > Issue Type: Sub-task > Components: Workbench Web >Reporter: TsungJui >Assignee: TsungJui >Priority: Minor > Labels: pull-request-available > > In the experiment page, update the + New experiment UI. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-566) [WEB] Create a new experiment through UI
[ https://issues.apache.org/jira/browse/SUBMARINE-566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-566: - Parent Issue: SUBMARINE-569 (was: SUBMARINE-98) > [WEB] Create a new experiment through UI > > > Key: SUBMARINE-566 > URL: https://issues.apache.org/jira/browse/SUBMARINE-566 > Project: Apache Submarine > Issue Type: Sub-task > Components: Backend Server >Reporter: TsungJui >Assignee: TsungJui >Priority: Major > > Implement the real functionality of creating a new experiment through UI > components. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-566) [WEB] Create a new experiment through UI
[ https://issues.apache.org/jira/browse/SUBMARINE-566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-566: - Target Version: 0.5.0 > [WEB] Create a new experiment through UI > > > Key: SUBMARINE-566 > URL: https://issues.apache.org/jira/browse/SUBMARINE-566 > Project: Apache Submarine > Issue Type: Sub-task > Components: Backend Server >Reporter: TsungJui >Assignee: TsungJui >Priority: Major > > Implement the real functionality of creating a new experiment through UI > components. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-552) [WEB] Implement experiment information charts page.
[ https://issues.apache.org/jira/browse/SUBMARINE-552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-552: - Parent Issue: SUBMARINE-569 (was: SUBMARINE-98) > [WEB] Implement experiment information charts page. > --- > > Key: SUBMARINE-552 > URL: https://issues.apache.org/jira/browse/SUBMARINE-552 > Project: Apache Submarine > Issue Type: Sub-task > Components: Workbench >Reporter: Wen-Chien,Ruan >Assignee: Wen-Chien,Ruan >Priority: Major > Labels: pull-request-available > Fix For: 0.4.0 > > > 1.Import ngx-charts module. > 2.Implement experiment information charts page with simulated data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-502) [WEB] Implement outline experiment information page.
[ https://issues.apache.org/jira/browse/SUBMARINE-502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-502: - Parent Issue: SUBMARINE-569 (was: SUBMARINE-98) > [WEB] Implement outline experiment information page. > > > Key: SUBMARINE-502 > URL: https://issues.apache.org/jira/browse/SUBMARINE-502 > Project: Apache Submarine > Issue Type: Sub-task > Components: Workbench >Reporter: HSIN YUAN, CHEN >Assignee: HSIN YUAN, CHEN >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > > Implement outline experiment information page. > Reference: [https://www.comet.ml/pingsutw/testing/view/new] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-522) [WEB] Improve experiment information page.
[ https://issues.apache.org/jira/browse/SUBMARINE-522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-522: - Parent Issue: SUBMARINE-569 (was: SUBMARINE-98) > [WEB] Improve experiment information page. > -- > > Key: SUBMARINE-522 > URL: https://issues.apache.org/jira/browse/SUBMARINE-522 > Project: Apache Submarine > Issue Type: Sub-task > Components: Workbench >Reporter: Wen-Chien,Ruan >Assignee: Wen-Chien,Ruan >Priority: Major > Labels: pull-request-available > Fix For: 0.4.0 > > > Improve experiment information page. > Discussed with [~pingsutw]. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-515) [web]Implement Hyper parameters in experiment page.
[ https://issues.apache.org/jira/browse/SUBMARINE-515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-515: - Parent Issue: SUBMARINE-569 (was: SUBMARINE-98) > [web]Implement Hyper parameters in experiment page. > --- > > Key: SUBMARINE-515 > URL: https://issues.apache.org/jira/browse/SUBMARINE-515 > Project: Apache Submarine > Issue Type: Sub-task > Components: Workbench Web >Reporter: Chia-Jou Kuo >Assignee: Chia-Jou Kuo >Priority: Major > Labels: None, pull-request-available > Fix For: 0.4.0 > > > Implement Hyper parameters in experiment page. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-514) [WEB] Implement experiment information output page.
[ https://issues.apache.org/jira/browse/SUBMARINE-514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-514: - Parent Issue: SUBMARINE-569 (was: SUBMARINE-98) > [WEB] Implement experiment information output page. > > > Key: SUBMARINE-514 > URL: https://issues.apache.org/jira/browse/SUBMARINE-514 > Project: Apache Submarine > Issue Type: Sub-task > Components: Workbench >Reporter: Wen-Chien,Ruan >Assignee: Wen-Chien,Ruan >Priority: Major > Labels: pull-request-available > Fix For: 0.4.0 > > > Show log about experiment. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-495) [WEB] Submarine experiment list/delete in workbench with Angular
[ https://issues.apache.org/jira/browse/SUBMARINE-495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-495: - Parent Issue: SUBMARINE-569 (was: SUBMARINE-98) > [WEB] Submarine experiment list/delete in workbench with Angular > - > > Key: SUBMARINE-495 > URL: https://issues.apache.org/jira/browse/SUBMARINE-495 > Project: Apache Submarine > Issue Type: Sub-task > Components: Workbench >Reporter: Kevin Su >Assignee: Kevin Su >Priority: Minor > Fix For: 0.4.0 > > > Implement list/delete experiment in workbench with Angular. > will start working on submit/edit, after submarine experiment spec refactor. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-569) [Umbrella] Submarine Experiment UI
[ https://issues.apache.org/jira/browse/SUBMARINE-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-569: - Target Version: 0.5.0 > [Umbrella] Submarine Experiment UI > -- > > Key: SUBMARINE-569 > URL: https://issues.apache.org/jira/browse/SUBMARINE-569 > Project: Apache Submarine > Issue Type: New Feature > Components: experiment, experiment template, Workbench Web >Reporter: Wangda Tan >Priority: Major > > Umbrella for all experiment UI related development works. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Created] (SUBMARINE-569) [Umbrella] Submarine Experiment UI
Wangda Tan created SUBMARINE-569: Summary: [Umbrella] Submarine Experiment UI Key: SUBMARINE-569 URL: https://issues.apache.org/jira/browse/SUBMARINE-569 Project: Apache Submarine Issue Type: New Feature Components: experiment, experiment template, Workbench Web Reporter: Wangda Tan Umbrella for all experiment UI related development works. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-559) [API] Define Swagger API for pre-defined template registration/delete, etc.
[ https://issues.apache.org/jira/browse/SUBMARINE-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17160773#comment-17160773 ] Wangda Tan commented on SUBMARINE-559: -- [~jotjohnting], the "template" word is general and overloaded, I suggest to use "experiment-template" in all the APIs. > [API] Define Swagger API for pre-defined template registration/delete, etc. > --- > > Key: SUBMARINE-559 > URL: https://issues.apache.org/jira/browse/SUBMARINE-559 > Project: Apache Submarine > Issue Type: Sub-task > Components: experiment template >Reporter: JohnTing >Assignee: JohnTing >Priority: Major > Fix For: 0.5.0 > > > Make basic rest api for management template. > get, post, del, patch > [http://localhost/V1/template] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-507) [Umbrella] Submarine Environment Management
[ https://issues.apache.org/jira/browse/SUBMARINE-507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-507: - Fix Version/s: 0.5.0 > [Umbrella] Submarine Environment Management > --- > > Key: SUBMARINE-507 > URL: https://issues.apache.org/jira/browse/SUBMARINE-507 > Project: Apache Submarine > Issue Type: New Feature >Reporter: Manikandan R >Assignee: Manikandan R >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > > Scope of this JIRA is to support environment management. It includes the > following: > 1. Create Environment > 2. Update Environment > 3. Delete Environment > 4. List Environments > In addition, this JIRA should also ensures that environments has been > persisted like experiments so that it can used for later use. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-507) [Umbrella] Submarine Environment Management
[ https://issues.apache.org/jira/browse/SUBMARINE-507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-507: - Target Version: (was: 0.5.0) > [Umbrella] Submarine Environment Management > --- > > Key: SUBMARINE-507 > URL: https://issues.apache.org/jira/browse/SUBMARINE-507 > Project: Apache Submarine > Issue Type: New Feature >Reporter: Manikandan R >Assignee: Manikandan R >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > > Scope of this JIRA is to support environment management. It includes the > following: > 1. Create Environment > 2. Update Environment > 3. Delete Environment > 4. List Environments > In addition, this JIRA should also ensures that environments has been > persisted like experiments so that it can used for later use. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-548) [Umbrella] Predefined Experiment
[ https://issues.apache.org/jira/browse/SUBMARINE-548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-548: - Component/s: (was: K8s Submitter) experiment template > [Umbrella] Predefined Experiment > > > Key: SUBMARINE-548 > URL: https://issues.apache.org/jira/browse/SUBMARINE-548 > Project: Apache Submarine > Issue Type: New Feature > Components: experiment template >Reporter: JohnTing >Assignee: JohnTing >Priority: Major > Fix For: 0.5.0 > > > Predefined-experiment features > * [API] Define Experiment API for pre-defined template > * [SDK] Add Python SDK to support pre-defined experiment > * [UI] Allow Run pre-defined experiment > * [API] Define Swagger API for pre-defined template submission > * [API] Define Swagger API for pre-defined template registration/delete, etc. > * [Sever] Support submit pre-defined template, and translate it to actual job > [https://github.com/apache/submarine/blob/master/docs/design/experiment-implementation.md#support-predefined-experiment-templates] > [https://cwiki.apache.org/confluence/display/SUBMARINE/Roadmap] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Assigned] (SUBMARINE-529) Change apiVersion
[ https://issues.apache.org/jira/browse/SUBMARINE-529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan reassigned SUBMARINE-529: Assignee: TsungJui > Change apiVersion > - > > Key: SUBMARINE-529 > URL: https://issues.apache.org/jira/browse/SUBMARINE-529 > Project: Apache Submarine > Issue Type: Improvement > Components: Database >Reporter: TsungJui >Assignee: TsungJui >Priority: Minor > Labels: pull-request-available > Fix For: 0.4.0 > > > k8s installed by kind 0.8.0 seems that it does not support apiVersion > apps/v1beta1 anymore. > Change to apps/v1 to build successfully. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-546) Update Submarine Website for 0.4.0 release
[ https://issues.apache.org/jira/browse/SUBMARINE-546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17144568#comment-17144568 ] Wangda Tan commented on SUBMARINE-546: -- [~liuxun], [~jiwq], [~pingsutw] can you help to take care of it? I previously have trouble to edit website contents. We should update them before 0.4.0 release, it should be relatively simple changes. > Update Submarine Website for 0.4.0 release > -- > > Key: SUBMARINE-546 > URL: https://issues.apache.org/jira/browse/SUBMARINE-546 > Project: Apache Submarine > Issue Type: Task > Components: Doc >Reporter: Wangda Tan >Priority: Blocker > > Several things we need update before 0.4.0 release: > * (Blocker): "Get Started" Link is broken. I suggest to update it to "User > Doc", and link to > [https://github.com/apache/submarine/blob/master/docs/user-guide-home.md] > * (Blocker): Kubernetes (In-Progress), we need to update the text saying it > is supported. > * (Critical): It is not easy to find link to Github repo (Now it is under > "Community"), can we make it easier to find? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-546) Update Submarine Website for 0.4.0 release
[ https://issues.apache.org/jira/browse/SUBMARINE-546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-546: - Description: Several things we need update before 0.4.0 release: * (Blocker): "Get Started" Link is broken. I suggest to update it to "User Doc", and link to [https://github.com/apache/submarine/blob/master/docs/user-guide-home.md] * (Blocker): Kubernetes (In-Progress), we need to update the text saying it is supported. * (Critical): It is not easy to find link to Github repo (Now it is under "Community"), can we make it easier to find? was: Several things we need update before 0.4.0 release: * (Blocker): "Get Started" Link is broken. * (Blocker): Kubernetes (In-Progress), we need to update the text saying it is supported. * (Critical): It is not easy to find link to Github repo (Now it is under "Community"), can we make it easier to find? > Update Submarine Website for 0.4.0 release > -- > > Key: SUBMARINE-546 > URL: https://issues.apache.org/jira/browse/SUBMARINE-546 > Project: Apache Submarine > Issue Type: Task > Components: Doc >Reporter: Wangda Tan >Priority: Blocker > > Several things we need update before 0.4.0 release: > * (Blocker): "Get Started" Link is broken. I suggest to update it to "User > Doc", and link to > [https://github.com/apache/submarine/blob/master/docs/user-guide-home.md] > * (Blocker): Kubernetes (In-Progress), we need to update the text saying it > is supported. > * (Critical): It is not easy to find link to Github repo (Now it is under > "Community"), can we make it easier to find? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Created] (SUBMARINE-546) Update Submarine Website for 0.4.0 release
Wangda Tan created SUBMARINE-546: Summary: Update Submarine Website for 0.4.0 release Key: SUBMARINE-546 URL: https://issues.apache.org/jira/browse/SUBMARINE-546 Project: Apache Submarine Issue Type: Task Components: Doc Reporter: Wangda Tan Several things we need update before 0.4.0 release: * (Blocker): "Get Started" Link is broken. * (Blocker): Kubernetes (In-Progress), we need to update the text saying it is supported. * (Critical): It is not easy to find link to Github repo (Now it is under "Community"), can we make it easier to find? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Created] (SUBMARINE-545) Additional Changes to User Doc (K8s)
Wangda Tan created SUBMARINE-545: Summary: Additional Changes to User Doc (K8s) Key: SUBMARINE-545 URL: https://issues.apache.org/jira/browse/SUBMARINE-545 Project: Apache Submarine Issue Type: Bug Components: Doc Reporter: Wangda Tan Additional changes to user doc, which includes: * KIND. * How it is organized, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Assigned] (SUBMARINE-545) Additional Changes to User Doc (K8s)
[ https://issues.apache.org/jira/browse/SUBMARINE-545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan reassigned SUBMARINE-545: Assignee: Wangda Tan > Additional Changes to User Doc (K8s) > > > Key: SUBMARINE-545 > URL: https://issues.apache.org/jira/browse/SUBMARINE-545 > Project: Apache Submarine > Issue Type: Task > Components: Doc >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > > Additional changes to user doc, which includes: > * KIND. > * How it is organized, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-545) Additional Changes to User Doc (K8s)
[ https://issues.apache.org/jira/browse/SUBMARINE-545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-545: - Issue Type: Task (was: Bug) > Additional Changes to User Doc (K8s) > > > Key: SUBMARINE-545 > URL: https://issues.apache.org/jira/browse/SUBMARINE-545 > Project: Apache Submarine > Issue Type: Task > Components: Doc >Reporter: Wangda Tan >Priority: Blocker > > Additional changes to user doc, which includes: > * KIND. > * How it is organized, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-542) [SDK] get_log error when experiment is not started
[
https://issues.apache.org/jira/browse/SUBMARINE-542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17141692#comment-17141692
]
Wangda Tan commented on SUBMARINE-542:
--
Last 100 lines of the submarine server:
{code:java}
K8s submitter: parse Job object failed by ConflictK8s submitter: parse Job
object failed by Conflictio.kubernetes.client.ApiException: Conflict at
io.kubernetes.client.ApiClient.handleResponse(ApiClient.java:882) at
io.kubernetes.client.ApiClient.execute(ApiClient.java:798) at
io.kubernetes.client.apis.CustomObjectsApi.createNamespacedCustomObjectWithHttpInfo(CustomObjectsApi.java:349)
at
io.kubernetes.client.apis.CustomObjectsApi.createNamespacedCustomObject(CustomObjectsApi.java:330)
at
org.apache.submarine.server.submitter.k8s.K8sSubmitter.createExperiment(K8sSubmitter.java:102)
at
org.apache.submarine.server.experiment.ExperimentManager.createExperiment(ExperimentManager.java:88)
at
org.apache.submarine.server.rest.ExperimentRestApi.createExperiment(ExperimentRestApi.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
at
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at
org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at
org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at
org.glassfish.jersey.internal.Errors.process(Errors.java:316) at
org.glassfish.jersey.internal.Errors.process(Errors.java:298) at
org.glassfish.jersey.internal.Errors.process(Errors.java:268) at
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
at
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
at
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at
org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873) at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542) at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1700)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1667)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:61) at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:505) at
[jira] [Updated] (SUBMARINE-542) [SDK] get_log error when experiment is pulling docker images
[
https://issues.apache.org/jira/browse/SUBMARINE-542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated SUBMARINE-542:
-
Fix Version/s: (was: 0.4.0)
> [SDK] get_log error when experiment is pulling docker images
>
>
> Key: SUBMARINE-542
> URL: https://issues.apache.org/jira/browse/SUBMARINE-542
> Project: Apache Submarine
> Issue Type: Bug
> Components: SDK
>Reporter: Kevin Su
>Assignee: Kevin Su
>Priority: Major
>
> >>> submarine_client.get_log(id)
> {code:java}
> IndexError Traceback (most recent call last)
> in
> > 1
> submarine_client.get_log(id)~/opt/anaconda3/envs/python3-7/lib/python3.7/site-packages/submarine/experiment/api/experiment_client.py
> in get_log(self, id, master)
> 124
> 125 if master is True:
> --> 126 log_contents = [log_contents[0]]
> 127
> 128 for log_content in log_contents:IndexError: list index out of
> range{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-542) [SDK] get_log error when experiment is pulling docker images
[
https://issues.apache.org/jira/browse/SUBMARINE-542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated SUBMARINE-542:
-
Priority: Blocker (was: Major)
> [SDK] get_log error when experiment is pulling docker images
>
>
> Key: SUBMARINE-542
> URL: https://issues.apache.org/jira/browse/SUBMARINE-542
> Project: Apache Submarine
> Issue Type: Bug
> Components: SDK
>Reporter: Kevin Su
>Assignee: Kevin Su
>Priority: Blocker
>
> >>> submarine_client.get_log(id)
> {code:java}
> IndexError Traceback (most recent call last)
> in
> > 1
> submarine_client.get_log(id)~/opt/anaconda3/envs/python3-7/lib/python3.7/site-packages/submarine/experiment/api/experiment_client.py
> in get_log(self, id, master)
> 124
> 125 if master is True:
> --> 126 log_contents = [log_contents[0]]
> 127
> 128 for log_content in log_contents:IndexError: list index out of
> range{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-543) Issues of Submarine KIND version
[
https://issues.apache.org/jira/browse/SUBMARINE-543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17141677#comment-17141677
]
Wangda Tan commented on SUBMARINE-543:
--
[~jiwq], [~ztang], [~liuxun323]
> Issues of Submarine KIND version
>
>
> Key: SUBMARINE-543
> URL: https://issues.apache.org/jira/browse/SUBMARINE-543
> Project: Apache Submarine
> Issue Type: Bug
> Components: Doc
>Reporter: Wangda Tan
>Priority: Major
>
> I'm trying to follow Submarine doc to install a cluster, I found the default
> version 1.15.6 which is mentioned by:
> [https://github.com/tangzhankun/Aapche-Submarine/blob/SUBMARINE-528/docs/userdocs/k8s/kind.md]
> Has issues, core DNS pod will fail after start:
> {code:java}
> ~ kubectl get pods -n kube-system --context kind-submarine
> NAME READY STATUS RESTARTS
> AGE
> coredns-5c98db65d4-8n42p 0/1 CrashLoopBackOff 2
> 53s
> coredns-5c98db65d4-j8m6t 0/1 CrashLoopBackOff 2
> 53s {code}
> I found KIND community reported very similar issue:
> [https://github.com/kubernetes-sigs/kind/issues/1659]
> I switched to
> kindest/node:v1.14.10@sha256:6cd43ff41ae9f02bb46c8f455d5323819aec858b99534a290517ebc181b443c6
> and now it works.
> Full command to create cluster
> {code:java}
> kind create cluster --image
> kindest/node:v1.14.10@sha256:6cd43ff41ae9f02bb46c8f455d5323819aec858b99534a290517ebc181b443c6
> --name submarine {code}
> How to test it (I follow the calico guide:
> [https://docs.projectcalico.org/getting-started/kubernetes/hardway/test-networking])
> {code:java}
> kubectl run pingtest --image=busybox --replicas=1 -- sleep infinity
> kubectl get pod -l run=pingtest -o wide
> kubectl exec -ti pingtest- sh
> And ping www.google.com (or other external website should success){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-543) Issues of Submarine KIND version
[
https://issues.apache.org/jira/browse/SUBMARINE-543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated SUBMARINE-543:
-
Priority: Blocker (was: Major)
> Issues of Submarine KIND version
>
>
> Key: SUBMARINE-543
> URL: https://issues.apache.org/jira/browse/SUBMARINE-543
> Project: Apache Submarine
> Issue Type: Bug
> Components: Doc
>Reporter: Wangda Tan
>Priority: Blocker
>
> I'm trying to follow Submarine doc to install a cluster, I found the default
> version 1.15.6 which is mentioned by:
> [https://github.com/tangzhankun/Aapche-Submarine/blob/SUBMARINE-528/docs/userdocs/k8s/kind.md]
> Has issues, core DNS pod will fail after start:
> {code:java}
> ~ kubectl get pods -n kube-system --context kind-submarine
> NAME READY STATUS RESTARTS
> AGE
> coredns-5c98db65d4-8n42p 0/1 CrashLoopBackOff 2
> 53s
> coredns-5c98db65d4-j8m6t 0/1 CrashLoopBackOff 2
> 53s {code}
> I found KIND community reported very similar issue:
> [https://github.com/kubernetes-sigs/kind/issues/1659]
> I switched to
> kindest/node:v1.14.10@sha256:6cd43ff41ae9f02bb46c8f455d5323819aec858b99534a290517ebc181b443c6
> and now it works.
> Full command to create cluster
> {code:java}
> kind create cluster --image
> kindest/node:v1.14.10@sha256:6cd43ff41ae9f02bb46c8f455d5323819aec858b99534a290517ebc181b443c6
> --name submarine {code}
> How to test it (I follow the calico guide:
> [https://docs.projectcalico.org/getting-started/kubernetes/hardway/test-networking])
> {code:java}
> kubectl run pingtest --image=busybox --replicas=1 -- sleep infinity
> kubectl get pod -l run=pingtest -o wide
> kubectl exec -ti pingtest- sh
> And ping www.google.com (or other external website should success){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-543) Issues of Submarine KIND version
[
https://issues.apache.org/jira/browse/SUBMARINE-543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated SUBMARINE-543:
-
Description:
I'm trying to follow Submarine doc to install a cluster, I found the default
version 1.15.6 which is mentioned by:
[https://github.com/tangzhankun/Aapche-Submarine/blob/SUBMARINE-528/docs/userdocs/k8s/kind.md]
Has issues, core DNS pod will fail after start:
{code:java}
~ kubectl get pods -n kube-system --context kind-submarine
NAME READY STATUS RESTARTS
AGE
coredns-5c98db65d4-8n42p 0/1 CrashLoopBackOff 2
53s
coredns-5c98db65d4-j8m6t 0/1 CrashLoopBackOff 2
53s {code}
I found KIND community reported very similar issue:
[https://github.com/kubernetes-sigs/kind/issues/1659]
I switched to
kindest/node:v1.14.10@sha256:6cd43ff41ae9f02bb46c8f455d5323819aec858b99534a290517ebc181b443c6
and now it works.
Full command to create cluster
{code:java}
kind create cluster --image
kindest/node:v1.14.10@sha256:6cd43ff41ae9f02bb46c8f455d5323819aec858b99534a290517ebc181b443c6
--name submarine {code}
How to test it (I follow the calico guide:
[https://docs.projectcalico.org/getting-started/kubernetes/hardway/test-networking])
{code:java}
kubectl run pingtest --image=busybox --replicas=1 -- sleep infinity
kubectl get pod -l run=pingtest -o wide
kubectl exec -ti pingtest- sh
And ping www.google.com (or other external website should success){code}
was:
I'm trying to follow Submarine doc to install a cluster, I found the default
version 1.15.6 which is mentioned by:
[https://github.com/tangzhankun/Aapche-Submarine/blob/SUBMARINE-528/docs/userdocs/k8s/kind.md]
Has issues, core DNS pod will fail after start:
{code:java}
~ kubectl get pods -n kube-system --context kind-submarine
NAME READY STATUS RESTARTS
AGE
coredns-5c98db65d4-8n42p 0/1 CrashLoopBackOff 2
53s
coredns-5c98db65d4-j8m6t 0/1 CrashLoopBackOff 2
53s {code}
I found KIND community reported very similar issue:
[https://github.com/kubernetes-sigs/kind/issues/1659]
I switched to
kindest/node:v1.14.10@sha256:6cd43ff41ae9f02bb46c8f455d5323819aec858b99534a290517ebc181b443c6
and now it works.
Full command to create cluster
{code:java}
kind create cluster --image
kindest/node:v1.14.10@sha256:6cd43ff41ae9f02bb46c8f455d5323819aec858b99534a290517ebc181b443c6
--name submarine {code}
> Issues of Submarine KIND version
>
>
> Key: SUBMARINE-543
> URL: https://issues.apache.org/jira/browse/SUBMARINE-543
> Project: Apache Submarine
> Issue Type: Bug
> Components: Doc
>Reporter: Wangda Tan
>Priority: Major
>
> I'm trying to follow Submarine doc to install a cluster, I found the default
> version 1.15.6 which is mentioned by:
> [https://github.com/tangzhankun/Aapche-Submarine/blob/SUBMARINE-528/docs/userdocs/k8s/kind.md]
> Has issues, core DNS pod will fail after start:
> {code:java}
> ~ kubectl get pods -n kube-system --context kind-submarine
> NAME READY STATUS RESTARTS
> AGE
> coredns-5c98db65d4-8n42p 0/1 CrashLoopBackOff 2
> 53s
> coredns-5c98db65d4-j8m6t 0/1 CrashLoopBackOff 2
> 53s {code}
> I found KIND community reported very similar issue:
> [https://github.com/kubernetes-sigs/kind/issues/1659]
> I switched to
> kindest/node:v1.14.10@sha256:6cd43ff41ae9f02bb46c8f455d5323819aec858b99534a290517ebc181b443c6
> and now it works.
> Full command to create cluster
> {code:java}
> kind create cluster --image
> kindest/node:v1.14.10@sha256:6cd43ff41ae9f02bb46c8f455d5323819aec858b99534a290517ebc181b443c6
> --name submarine {code}
> How to test it (I follow the calico guide:
> [https://docs.projectcalico.org/getting-started/kubernetes/hardway/test-networking])
> {code:java}
> kubectl run pingtest --image=busybox --replicas=1 -- sleep infinity
> kubectl get pod -l run=pingtest -o wide
> kubectl exec -ti pingtest- sh
> And ping www.google.com (or other external website should success){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Created] (SUBMARINE-543) Issues of Submarine KIND version
Wangda Tan created SUBMARINE-543:
Summary: Issues of Submarine KIND version
Key: SUBMARINE-543
URL: https://issues.apache.org/jira/browse/SUBMARINE-543
Project: Apache Submarine
Issue Type: Bug
Components: Doc
Reporter: Wangda Tan
I'm trying to follow Submarine doc to install a cluster, I found the default
version 1.15.6 which is mentioned by:
[https://github.com/tangzhankun/Aapche-Submarine/blob/SUBMARINE-528/docs/userdocs/k8s/kind.md]
Has issues, core DNS pod will fail after start:
{code:java}
~ kubectl get pods -n kube-system --context kind-submarine
NAME READY STATUS RESTARTS
AGE
coredns-5c98db65d4-8n42p 0/1 CrashLoopBackOff 2
53s
coredns-5c98db65d4-j8m6t 0/1 CrashLoopBackOff 2
53s {code}
I found KIND community reported very similar issue:
[https://github.com/kubernetes-sigs/kind/issues/1659]
I switched to
kindest/node:v1.14.10@sha256:6cd43ff41ae9f02bb46c8f455d5323819aec858b99534a290517ebc181b443c6
and now it works.
Full command to create cluster
{code:java}
kind create cluster --image
kindest/node:v1.14.10@sha256:6cd43ff41ae9f02bb46c8f455d5323819aec858b99534a290517ebc181b443c6
--name submarine {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-296) [SDK] Submarine pipeline example
[ https://issues.apache.org/jira/browse/SUBMARINE-296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-296: - Fix Version/s: 0.4.0 > [SDK] Submarine pipeline example > - > > Key: SUBMARINE-296 > URL: https://issues.apache.org/jira/browse/SUBMARINE-296 > Project: Apache Submarine > Issue Type: Sub-task > Components: SDK >Reporter: Kevin Su >Assignee: Kevin Su >Priority: Major > Labels: pull-request-available > Fix For: 0.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > quick start deep-learning based CTR example of submarine pipeline -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-525) [doc] Model training example on k8s
[ https://issues.apache.org/jira/browse/SUBMARINE-525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-525: - Priority: Critical (was: Minor) > [doc] Model training example on k8s > --- > > Key: SUBMARINE-525 > URL: https://issues.apache.org/jira/browse/SUBMARINE-525 > Project: Apache Submarine > Issue Type: Improvement > Components: Doc >Reporter: Kevin Su >Assignee: Kevin Su >Priority: Critical > Fix For: 0.4.0 > > > improve model training on k8s doc and add PyTorch training example > related to > [https://github.com/apache/submarine/blob/master/docs/userdocs/k8s/README.md] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-525) [doc] Model training example on k8s
[ https://issues.apache.org/jira/browse/SUBMARINE-525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-525: - Fix Version/s: (was: 0.4.0) > [doc] Model training example on k8s > --- > > Key: SUBMARINE-525 > URL: https://issues.apache.org/jira/browse/SUBMARINE-525 > Project: Apache Submarine > Issue Type: Improvement > Components: Doc >Reporter: Kevin Su >Assignee: Kevin Su >Priority: Critical > > improve model training on k8s doc and add PyTorch training example > related to > [https://github.com/apache/submarine/blob/master/docs/userdocs/k8s/README.md] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-507) Submarine Environment Management
[
https://issues.apache.org/jira/browse/SUBMARINE-507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17121444#comment-17121444
]
Wangda Tan commented on SUBMARINE-507:
--
[~ztang], good point, here's my thoughts:
{quote}Will the kernel spec has hard dependencies on the conda component
versions when enabling?
{quote}
Based on the Anaconda's typical usage, version of Python is part of the kernel,
so in that case. Python inside the image will be ignored. (Anaconda uses
environment variables to only pass whatever libraries when the kernel is
activated).
{quote}[~wangda], for the kernel spec, is the sample spec generated by "condo
export"? It seems hard to write that by hand.
{quote}
That's correct, ideally user should not write the spec. Once a kernel is
activated, all the activities like {{pip install}} will be tracked and when
user call {{conda export}} such changes will be part of the exported conda spec.
> Submarine Environment Management
>
>
> Key: SUBMARINE-507
> URL: https://issues.apache.org/jira/browse/SUBMARINE-507
> Project: Apache Submarine
> Issue Type: New Feature
>Reporter: Manikandan R
>Assignee: Manikandan R
>Priority: Major
> Labels: pull-request-available
>
> Scope of this JIRA is to support environment management. It includes the
> following:
> 1. Create Environment
> 2. Update Environment
> 3. Delete Environment
> 4. List Environments
> In addition, this JIRA should also ensures that environments has been
> persisted like experiments so that it can used for later use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-507) Submarine Environment Management
[
https://issues.apache.org/jira/browse/SUBMARINE-507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120592#comment-17120592
]
Wangda Tan commented on SUBMARINE-507:
--
[~maniraj...@gmail.com],
{quote} # How do we decide that we need to talk to Docker hub's
"apache/submarine" or end-user's existing Docker registry? Using docker image
name?{quote}
Yes, it should based on the image name and Handled by Docker daemon.
{quote}2. For both the categories, Can we safely assume images will be always
available for use at run time? Nothing to worry about this while creating
environment?
{quote}
Yes, if it has any issue (like image cannot be pulled), the experiment run will
fail (naturally).
{quote}3. For first category of base images ("apache/submarine"),When & How are
we going to create images and release to "apache/submarine"? Using docker file
manually? Admin would be doing this?
{quote}
This is part of the Submarine release process, I believe we already doing this,
cc:[~ztang].
{quote}4. For second category of base images, Can we assume end users would
have created the image in their docker registry?
{quote}
Yes, for whatever base image specified, user need to ensure it can be pulled.
{quote}5. It seems we are in favour of storing environment spec as is in a
string based column in "environments" table?
{quote}
It looks like a good solution to me, I'm not sure if there's any other
alternatives.
{quote}6. conda env activation happens at run time while running the notebook
or experiment. When are we going to create an env in conda?
{quote}
Yes, activation only happens at run time. To create an env, I think we can have
following Python APIs:
1) Create Environment using SDK (pseudo code)
{code:java}
env = create_new_environment("my_env")
# Set Docker image
env.set_docker_image("apache/submarine-123:123")
# Set conda kernel using spec
env.add_conda_kernel(conda_kernel_from_spec("""
name: team_default_python_3.7
channels:
- defaults
dependencies:
- _ipyw_jlab_nb_ext_conf=0.1.0=py37_0
- alabaster=0.7.12=py37_0
- anaconda=2020.02=py37_0
- anaconda-client=1.7.2=py37_0
- anaconda-navigator=1.9.12=py37_0
""")
# Alternatively, set conda kernel using APIs
conda_kernel = create_new_conda_kernel("my_kernel")
conda_kernel.set_channels(["..."])
conda_kernel.add_pip_dependency("...")
# Finally, save the env, this will save to metadata
submarine.register_env(env)
{code}
2) Similarily, we need List API, Get API, Delete API for environment.
Thoughts?
> Submarine Environment Management
>
>
> Key: SUBMARINE-507
> URL: https://issues.apache.org/jira/browse/SUBMARINE-507
> Project: Apache Submarine
> Issue Type: New Feature
>Reporter: Manikandan R
>Assignee: Manikandan R
>Priority: Major
> Labels: pull-request-available
>
> Scope of this JIRA is to support environment management. It includes the
> following:
> 1. Create Environment
> 2. Update Environment
> 3. Delete Environment
> 4. List Environments
> In addition, this JIRA should also ensures that environments has been
> persisted like experiments so that it can used for later use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-519) Overhaul Structure of Submarine Github Markdown Files
[ https://issues.apache.org/jira/browse/SUBMARINE-519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120043#comment-17120043 ] Wangda Tan commented on SUBMARINE-519: -- Added WIP PR, [~ztang], [~liuxun323], [~pingsutw], [~jiwq], can you help with patch review and give some feedback when you get a chance? > Overhaul Structure of Submarine Github Markdown Files > - > > Key: SUBMARINE-519 > URL: https://issues.apache.org/jira/browse/SUBMARINE-519 > Project: Apache Submarine > Issue Type: Sub-task > Components: Doc >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > > Targets: > 1) Cleanup obsolete contents. > 2) Update vision to the latest vision. > 3) Make a clear boundary between the developer doc and user doc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-507) Submarine Environment Management
[
https://issues.apache.org/jira/browse/SUBMARINE-507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119816#comment-17119816
]
Wangda Tan commented on SUBMARINE-507:
--
Thanks [~maniraj...@gmail.com], [~ztang] for the comments,
For the environment, I don't think we should get into image storage / Docker
registry setup business, it should just be a Docker image name and K8s can
figure out what credentials to pull the docker image based on secrets stored in
the namespace. And users are responsible for setting up Docker registry and K8s
secrets.
{quote}For base image storage, one simple way, to begin with, might be storing
the submarine base image or release them in Docker hub's "apache/submarine".
And build an entry script to activate the conda spec at runtime.
{quote}
This make sense to me, and this is just an example, user can choose whatever
base Docker image they want to use, it could borrow from the apache/submarine's
base image.
{quote}For end-user custom base image storage, I'd prefer that we can integrate
the end-user's existing Docker registry as a starting.
{quote}
Make sense.
One thing I realized is, if user only want to use a Docker image (instead of
Anaconda), asking users to create an environment, and put Docker image like:
{code:java}
Environment:
name: my-env
docker-image: example.com/my-docker-image:0.1.2
In experiment, user specify
Experiment:
name: "My-tf-job"
environment: "my-env"
task:
script: ...
resource: ...{code}
It is purely an overhead and bad user experiences.
How about we can provide a short cut to specify an "anonymous/embeded
Environment" as part of the Experiment, for example:
{code:java}
Experiment:
name: "My-tf-job"
environment: "my-env" // point to a "named" environment, or
environment:
docker-image: example.com/my-docker-image:0.1.2
kernel-name: ...
task:
script: ...
resource: ... {code}
If it is possible, we should make the "embedded environment" be part of 0.4.0
itself, so we don't have to change this API when we release 0.5.0
Thoughts? [~ztang], [~maniraj...@gmail.com]
> Submarine Environment Management
>
>
> Key: SUBMARINE-507
> URL: https://issues.apache.org/jira/browse/SUBMARINE-507
> Project: Apache Submarine
> Issue Type: New Feature
>Reporter: Manikandan R
>Assignee: Manikandan R
>Priority: Major
> Labels: pull-request-available
>
> Scope of this JIRA is to support environment management. It includes the
> following:
> 1. Create Environment
> 2. Update Environment
> 3. Delete Environment
> 4. List Environments
> In addition, this JIRA should also ensures that environments has been
> persisted like experiments so that it can used for later use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-519) Overhaul Structure of Submarine Github Markdown Files
[ https://issues.apache.org/jira/browse/SUBMARINE-519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117997#comment-17117997 ] Wangda Tan commented on SUBMARINE-519: -- Marked it to be 0.4.0 blocker so we can get docs fixed before 0.4.0 release. > Overhaul Structure of Submarine Github Markdown Files > - > > Key: SUBMARINE-519 > URL: https://issues.apache.org/jira/browse/SUBMARINE-519 > Project: Apache Submarine > Issue Type: Sub-task > Components: Doc >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > > Targets: > 1) Cleanup obsolete contents. > 2) Update vision to the latest vision. > 3) Make a clear boundary between the developer doc and user doc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-519) Overhaul Structure of Submarine Github Markdown Files
[ https://issues.apache.org/jira/browse/SUBMARINE-519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-519: - Target Version: 0.4.0 > Overhaul Structure of Submarine Github Markdown Files > - > > Key: SUBMARINE-519 > URL: https://issues.apache.org/jira/browse/SUBMARINE-519 > Project: Apache Submarine > Issue Type: Sub-task > Components: Doc >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > > Targets: > 1) Cleanup obsolete contents. > 2) Update vision to the latest vision. > 3) Make a clear boundary between the developer doc and user doc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-519) Overhaul Structure of Submarine Github Markdown Files
[ https://issues.apache.org/jira/browse/SUBMARINE-519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-519: - Priority: Blocker (was: Major) > Overhaul Structure of Submarine Github Markdown Files > - > > Key: SUBMARINE-519 > URL: https://issues.apache.org/jira/browse/SUBMARINE-519 > Project: Apache Submarine > Issue Type: Sub-task > Components: Doc >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > > Targets: > 1) Cleanup obsolete contents. > 2) Update vision to the latest vision. > 3) Make a clear boundary between the developer doc and user doc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-519) Overhaul Structure of Submarine Github Markdown Files
[ https://issues.apache.org/jira/browse/SUBMARINE-519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-519: - Description: Targets: 1) Cleanup obsolete contents. 2) Update vision to the latest vision. 3) Make a clear boundary between the developer doc and user doc. > Overhaul Structure of Submarine Github Markdown Files > - > > Key: SUBMARINE-519 > URL: https://issues.apache.org/jira/browse/SUBMARINE-519 > Project: Apache Submarine > Issue Type: Sub-task > Components: Doc >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > > Targets: > 1) Cleanup obsolete contents. > 2) Update vision to the latest vision. > 3) Make a clear boundary between the developer doc and user doc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Assigned] (SUBMARINE-519) Overhaul Structure of Submarine Github Markdown Files
[ https://issues.apache.org/jira/browse/SUBMARINE-519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan reassigned SUBMARINE-519: Assignee: Wangda Tan > Overhaul Structure of Submarine Github Markdown Files > - > > Key: SUBMARINE-519 > URL: https://issues.apache.org/jira/browse/SUBMARINE-519 > Project: Apache Submarine > Issue Type: Sub-task > Components: Doc >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Created] (SUBMARINE-519) Overhaul Structure of Submarine Github Markdown Files
Wangda Tan created SUBMARINE-519: Summary: Overhaul Structure of Submarine Github Markdown Files Key: SUBMARINE-519 URL: https://issues.apache.org/jira/browse/SUBMARINE-519 Project: Apache Submarine Issue Type: Sub-task Components: Doc Reporter: Wangda Tan -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-518) [Umbrella] Overhaul Submarine Documentation And Website
[ https://issues.apache.org/jira/browse/SUBMARINE-518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117996#comment-17117996 ] Wangda Tan commented on SUBMARINE-518: -- We also want to make use of tools like Sphinx to generate static html from source control file, it will be especially useful for Python-related APIs, etc. For example: [https://pytorch.org/docs/stable/index.html] As we're doing more Python SDK API, we also need to get this documentation versioned since we will have different APIs between versions, and tools like Sphinx can generate API doc from source file, which help a lot of efforts. Ideally, documentation can be built nightly so there's always a doc point to the master branch (Like pytorch example mentioned above). In the short term, it is fine to make documentation part of Github, but we need to make a clear structure between these markdown files. > [Umbrella] Overhaul Submarine Documentation And Website > --- > > Key: SUBMARINE-518 > URL: https://issues.apache.org/jira/browse/SUBMARINE-518 > Project: Apache Submarine > Issue Type: Task >Reporter: Wangda Tan >Priority: Major > > Now we have a number of issues of our Documentation and Website: > 1) There's no enough distinguish between website / Github documentation. > 2) There's no site to host documentation based on versions. > We need find a way to solve this problem, that's the main purpose of the > Jira. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-518) [Umbrella] Overhaul Submarine Documentation And Website
[ https://issues.apache.org/jira/browse/SUBMARINE-518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117994#comment-17117994 ] Wangda Tan commented on SUBMARINE-518: -- Here's the proposal: Make clear responsibilities between website and github docs: * Website is responsible for: ** Showing the vision of the project. ** Provide links to *User-Documentation(How to use, how to install, how to build)*/Download. ** Provide links to Github, Blogpost. * Roadmap is documented on WIKI: ** [https://cwiki.apache.org/confluence/display/SUBMARINE/Roadmap] ** The reason to put roadmap on Atlassian WIKI is, it is frequently changing, we want release managers can easier modify roadmaps. ** Also, integration from confluent to Jira is helpful to track the release progress. * Github is responsible for: ** Source control of following contents: *** Landing page of Github (README.md) should include following contents. (Short) Description of the project Link to the User Documentation. *** CONTRIBUTOR.md > [Umbrella] Overhaul Submarine Documentation And Website > --- > > Key: SUBMARINE-518 > URL: https://issues.apache.org/jira/browse/SUBMARINE-518 > Project: Apache Submarine > Issue Type: Task >Reporter: Wangda Tan >Priority: Major > > Now we have a number of issues of our Documentation and Website: > 1) There's no enough distinguish between website / Github documentation. > 2) There's no site to host documentation based on versions. > We need find a way to solve this problem, that's the main purpose of the > Jira. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Created] (SUBMARINE-518) [Umbrella] Overhaul Submarine Documentation And Website
Wangda Tan created SUBMARINE-518: Summary: [Umbrella] Overhaul Submarine Documentation And Website Key: SUBMARINE-518 URL: https://issues.apache.org/jira/browse/SUBMARINE-518 Project: Apache Submarine Issue Type: Task Reporter: Wangda Tan Now we have a number of issues of our Documentation and Website: 1) There's no enough distinguish between website / Github documentation. 2) There's no site to host documentation based on versions. We need find a way to solve this problem, that's the main purpose of the Jira. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-507) [Umbrella] Submarine Environment Management
[ https://issues.apache.org/jira/browse/SUBMARINE-507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116299#comment-17116299 ] Wangda Tan commented on SUBMARINE-507: -- Thanks [~maniraj...@gmail.com], I just took a brief look at the patch, have some questions: 1) is this patch on top of the SUBMARINE-506? 2) We need to make sure the patch is using the same swagger framework. 3) How about persistent storage of the environment metadata? It should be similar to the experiment APIs. 4) I think for now we don't have to include VM Image field till we can support it. Also for the Jira: 1) Can you include the description of the problem you want to solve? 2) It looks like filing one Jira for each C/R/U/D seems overkill, we should have a good API definition for these basic operations first. Adding [~tangzhankun], [~pingsutw] to help with patch review. > [Umbrella] Submarine Environment Management > --- > > Key: SUBMARINE-507 > URL: https://issues.apache.org/jira/browse/SUBMARINE-507 > Project: Apache Submarine > Issue Type: New Feature >Reporter: Manikandan R >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-499) Refactor job API to experiment API
[ https://issues.apache.org/jira/browse/SUBMARINE-499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110471#comment-17110471 ] Wangda Tan commented on SUBMARINE-499: -- [~jiwq], [https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html] mentioned Tensorboard can be supported by PyTorch (officially). I think we should put this to at least PyTorch and Tensorflow spec. > Refactor job API to experiment API > -- > > Key: SUBMARINE-499 > URL: https://issues.apache.org/jira/browse/SUBMARINE-499 > Project: Apache Submarine > Issue Type: New Feature > Components: Backend Server >Reporter: Zhankun Tang >Assignee: Wanqiang Ji >Priority: Major > Fix For: 0.4.0 > > > With Wangda's new design agreed, we're going to refactor job API spec to > experiment spec. > Refer > [https://docs.google.com/document/d/1T-NowARlz0EGrsMiAdUmA6af1b_G4bUPu-eKi7nvuk8/edit#] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-499) Refactor job API to experiment API
[ https://issues.apache.org/jira/browse/SUBMARINE-499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108676#comment-17108676 ] Wangda Tan commented on SUBMARINE-499: -- Thanks [~jiwq]! > Refactor job API to experiment API > -- > > Key: SUBMARINE-499 > URL: https://issues.apache.org/jira/browse/SUBMARINE-499 > Project: Apache Submarine > Issue Type: New Feature > Components: Backend Server >Reporter: Zhankun Tang >Assignee: Wanqiang Ji >Priority: Major > Fix For: 0.4.0 > > > With Wangda's new design agreed, we're going to refactor job API spec to > experiment spec. > Refer > [https://docs.google.com/document/d/1T-NowARlz0EGrsMiAdUmA6af1b_G4bUPu-eKi7nvuk8/edit#] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-494) Design for notebook support in Submarine
[ https://issues.apache.org/jira/browse/SUBMARINE-494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103975#comment-17103975 ] Wangda Tan commented on SUBMARINE-494: -- [~lowc1012], [~liuxun] here's an initial draft for notebook support, please review and let me know what're your thoughts. Thanks. [https://github.com/wangdatan/submarine/blob/SUBMARINE-494/docs/design/notebook-implementation.md] > Design for notebook support in Submarine > > > Key: SUBMARINE-494 > URL: https://issues.apache.org/jira/browse/SUBMARINE-494 > Project: Apache Submarine > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-494) Design for notebook support in Submarine
[ https://issues.apache.org/jira/browse/SUBMARINE-494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-494: - Target Version: 0.5.0 > Design for notebook support in Submarine > > > Key: SUBMARINE-494 > URL: https://issues.apache.org/jira/browse/SUBMARINE-494 > Project: Apache Submarine > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Created] (SUBMARINE-494) Design for notebook support in Submarine
Wangda Tan created SUBMARINE-494: Summary: Design for notebook support in Submarine Key: SUBMARINE-494 URL: https://issues.apache.org/jira/browse/SUBMARINE-494 Project: Apache Submarine Issue Type: Sub-task Reporter: Wangda Tan -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Assigned] (SUBMARINE-494) Design for notebook support in Submarine
[ https://issues.apache.org/jira/browse/SUBMARINE-494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan reassigned SUBMARINE-494: Assignee: Wangda Tan > Design for notebook support in Submarine > > > Key: SUBMARINE-494 > URL: https://issues.apache.org/jira/browse/SUBMARINE-494 > Project: Apache Submarine > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-477) [Umbrella] Support Jupyter notebook
[ https://issues.apache.org/jira/browse/SUBMARINE-477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-477: - Summary: [Umbrella] Support Jupyter notebook (was: [Umbrella] Support jupyterlab) > [Umbrella] Support Jupyter notebook > --- > > Key: SUBMARINE-477 > URL: https://issues.apache.org/jira/browse/SUBMARINE-477 > Project: Apache Submarine > Issue Type: New Feature > Components: Workbench Notebook >Reporter: Liu Xun >Priority: Major > > Python code writing by running jupyter container in submarine -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-481) Use Swagger to describe and document submarine RESTful APIs
[ https://issues.apache.org/jira/browse/SUBMARINE-481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097715#comment-17097715 ] Wangda Tan commented on SUBMARINE-481: -- Thanks [~pingsutw], I just posted some comments to [https://docs.google.com/document/d/1T-NowARlz0EGrsMiAdUmA6af1b_G4bUPu-eKi7nvuk8/edit#heading=h.t713oano4cl4, |https://docs.google.com/document/d/1T-NowARlz0EGrsMiAdUmA6af1b_G4bUPu-eKi7nvuk8/edit#heading=h.t713oano4cl4] I will let [~sunilg]/[~tangzhankun], etc. to help with further reviews especially how we do implementation by using Swagger. Swagger 3.0 should be a better choice as we have some recent projects are struggling to upgrade to 3.0. 3.0 has some important features like inheritance support, etc. [https://swagger.io/docs/specification/data-models/inheritance-and-polymorphism/] [~jiwq], do you have any more updates about how much work needed to use the new API? Draft see [https://docs.google.com/document/d/1T-NowARlz0EGrsMiAdUmA6af1b_G4bUPu-eKi7nvuk8/edit#heading=h.t713oano4cl4] > Use Swagger to describe and document submarine RESTful APIs > --- > > Key: SUBMARINE-481 > URL: https://issues.apache.org/jira/browse/SUBMARINE-481 > Project: Apache Submarine > Issue Type: Improvement > Components: Doc >Reporter: Kevin Su >Assignee: Kevin Su >Priority: Major > Labels: pull-request-available > Attachments: experiment.yaml, swagger.yaml > > > Follow > [JobManagerRestApi.java|https://github.com/apache/submarine/blob/31f9322216307f958a1c3ec79e8a09cb0a5f5b5e/submarine-server/server-core/src/main/java/org/apache/submarine/server/rest/JobManagerRestApi.java#L44] > and [Job.java > |https://github.com/apache/submarine/blob/master/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/job/Job.java], > [submarine-server > doc|https://github.com/apache/submarine/tree/master/docs/submarine-server] to > define our job API by swagger > We could copy swagger.ymal to [https://editor.swagger.io/] so that we could > automatically generate doc and related client API. > Anything I miss or something that needs to modify, please let me know > cc [~leftnoteasy] [~liuxun] [~jiwq] [~tangzhankun] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-481) Use Swagger to describe and document submarine RESTful APIs
[ https://issues.apache.org/jira/browse/SUBMARINE-481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094814#comment-17094814 ] Wangda Tan commented on SUBMARINE-481: -- [~tangzhankun], [~jiwq], I'm wondering how much effort we need to update our existing code to use the job spec. but I do think we need to update it before 0.4.0 release. > Use Swagger to describe and document submarine RESTful APIs > --- > > Key: SUBMARINE-481 > URL: https://issues.apache.org/jira/browse/SUBMARINE-481 > Project: Apache Submarine > Issue Type: Improvement > Components: Doc >Reporter: Kevin Su >Priority: Major > Labels: pull-request-available > Attachments: swagger.yaml > > > Follow > [JobManagerRestApi.java|https://github.com/apache/submarine/blob/31f9322216307f958a1c3ec79e8a09cb0a5f5b5e/submarine-server/server-core/src/main/java/org/apache/submarine/server/rest/JobManagerRestApi.java#L44] > and [Job.java > |https://github.com/apache/submarine/blob/master/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/job/Job.java], > [submarine-server > doc|https://github.com/apache/submarine/tree/master/docs/submarine-server] to > define our job API by swagger > We could copy swagger.ymal to [https://editor.swagger.io/] so that we could > automatically generate doc and related client API. > Anything I miss or something that needs to modify, please let me know > cc [~leftnoteasy] [~liuxun] [~jiwq] [~tangzhankun] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-481) Use Swagger to describe and document submarine RESTful APIs
[
https://issues.apache.org/jira/browse/SUBMARINE-481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094813#comment-17094813
]
Wangda Tan commented on SUBMARINE-481:
--
Thanks [~pingsutw], I still saw it is using "Job" as the name, I think we
should switch to "Experiment" for the first cut.
Here're my rough ideas about how to define the experiments API:
Top-level APIs:
- POST /experiment : add new experiment
- GET /experiment/ \{experiment-id}: get an experiment
- GET /experiment/list (with parameters): get experiments with parameters.
- DELETE /experiment/\{experiment-id}
: delete an experiment.
Objects definition:
{code:java}
Experiment {
// runtime status of an experiment
ExperimentStatus: {
ExperimentState: Running (stringm, common states of Experiments,
such as Running, New, Pending, Completed, Failed, etc.)
startTime: string, (in UTC format, like 2020-04-28T19:01:37Z)
runningTimeInMilliSeconds: long (how long the job is running),
terminatedTime: string, (in UTC format, empty if not terminated).
},
// experiment spec (static)
Experiment: {
// below are common fields for all experiments
name: $unique-name-for-the-experiment (string)
environment: $name-for-environment (string)
// Where we can sync the code,
code:
$ref: "#/definitions/CodeSpec"
timeout:
$ref: "#/definitions/Time"
type: string with following values:
("Script", "Tensorflow", "PyTorch", "Template")
// based on the type, one of the following types
parameters:
oneOf: // Use oneOf keyword of swagger, see
https://swagger.io/docs/specification/data-models/inheritance-and-polymorphism/
- $ref: "#/definitions/ScriptParameterSpec"
- $ref: "#/definitions/TensorflowParameterSpec"
- ...
},
}
CodeSpec {
syncMode: s3/git (string)
url: string (git://, or s3://)
}
ScriptParameterSpec {
cli:
- $ref: "#/definitions/CommandLineOptions"
resource:
- $ref: "#/definitions/ResourceSpec"
}
TensorflowParameterSpec {
// reference to the below like:
ps:
environment: "team-default-ml-cpu"
resource_constraint:
res="mem=20gb, vcore=3, gpu=0"
worker:
environment: "team-default-ml-gpu"
resource_constraint:
res="mem=20gb, vcore=3, gpu=2"
}
Time {
time: 30
unit: "Minute" (or other units)
}
CommandLineOptions {
type: array
items:
type: string
}
ResourceSpec {
// Should be a list of key (resource type) to value, and unit.
like cpu=10, memory = 20 GB, etc.
}
{code}
> Use Swagger to describe and document submarine RESTful APIs
> ---
>
> Key: SUBMARINE-481
> URL: https://issues.apache.org/jira/browse/SUBMARINE-481
> Project: Apache Submarine
> Issue Type: Improvement
> Components: Doc
>Reporter: Kevin Su
>Priority: Major
> Labels: pull-request-available
> Attachments: swagger.yaml
>
>
> Follow
> [JobManagerRestApi.java|https://github.com/apache/submarine/blob/31f9322216307f958a1c3ec79e8a09cb0a5f5b5e/submarine-server/server-core/src/main/java/org/apache/submarine/server/rest/JobManagerRestApi.java#L44]
> and [Job.java
> |https://github.com/apache/submarine/blob/master/submarine-server/server-api/src/main/java/org/apache/submarine/server/api/job/Job.java],
> [submarine-server
> doc|https://github.com/apache/submarine/tree/master/docs/submarine-server] to
> define our job API by swagger
> We could copy swagger.ymal to [https://editor.swagger.io/] so that we could
> automatically generate doc and related client API.
> Anything I miss or something that needs to modify, please let me know
> cc [~leftnoteasy] [~liuxun] [~jiwq] [~tangzhankun]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-470) [web]Implement database connection in department page with Angular
[ https://issues.apache.org/jira/browse/SUBMARINE-470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-470: - Fix Version/s: (was: 0.5.0) > [web]Implement database connection in department page with Angular > -- > > Key: SUBMARINE-470 > URL: https://issues.apache.org/jira/browse/SUBMARINE-470 > Project: Apache Submarine > Issue Type: Sub-task > Components: Workbench Web >Reporter: HSIN YUAN, CHEN >Assignee: HSIN YUAN, CHEN >Priority: Major > > Implement database connection in department page with Angular. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-470) [web]Implement database connection in department page with Angular
[ https://issues.apache.org/jira/browse/SUBMARINE-470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-470: - Target Version: 0.5.0 > [web]Implement database connection in department page with Angular > -- > > Key: SUBMARINE-470 > URL: https://issues.apache.org/jira/browse/SUBMARINE-470 > Project: Apache Submarine > Issue Type: Sub-task > Components: Workbench Web >Reporter: HSIN YUAN, CHEN >Assignee: HSIN YUAN, CHEN >Priority: Major > > Implement database connection in department page with Angular. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-477) [Umbrella] Support jupyterlab
[ https://issues.apache.org/jira/browse/SUBMARINE-477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-477: - Target Version: 0.5.0 > [Umbrella] Support jupyterlab > - > > Key: SUBMARINE-477 > URL: https://issues.apache.org/jira/browse/SUBMARINE-477 > Project: Apache Submarine > Issue Type: New Feature > Components: Workbench Notebook >Reporter: Liu Xun >Priority: Major > > Python code writing by running jupyter container in submarine -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-477) [Umbrella] Support jupyterlab
[ https://issues.apache.org/jira/browse/SUBMARINE-477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-477: - Fix Version/s: (was: 0.5.0) > [Umbrella] Support jupyterlab > - > > Key: SUBMARINE-477 > URL: https://issues.apache.org/jira/browse/SUBMARINE-477 > Project: Apache Submarine > Issue Type: New Feature > Components: Workbench Notebook >Reporter: Liu Xun >Priority: Major > > Python code writing by running jupyter container in submarine -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-478) Running jupyterlab container in KIND
[ https://issues.apache.org/jira/browse/SUBMARINE-478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-478: - Target Version: 0.5.0 > Running jupyterlab container in KIND > - > > Key: SUBMARINE-478 > URL: https://issues.apache.org/jira/browse/SUBMARINE-478 > Project: Apache Submarine > Issue Type: Sub-task > Components: Workbench Notebook >Reporter: Liu Xun >Assignee: Ryan Lo >Priority: Major > Attachments: jupyter_template.yaml.jinja > > > Use jupyter's yaml file to run a jupyter container, > 1. First support mounting local path into jupyter > 2. Support setting jupyter password > 3. Run by creating jupyter pod in KIND > 4. Provide simple Python library support (if the submarine project has a > relatively complete Python image, it can also be used) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-478) Running jupyterlab container in KIND
[ https://issues.apache.org/jira/browse/SUBMARINE-478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-478: - Fix Version/s: (was: 0.5.0) > Running jupyterlab container in KIND > - > > Key: SUBMARINE-478 > URL: https://issues.apache.org/jira/browse/SUBMARINE-478 > Project: Apache Submarine > Issue Type: Sub-task > Components: Workbench Notebook >Reporter: Liu Xun >Assignee: Ryan Lo >Priority: Major > Attachments: jupyter_template.yaml.jinja > > > Use jupyter's yaml file to run a jupyter container, > 1. First support mounting local path into jupyter > 2. Support setting jupyter password > 3. Run by creating jupyter pod in KIND > 4. Provide simple Python library support (if the submarine project has a > relatively complete Python image, it can also be used) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-482) [WEB]Implement database connection in team page with Angular
[ https://issues.apache.org/jira/browse/SUBMARINE-482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-482: - Fix Version/s: (was: 0.5.0) Target Version: 0.5.0 > [WEB]Implement database connection in team page with Angular > > > Key: SUBMARINE-482 > URL: https://issues.apache.org/jira/browse/SUBMARINE-482 > Project: Apache Submarine > Issue Type: Sub-task > Components: Workbench Web >Reporter: Wen-Chien,Ruan >Assignee: Wen-Chien,Ruan >Priority: Major > > Implement database connection in team page with Angular. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Created] (SUBMARINE-469) Update Submarine Architecture Doc
Wangda Tan created SUBMARINE-469: Summary: Update Submarine Architecture Doc Key: SUBMARINE-469 URL: https://issues.apache.org/jira/browse/SUBMARINE-469 Project: Apache Submarine Issue Type: Improvement Components: Doc Reporter: Wangda Tan We have many discussions recently and made some proposals for the Submarine roadmap and corresponding roadmaps. We need to update our docs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-230) [Umbrella] Submarine Interpreter Module
[ https://issues.apache.org/jira/browse/SUBMARINE-230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-230: - Target Version: 0.5.0 (was: 0.4.0) > [Umbrella] Submarine Interpreter Module > --- > > Key: SUBMARINE-230 > URL: https://issues.apache.org/jira/browse/SUBMARINE-230 > Project: Apache Submarine > Issue Type: New Feature > Components: Workbench Interpreter >Reporter: Liu Xun >Assignee: Liu Xun >Priority: Major > > Because the user needs to write code and execution in the workbench (such as > executing Python, Spark sql), in order to avoid user resource competition and > code security control when the user executes the code, Workbench will create > a Docker container for each user, User can write code in own container, bind > the Github account for code version management, execute python and spark > programs. > 2. Interpreter > Submarine provides multi-language execution for notebooks in workbench > through interpreters in a variety of programming languages (for example, > python interpreter, spark interpreter). > 2.1 Interpreter execution flow > 1. The user enters the code in the workbench notebook and clicks the execute > button. First the workbench server starts the interpreter process. > 2. The workbench server sends the user's code to the interpreter process via > the interface for execution, and gets the execution result, returning to the > workbench notebook for display. > Design doc: > https://docs.google.com/document/d/1LJc5hRTWfIs6K5f7vMgGC_FUizeQf-qqiWICqO8SoR8/edit# -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Resolved] (SUBMARINE-403) [Umbrella] Support PyTorch job management on K8s in submarine server
[ https://issues.apache.org/jira/browse/SUBMARINE-403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan resolved SUBMARINE-403. -- Resolution: Fixed > [Umbrella] Support PyTorch job management on K8s in submarine server > > > Key: SUBMARINE-403 > URL: https://issues.apache.org/jira/browse/SUBMARINE-403 > Project: Apache Submarine > Issue Type: Improvement > Components: Backend Server >Reporter: Zhankun Tang >Priority: Critical > > As one of the popular deep learning framework, PyTorch support in > submarine-server is an appealing feature. > This umbrella JIRA may consist of tasks: > 1. Deployment of PyTorch operator on k8s. It can be script-based and then > submarine-operator based. > 2. Job CRUD in the submarine server. > 3. PySubmarine client for managing PyTorch jobs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-98) [Umbrella] Submarine web page
[ https://issues.apache.org/jira/browse/SUBMARINE-98?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-98: Target Version: 0.5.0 (was: 0.4.0) > [Umbrella] Submarine web page > - > > Key: SUBMARINE-98 > URL: https://issues.apache.org/jira/browse/SUBMARINE-98 > Project: Apache Submarine > Issue Type: New Feature > Components: Workbench Web >Reporter: Xun Liu >Assignee: Hao Zhou >Priority: Major > Labels: submarine-external-repo > > Submarine can be managed via submarine web: > 1. Support multi-user and multi-role > 2. Support for allocation management of server resources > 3. Support data development, algorithm development, and model launch through > notebook > 4. Support workflow workflow orchestration > 5. Support service monitoring after the model goes online -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-148) [Umbrella] Submarine-sdk tracking
[ https://issues.apache.org/jira/browse/SUBMARINE-148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-148: - Target Version: (was: 0.4.0) > [Umbrella] Submarine-sdk tracking > - > > Key: SUBMARINE-148 > URL: https://issues.apache.org/jira/browse/SUBMARINE-148 > Project: Apache Submarine > Issue Type: New Feature > Components: SDK >Reporter: Kevin Su >Assignee: Kevin Su >Priority: Major > > The submarine-sdk support Python, Scala, R language for algorithm > development, The SDK is provided to help developers use submarine's internal > data caching, data exchange, and task tracking to more efficiently improve > the development and execution of machine learning tasks. > > pysubmarine *tracking* module is a python API for logging parameters, code > versions, metrics > 1. Add the sumbrine-sdk/pysubmarine module. > 2. pysubmarine supports mysql/HDFS in two ways to store metric data. > Priority to do mysql, in order to performance requires python to directly > write mysql data, do not take the submarine-sever RESTful interface. > 3. Add metric, param instance to the submarine-server database and rest > 4. Add Metric tracking ui to submarine-web > 5. Add a metric graphical display to submarine-web -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-76) Create factory class to produce either TensorFlow or PyTorch service spec for Native Service
[ https://issues.apache.org/jira/browse/SUBMARINE-76?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-76: Target Version: (was: 0.4.0) > Create factory class to produce either TensorFlow or PyTorch service spec for > Native Service > > > Key: SUBMARINE-76 > URL: https://issues.apache.org/jira/browse/SUBMARINE-76 > Project: Apache Submarine > Issue Type: Improvement > Components: YARN-native-service >Reporter: Szilard Nemeth >Priority: Minor > > In YarnServiceJobSubmitter#submitJob, we submit either a TF / PyTorch job > based on the Service spec. > The code could be improved by creating a factory class to produce the service > spec as the 2 types of Service specs are having very similar dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-53) Generate Service spec + launch script for distributed PyTorch learning job
[ https://issues.apache.org/jira/browse/SUBMARINE-53?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-53: Target Version: (was: 0.4.0) > Generate Service spec + launch script for distributed PyTorch learning job > -- > > Key: SUBMARINE-53 > URL: https://issues.apache.org/jira/browse/SUBMARINE-53 > Project: Apache Submarine > Issue Type: Sub-task > Components: YARN-native-service >Reporter: Szilard Nemeth >Priority: Major > > More info here: https://pytorch.org/tutorials/intermediate/dist_tuto.html -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-75) Add tests to HadoopEnvironmentSetup class
[ https://issues.apache.org/jira/browse/SUBMARINE-75?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-75: Target Version: (was: 0.4.0) > Add tests to HadoopEnvironmentSetup class > - > > Key: SUBMARINE-75 > URL: https://issues.apache.org/jira/browse/SUBMARINE-75 > Project: Apache Submarine > Issue Type: Improvement > Components: Commons >Reporter: Szilard Nemeth >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-81) Add documentation about YAML config parser with example config
[ https://issues.apache.org/jira/browse/SUBMARINE-81?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074955#comment-17074955 ] Wangda Tan commented on SUBMARINE-81: - [~adam.antal], are you still working on this, or plan to working on this in the short term? If no, can you remove target version? Trying to cleanup old JIRAs. > Add documentation about YAML config parser with example config > -- > > Key: SUBMARINE-81 > URL: https://issues.apache.org/jira/browse/SUBMARINE-81 > Project: Apache Submarine > Issue Type: Bug > Components: Doc >Reporter: Szilard Nemeth >Assignee: Adam Antal >Priority: Major > > YAML parser was added but there's no example documentation of a valid > configuration. > We need to fix this! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-77) Create a builder class for NS Component
[ https://issues.apache.org/jira/browse/SUBMARINE-77?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074956#comment-17074956 ] Wangda Tan commented on SUBMARINE-77: - [~adam.antal], are you still working on this, or plan to working on this in the short term? If no, can you remove target version? Trying to cleanup old JIRAs. > Create a builder class for NS Component > --- > > Key: SUBMARINE-77 > URL: https://issues.apache.org/jira/browse/SUBMARINE-77 > Project: Apache Submarine > Issue Type: Improvement > Components: Commons >Reporter: Szilard Nemeth >Assignee: Adam Antal >Priority: Major > > In TensorFlowPsComponent#createComponent, we create a component object. > According to [~adam.antal]'s comment given on one of my resolved jiras, this > method shouts for a builder that builds the Component object. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-148) [Umbrella] Submarine-sdk tracking
[ https://issues.apache.org/jira/browse/SUBMARINE-148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074953#comment-17074953 ] Wangda Tan commented on SUBMARINE-148: -- [~pingsutw], could you check [~weichiu]'s comment above? > [Umbrella] Submarine-sdk tracking > - > > Key: SUBMARINE-148 > URL: https://issues.apache.org/jira/browse/SUBMARINE-148 > Project: Apache Submarine > Issue Type: New Feature > Components: SDK >Reporter: Kevin Su >Assignee: Kevin Su >Priority: Major > > The submarine-sdk support Python, Scala, R language for algorithm > development, The SDK is provided to help developers use submarine's internal > data caching, data exchange, and task tracking to more efficiently improve > the development and execution of machine learning tasks. > > pysubmarine *tracking* module is a python API for logging parameters, code > versions, metrics > 1. Add the sumbrine-sdk/pysubmarine module. > 2. pysubmarine supports mysql/HDFS in two ways to store metric data. > Priority to do mysql, in order to performance requires python to directly > write mysql data, do not take the submarine-sever RESTful interface. > 3. Add metric, param instance to the submarine-server database and rest > 4. Add Metric tracking ui to submarine-web > 5. Add a metric graphical display to submarine-web -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-98) [Umbrella] Submarine web page
[ https://issues.apache.org/jira/browse/SUBMARINE-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074954#comment-17074954 ] Wangda Tan commented on SUBMARINE-98: - [~liuxun323], is it still target for 0.4.0? If no, we can move this JIRA and its children to 0.5.0+? > [Umbrella] Submarine web page > - > > Key: SUBMARINE-98 > URL: https://issues.apache.org/jira/browse/SUBMARINE-98 > Project: Apache Submarine > Issue Type: New Feature > Components: Workbench Web >Reporter: Xun Liu >Assignee: Hao Zhou >Priority: Major > Labels: submarine-external-repo > > Submarine can be managed via submarine web: > 1. Support multi-user and multi-role > 2. Support for allocation management of server resources > 3. Support data development, algorithm development, and model launch through > notebook > 4. Support workflow workflow orchestration > 5. Support service monitoring after the model goes online -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-184) PyTorch support for Tensorboard in yarnservice-runtime
[ https://issues.apache.org/jira/browse/SUBMARINE-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074952#comment-17074952 ] Wangda Tan commented on SUBMARINE-184: -- [~pingsutw], is it sometime we target for 0.4.0? If no, can you move it to future release or just remove the target version? > PyTorch support for Tensorboard in yarnservice-runtime > -- > > Key: SUBMARINE-184 > URL: https://issues.apache.org/jira/browse/SUBMARINE-184 > Project: Apache Submarine > Issue Type: Improvement >Reporter: Kevin Su >Assignee: Kevin Su >Priority: Major > > PyTorch 1.1 announced support for TensorBoard. > [https://pytorch.org/docs/stable/tensorboard.html] > [https://github.com/pytorch/pytorch/releases] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-225) [Umbrella] Submarine Client
[ https://issues.apache.org/jira/browse/SUBMARINE-225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074951#comment-17074951 ] Wangda Tan commented on SUBMARINE-225: -- [~jiwq], do we still need this JIRA given we have SUBMARINE-224 already? If we don't need this, can you close this ticket and its child as dup? > [Umbrella] Submarine Client > --- > > Key: SUBMARINE-225 > URL: https://issues.apache.org/jira/browse/SUBMARINE-225 > Project: Apache Submarine > Issue Type: New Feature > Components: Backend Server >Reporter: Wanqiang Ji >Priority: Major > > Maybe we can used the SC in the JIRA summary field to mark the task related > to submarine client. Such as [SC] Create the submarine-client module. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-180) [Umbrella] Submarine E2E Test Framework
[ https://issues.apache.org/jira/browse/SUBMARINE-180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074949#comment-17074949 ] Wangda Tan commented on SUBMARINE-180: -- Set assignee to unassigned since it has multiple assignees under the JIRA. Reopened this ticket to set the target version to 0.4.0 as there're remaining tasks > [Umbrella] Submarine E2E Test Framework > --- > > Key: SUBMARINE-180 > URL: https://issues.apache.org/jira/browse/SUBMARINE-180 > Project: Apache Submarine > Issue Type: New Feature >Reporter: Xun Liu >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > This is New Submarine Module. > In order to guarantee the quality of submarine software, we need to provide > an end-to-end testing framework, in which at least the following functions > are included: > 1. Simulate all user WEB operations in submarine workbench through firefox > version 31. (We can refer to the end-to-end test method of the Apache > Zeppelin. > [https://github.com/apache/zeppelin/tree/master/zeppelin-integration] > 2. In the submarine-inter module, the WEB simulator simulates the operation > of the web front end to ensure the correctness of all web front-end > operations in the submarine-workbench. > 3. Create a submarine machine learning run cluster through docker and perform > algorithm testing on all the machine learning frameworks we provide. It may > be a bit difficult to achieve this, and you can consider implementing it > later. > In Travis CI, the end-to-end test framework will be triggered. Only through > all E2E tests can the newly submitted code be guaranteed to be merged into > the code repository without breaking the functionality of the original system. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-180) [Umbrella] Submarine E2E Test Framework
[ https://issues.apache.org/jira/browse/SUBMARINE-180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-180: - Component/s: E2E Test > [Umbrella] Submarine E2E Test Framework > --- > > Key: SUBMARINE-180 > URL: https://issues.apache.org/jira/browse/SUBMARINE-180 > Project: Apache Submarine > Issue Type: New Feature > Components: E2E Test >Reporter: Xun Liu >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > This is New Submarine Module. > In order to guarantee the quality of submarine software, we need to provide > an end-to-end testing framework, in which at least the following functions > are included: > 1. Simulate all user WEB operations in submarine workbench through firefox > version 31. (We can refer to the end-to-end test method of the Apache > Zeppelin. > [https://github.com/apache/zeppelin/tree/master/zeppelin-integration] > 2. In the submarine-inter module, the WEB simulator simulates the operation > of the web front end to ensure the correctness of all web front-end > operations in the submarine-workbench. > 3. Create a submarine machine learning run cluster through docker and perform > algorithm testing on all the machine learning frameworks we provide. It may > be a bit difficult to achieve this, and you can consider implementing it > later. > In Travis CI, the end-to-end test framework will be triggered. Only through > all E2E tests can the newly submitted code be guaranteed to be merged into > the code repository without breaking the functionality of the original system. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Reopened] (SUBMARINE-180) [Umbrella] Submarine E2E Test Framework
[ https://issues.apache.org/jira/browse/SUBMARINE-180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan reopened SUBMARINE-180: -- > [Umbrella] Submarine E2E Test Framework > --- > > Key: SUBMARINE-180 > URL: https://issues.apache.org/jira/browse/SUBMARINE-180 > Project: Apache Submarine > Issue Type: New Feature >Reporter: Xun Liu >Assignee: Kai-Hsun Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.3.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > This is New Submarine Module. > In order to guarantee the quality of submarine software, we need to provide > an end-to-end testing framework, in which at least the following functions > are included: > 1. Simulate all user WEB operations in submarine workbench through firefox > version 31. (We can refer to the end-to-end test method of the Apache > Zeppelin. > [https://github.com/apache/zeppelin/tree/master/zeppelin-integration] > 2. In the submarine-inter module, the WEB simulator simulates the operation > of the web front end to ensure the correctness of all web front-end > operations in the submarine-workbench. > 3. Create a submarine machine learning run cluster through docker and perform > algorithm testing on all the machine learning frameworks we provide. It may > be a bit difficult to achieve this, and you can consider implementing it > later. > In Travis CI, the end-to-end test framework will be triggered. Only through > all E2E tests can the newly submitted code be guaranteed to be merged into > the code repository without breaking the functionality of the original system. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-180) [Umbrella] Submarine E2E Test Framework
[ https://issues.apache.org/jira/browse/SUBMARINE-180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-180: - Fix Version/s: (was: 0.3.0) Target Version: 0.4.0 > [Umbrella] Submarine E2E Test Framework > --- > > Key: SUBMARINE-180 > URL: https://issues.apache.org/jira/browse/SUBMARINE-180 > Project: Apache Submarine > Issue Type: New Feature >Reporter: Xun Liu >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > This is New Submarine Module. > In order to guarantee the quality of submarine software, we need to provide > an end-to-end testing framework, in which at least the following functions > are included: > 1. Simulate all user WEB operations in submarine workbench through firefox > version 31. (We can refer to the end-to-end test method of the Apache > Zeppelin. > [https://github.com/apache/zeppelin/tree/master/zeppelin-integration] > 2. In the submarine-inter module, the WEB simulator simulates the operation > of the web front end to ensure the correctness of all web front-end > operations in the submarine-workbench. > 3. Create a submarine machine learning run cluster through docker and perform > algorithm testing on all the machine learning frameworks we provide. It may > be a bit difficult to achieve this, and you can consider implementing it > later. > In Travis CI, the end-to-end test framework will be triggered. Only through > all E2E tests can the newly submitted code be guaranteed to be merged into > the code repository without breaking the functionality of the original system. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Assigned] (SUBMARINE-180) [Umbrella] Submarine E2E Test Framework
[ https://issues.apache.org/jira/browse/SUBMARINE-180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan reassigned SUBMARINE-180: Assignee: (was: Kai-Hsun Chen) > [Umbrella] Submarine E2E Test Framework > --- > > Key: SUBMARINE-180 > URL: https://issues.apache.org/jira/browse/SUBMARINE-180 > Project: Apache Submarine > Issue Type: New Feature >Reporter: Xun Liu >Priority: Major > Labels: pull-request-available > Fix For: 0.3.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > This is New Submarine Module. > In order to guarantee the quality of submarine software, we need to provide > an end-to-end testing framework, in which at least the following functions > are included: > 1. Simulate all user WEB operations in submarine workbench through firefox > version 31. (We can refer to the end-to-end test method of the Apache > Zeppelin. > [https://github.com/apache/zeppelin/tree/master/zeppelin-integration] > 2. In the submarine-inter module, the WEB simulator simulates the operation > of the web front end to ensure the correctness of all web front-end > operations in the submarine-workbench. > 3. Create a submarine machine learning run cluster through docker and perform > algorithm testing on all the machine learning frameworks we provide. It may > be a bit difficult to achieve this, and you can consider implementing it > later. > In Travis CI, the end-to-end test framework will be triggered. Only through > all E2E tests can the newly submitted code be guaranteed to be merged into > the code repository without breaking the functionality of the original system. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-351) [Umbrella] Support Job Management (e.g., submission) on YARN by REST APIs
[ https://issues.apache.org/jira/browse/SUBMARINE-351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-351: - Summary: [Umbrella] Support Job Management (e.g., submission) on YARN by REST APIs (was: Support Job Management (e.g., submission) on YARN by REST APIs) > [Umbrella] Support Job Management (e.g., submission) on YARN by REST APIs > - > > Key: SUBMARINE-351 > URL: https://issues.apache.org/jira/browse/SUBMARINE-351 > Project: Apache Submarine > Issue Type: New Feature > Components: Backend Server >Reporter: Adrian Hsu >Assignee: Adrian Hsu >Priority: Major > > This JIRA provides REST APIs for YARN to accept the generic job spec and > manages jobs, based on the SUBMARINE-224 (says, the REST APIs for K8s). Also, > this Issue follows the design of Submarine Server > ([https://docs.google.com/document/d/1vCjt2z3DoagfeaVQV0nfyWqSYLymoP53rDcPcz-YcSc/edit]) > provided by SUBMARINE-224. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-370) Add & update some documentation for MXNet
[ https://issues.apache.org/jira/browse/SUBMARINE-370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-370: - Labels: document (was: docuentation) > Add & update some documentation for MXNet > - > > Key: SUBMARINE-370 > URL: https://issues.apache.org/jira/browse/SUBMARINE-370 > Project: Apache Submarine > Issue Type: Improvement >Reporter: Ryan Lo >Assignee: Ryan Lo >Priority: Trivial > Labels: document > > Related to SUBMARINE-202. > After it be resolved, we need to add & update some documentation for MXNet > (e.g., "WriteDockerfileMXNet.md", "TonYRuntimeGuide.md" ) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
