[jira] [Updated] (SUBMARINE-319) kind-cluster-build.sh doesn't invoke installed kind
[ https://issues.apache.org/jira/browse/SUBMARINE-319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zac Zhou updated SUBMARINE-319: --- Fix Version/s: 0.3.0 Resolution: Fixed Status: Resolved (was: Patch Available) Issue resolved by pull request 127 [https://github.com/apache/submarine/pull/127] > kind-cluster-build.sh doesn't invoke installed kind > --- > > Key: SUBMARINE-319 > URL: https://issues.apache.org/jira/browse/SUBMARINE-319 > Project: Apache Submarine > Issue Type: Sub-task > Components: Cloud-native Deployment >Reporter: Zac Zhou >Assignee: Zac Zhou >Priority: Major > Labels: pull-request-available > Fix For: 0.3.0 > > Time Spent: 20m > Remaining Estimate: 0h > > kind-cluster-build.sh can download kind automatically. But > kind-cluster-build.sh doesn't use it. kind-cluster-build.sh invoke kind > command which is defined in PATH variable. > We need to let users to move downloaded kind and kubectl to system path > locations. So that users can directly use kind and kubectl to interact with > the local k8s cluster -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Work logged] (SUBMARINE-319) kind-cluster-build.sh doesn't invoke installed kind
[ https://issues.apache.org/jira/browse/SUBMARINE-319?focusedWorklogId=360728=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360728 ] ASF GitHub Bot logged work on SUBMARINE-319: Author: ASF GitHub Bot Created on: 17/Dec/19 06:49 Start Date: 17/Dec/19 06:49 Worklog Time Spent: 10m Work Description: asfgit commented on pull request #127: SUBMARINE-319. kind-cluster-build.sh doesn't invoke installed kind URL: https://github.com/apache/submarine/pull/127 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 360728) Time Spent: 20m (was: 10m) > kind-cluster-build.sh doesn't invoke installed kind > --- > > Key: SUBMARINE-319 > URL: https://issues.apache.org/jira/browse/SUBMARINE-319 > Project: Apache Submarine > Issue Type: Sub-task > Components: Cloud-native Deployment >Reporter: Zac Zhou >Assignee: Zac Zhou >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > kind-cluster-build.sh can download kind automatically. But > kind-cluster-build.sh doesn't use it. kind-cluster-build.sh invoke kind > command which is defined in PATH variable. > We need to let users to move downloaded kind and kubectl to system path > locations. So that users can directly use kind and kubectl to interact with > the local k8s cluster -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-319) kind-cluster-build.sh doesn't invoke installed kind
[ https://issues.apache.org/jira/browse/SUBMARINE-319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SUBMARINE-319: - Labels: pull-request-available (was: ) > kind-cluster-build.sh doesn't invoke installed kind > --- > > Key: SUBMARINE-319 > URL: https://issues.apache.org/jira/browse/SUBMARINE-319 > Project: Apache Submarine > Issue Type: Sub-task > Components: Cloud-native Deployment >Reporter: Zac Zhou >Assignee: Zac Zhou >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > kind-cluster-build.sh can download kind automatically. But > kind-cluster-build.sh doesn't use it. kind-cluster-build.sh invoke kind > command which is defined in PATH variable. > We need to let users to move downloaded kind and kubectl to system path > locations. So that users can directly use kind and kubectl to interact with > the local k8s cluster -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[GitHub] [submarine] asfgit closed pull request #127: SUBMARINE-319. kind-cluster-build.sh doesn't invoke installed kind
asfgit closed pull request #127: SUBMARINE-319. kind-cluster-build.sh doesn't invoke installed kind URL: https://github.com/apache/submarine/pull/127 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[submarine] branch master updated: SUBMARINE-319. kind-cluster-build.sh doesn't invoke installed kind
This is an automated email from the ASF dual-hosted git repository.
zhouquan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/submarine.git
The following commit(s) were added to refs/heads/master by this push:
new fc7e722 SUBMARINE-319. kind-cluster-build.sh doesn't invoke installed
kind
fc7e722 is described below
commit fc7e722da84fe6a7f0a195ed6f080fdd410b1931
Author: Zac Zhou
AuthorDate: Thu Dec 12 17:30:05 2019 +0800
SUBMARINE-319. kind-cluster-build.sh doesn't invoke installed kind
### What is this PR for?
kind-cluster-build.sh can download kind automatically. But
kind-cluster-build.sh doesn't use it. kind-cluster-build.sh invoke kind command
which is defined in PATH variable.
We need to let users to move downloaded kind and kubectl to system path
locations. So that users can directly use kind and kubectl to interact with
the local k8s cluster.
### What type of PR is it?
Improvement
### Todos
* [ ] - Task
### What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-319
### How should this be tested?
https://travis-ci.org/yuanzac/hadoop-submarine/builds/624064493?utm_source=github_status_medium=notification
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? Yes
Author: Zac Zhou
Closes #127 from yuanzac/topic/SUBMARINE-319 and squashes the following
commits:
4519d4d [Zac Zhou] SUBMARINE-319. kind-cluster-build.sh doesn't invoke
installed kind
---
submarine-cloud/hack/kind-cluster-build.sh | 19 ++
submarine-cloud/submarine-operator.md | 61 --
2 files changed, 45 insertions(+), 35 deletions(-)
diff --git a/submarine-cloud/hack/kind-cluster-build.sh
b/submarine-cloud/hack/kind-cluster-build.sh
index 67a557a..5e01fed 100755
--- a/submarine-cloud/hack/kind-cluster-build.sh
+++ b/submarine-cloud/hack/kind-cluster-build.sh
@@ -22,9 +22,6 @@ cd $ROOT
source $ROOT/hack/lib.sh
-hack::ensure_kubectl
-hack::ensure_kind
-
usage() {
cat /dev/null;then
echo "${requirement} have installed"
else
echo "this script needs ${requirement}, please install ${requirement}
first."
+if test ${requirement} = "kind"; then
+hack::ensure_kind
+echo "Please add $KIND_BIN to PATH variable or copy it to one of
the locations $PATH"
+fi
+if test ${requirement} = "kubectl"; then
+hack::ensure_kubectl
+echo "Please add $KUBECTL_BIN to PATH variable or copy it to one
of the locations $PATH"
+fi
exit 1
fi
done
@@ -148,7 +153,7 @@ export KUBECONFIG="$(kind get kubeconfig-path
--name=${clusterName})"
echo "deploy docker registry in kind"
registryNode=${clusterName}-control-plane
-registryNodeIP=$($KUBECTL_BIN get nodes ${registryNode} -o template
--template='{{range.status.addresses}}{{if eq .type
"InternalIP"}}{{.address}}{{end}}{{end}}')
+registryNodeIP=$(kubectl get nodes ${registryNode} -o template
--template='{{range.status.addresses}}{{if eq .type
"InternalIP"}}{{.address}}{{end}}{{end}}')
registryFile=${workDir}/registry.yaml
cat <${registryFile}
@@ -219,10 +224,10 @@ spec:
- tcp-listen:5000,fork,reuseaddr
- tcp-connect:${registryNodeIP}:5000
EOF
-$KUBECTL_BIN apply -f ${registryFile}
+kubectl apply -f ${registryFile}
echo "init submarine env"
-$KUBECTL_BIN create ns submarine-e2e
+kubectl create ns submarine-e2e
echo "# success create cluster:[${clusterName}] #"
diff --git a/submarine-cloud/submarine-operator.md
b/submarine-cloud/submarine-operator.md
index a785222..a2e72be 100644
--- a/submarine-cloud/submarine-operator.md
+++ b/submarine-cloud/submarine-operator.md
@@ -1,30 +1,41 @@
-# Submarine Operator Development Instructions
+# Submarine Operator Instructions
-See also official sample：https://github.com/kubernetes/sample-controller
+## Build submarine operator
-# Implementation steps
+### Prerequisite
-## 1. Submit the CRD template and its instantiation object to the k8s cluster,
so that k8s can recognize
+1 Golang environment is required.
+
+2 Submarine project should be in the path of ${GOPATH}/src/github.com/apache/.
+Alternatively, we can create a soft link named submarine, pointing to submarine
+repo, under the path of ${GOPATH}/src/github.com/apache/.
+
+### Build submarine operator binary
+```
+cd ${GOPATH}/src/github.com/apache/submarine/submarine-cloud
Re: [DISCUSS] Enable github security notifications to all Submarine committers
Thanx Weichiu for initiating this. +1 from me as well. -Ayush > On 17-Dec-2019, at 12:00 PM, Wei-Chiu Chuang wrote: > > Oh wait, I sent to the old submarine dev mailing alias. > > The new alias is dev@submarine.apache.org > > > >> On Mon, Dec 16, 2019 at 10:19 PM Sunil Govindan wrote: >> >> I am +1 to this. >> We found this helped in few cases. >> >> - Sunil >> >> On Tue, Dec 17, 2019 at 11:43 AM Wei-Chiu Chuang >> wrote: >> >>> Similar to INFRA-19327 < >> https://issues.apache.org/jira/browse/INFRA-19327 , >>> I would like to see if the committers are interested in seeing github >>> security vulnerability notification. >>> >>> The Hadoop project has it enabled for all Hadoop committers already. >>> >>> If people think this is a good idea, I will then go ahead and file >> another >>> INFRA jira. Otherwise, every committer can decide to enable notification >>> for him/herself by filing an INFRA jira too. >>> >>> Thanks, >>> Weichiu >>> >> - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[NOTICE] Change of Submarine dev mailing alias
Hi community! In case you didn't realize it like me just 5 minutes ago, the Submarine project has its own independent mailing alias. So instead of sending dev messages to submarine-...@hadoop.apach.org, send it to dev@submarine.apache.org You should also subscribe to the new mailing list too by sending an email to dev-subscr...@submarine.apache.org. Best, Weichiu
Re: [DISCUSS] Enable github security notifications to all Submarine committers
Oh wait, I sent to the old submarine dev mailing alias. The new alias is dev@submarine.apache.org On Mon, Dec 16, 2019 at 10:19 PM Sunil Govindan wrote: > I am +1 to this. > We found this helped in few cases. > > - Sunil > > On Tue, Dec 17, 2019 at 11:43 AM Wei-Chiu Chuang > wrote: > > > Similar to INFRA-19327 < > https://issues.apache.org/jira/browse/INFRA-19327 > > >, > > I would like to see if the committers are interested in seeing github > > security vulnerability notification. > > > > The Hadoop project has it enabled for all Hadoop committers already. > > > > If people think this is a good idea, I will then go ahead and file > another > > INFRA jira. Otherwise, every committer can decide to enable notification > > for him/herself by filing an INFRA jira too. > > > > Thanks, > > Weichiu > > >
[jira] [Commented] (SUBMARINE-226) [Notebook] Initialize workbench notebook framework
[ https://issues.apache.org/jira/browse/SUBMARINE-226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997921#comment-16997921 ] Liu Xun commented on SUBMARINE-226: --- Refactor notebook using anguleJS, so close this jira > [Notebook] Initialize workbench notebook framework > -- > > Key: SUBMARINE-226 > URL: https://issues.apache.org/jira/browse/SUBMARINE-226 > Project: Apache Submarine > Issue Type: Sub-task > Components: Workbench Notebook >Reporter: Liu Xun >Assignee: Liu Xun >Priority: Major > > 1. Create a notebook module in the workbench web > 2. Can open the notebook page in workbench web. > 3. Support markdown for document writing, and support python and spark.sql > for code writing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Resolved] (SUBMARINE-226) [Notebook] Initialize workbench notebook framework
[ https://issues.apache.org/jira/browse/SUBMARINE-226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Xun resolved SUBMARINE-226. --- Resolution: Not A Problem > [Notebook] Initialize workbench notebook framework > -- > > Key: SUBMARINE-226 > URL: https://issues.apache.org/jira/browse/SUBMARINE-226 > Project: Apache Submarine > Issue Type: Sub-task > Components: Workbench Notebook >Reporter: Liu Xun >Assignee: Liu Xun >Priority: Major > > 1. Create a notebook module in the workbench web > 2. Can open the notebook page in workbench web. > 3. Support markdown for document writing, and support python and spark.sql > for code writing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-231) [Interpreter] Introducing zeppelin via submodule
[ https://issues.apache.org/jira/browse/SUBMARINE-231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Xun updated SUBMARINE-231: -- Resolution: Fixed Status: Resolved (was: Patch Available) > [Interpreter] Introducing zeppelin via submodule > > > Key: SUBMARINE-231 > URL: https://issues.apache.org/jira/browse/SUBMARINE-231 > Project: Apache Submarine > Issue Type: Sub-task > Components: Workbench Interpreter >Reporter: Liu Xun >Assignee: Liu Xun >Priority: Major > Attachments: Untitled.png > > Time Spent: 20m > Remaining Estimate: 0h > > # First, fork Apache Zeppelin's code repository in the hadoopsubmarine > account, which ensures that hadoopsubmarine/zeppelin can always follow > apache/zeppelin code updates. > # Create a submarine branch in hadoopsubmarine/zeppelin and customize the > development on this branch. > # The hadoopsubmarine/zeppelin repository is contained via submodule in the > apache/submarine repository. > # In the /submarine/submarine-workbench/submarine-interpreter in the > apache/submarine repository, we import the JAR package of our customized > zeppelin interpreter in submodule for use. > Design doc: > [https://docs.google.com/document/d/1LJc5hRTWfIs6K5f7vMgGC_FUizeQf-qqiWICqO8SoR8/edit#] > !Untitled.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-310) Support golang and java mixed compilation
[ https://issues.apache.org/jira/browse/SUBMARINE-310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Xun updated SUBMARINE-310: -- Resolution: Invalid Status: Resolved (was: Patch Available) > Support golang and java mixed compilation > - > > Key: SUBMARINE-310 > URL: https://issues.apache.org/jira/browse/SUBMARINE-310 > Project: Apache Submarine > Issue Type: Improvement >Reporter: Liu Xun >Assignee: Liu Xun >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Submarine needs to support both the yarn and k8s resource scheduling systems > for job operation. > Some modules (submitter-yarn) are developed using java, and some modules > (submarine-operator / submitter-k8s) are developed using golang. > So submarine needs to support both mvn and go build compilation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-310) Support golang and java mixed compilation
[ https://issues.apache.org/jira/browse/SUBMARINE-310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997918#comment-16997918 ] Liu Xun commented on SUBMARINE-310: --- After discussion with the community, keep the original module structure unchanged, add the submarine-cloud folder to the project root directory, all submarine-cloud folders are developed using the Go language to support submarine on k8s (aws, gce, azure in the future). Therefore, this PR will be closed. > Support golang and java mixed compilation > - > > Key: SUBMARINE-310 > URL: https://issues.apache.org/jira/browse/SUBMARINE-310 > Project: Apache Submarine > Issue Type: Improvement >Reporter: Liu Xun >Assignee: Liu Xun >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Submarine needs to support both the yarn and k8s resource scheduling systems > for job operation. > Some modules (submitter-yarn) are developed using java, and some modules > (submarine-operator / submitter-k8s) are developed using golang. > So submarine needs to support both mvn and go build compilation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-132) [DB] Submarine Database Design
[ https://issues.apache.org/jira/browse/SUBMARINE-132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Xun updated SUBMARINE-132: -- Labels: (was: submarine-server) > [DB] Submarine Database Design > -- > > Key: SUBMARINE-132 > URL: https://issues.apache.org/jira/browse/SUBMARINE-132 > Project: Apache Submarine > Issue Type: Sub-task > Components: Workbench >Reporter: Xun Liu >Assignee: Liu Xun >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Resolved] (SUBMARINE-132) [DB] Submarine Database Design
[ https://issues.apache.org/jira/browse/SUBMARINE-132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Xun resolved SUBMARINE-132. --- Resolution: Fixed > [DB] Submarine Database Design > -- > > Key: SUBMARINE-132 > URL: https://issues.apache.org/jira/browse/SUBMARINE-132 > Project: Apache Submarine > Issue Type: Sub-task > Components: Workbench >Reporter: Xun Liu >Assignee: Liu Xun >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Assigned] (SUBMARINE-25) [Submarine] Manage submarine's docker image with docker hub
[ https://issues.apache.org/jira/browse/SUBMARINE-25?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Xun reassigned SUBMARINE-25: Assignee: Liu Xun (was: Xun Liu) > [Submarine] Manage submarine's docker image with docker hub > --- > > Key: SUBMARINE-25 > URL: https://issues.apache.org/jira/browse/SUBMARINE-25 > Project: Apache Submarine > Issue Type: New Feature > Components: Mini Submarine >Reporter: Xun Liu >Assignee: Liu Xun >Priority: Major > > Currently the deep learning algorithm file is packaged into the docker image > file. > In the formal use of the deep learning system, the algorithm files are > generally divided into the following ways: > 1. The fixed algorithm will be made into a separate docker image, which is > convenient for maintenance and testing. Image files with different algorithms > built in are managed by mirrored warehouse. > 2. We also need to dynamically pass in algorithm scripts so that algorithm > developers can test and run algorithm scripts without packaging. > 3. In addition, submarine needs to support different deep learning frameworks > in the future. > We need to manage the image that submarine needs to use through the docker > hub. And we need to develop some specifications to manage these images. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Resolved] (SUBMARINE-90) Submarine 0.3.0 website
[ https://issues.apache.org/jira/browse/SUBMARINE-90?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang resolved SUBMARINE-90. --- Resolution: Fixed > Submarine 0.3.0 website > --- > > Key: SUBMARINE-90 > URL: https://issues.apache.org/jira/browse/SUBMARINE-90 > Project: Apache Submarine > Issue Type: Sub-task > Components: Doc >Reporter: Xun Liu >Assignee: Xun Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-173) [SC] Create the submarine-client REST module
[ https://issues.apache.org/jira/browse/SUBMARINE-173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-173: --- Target Version: 0.4.0 (was: 0.3.0) > [SC] Create the submarine-client REST module > > > Key: SUBMARINE-173 > URL: https://issues.apache.org/jira/browse/SUBMARINE-173 > Project: Apache Submarine > Issue Type: Sub-task > Components: Backend Server >Reporter: Wanqiang Ji >Assignee: Wanqiang Ji >Priority: Major > > The client as the default implements for submarine server RESTful API, it > provides all the features which declared in the RESTful API. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-173) [SC] Create the submarine-client REST module
[ https://issues.apache.org/jira/browse/SUBMARINE-173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-173: --- Summary: [SC] Create the submarine-client REST module (was: [SC] Create the submarine-client module) > [SC] Create the submarine-client REST module > > > Key: SUBMARINE-173 > URL: https://issues.apache.org/jira/browse/SUBMARINE-173 > Project: Apache Submarine > Issue Type: Sub-task > Components: Backend Server >Reporter: Wanqiang Ji >Assignee: Wanqiang Ji >Priority: Major > > The client as the default implements for submarine server RESTful API, it > provides all the features which declared in the RESTful API. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[GitHub] [submarine] jojochuang commented on a change in pull request #108: SUBMARINE-302. Split the submitter-yarnservice from travis into job
jojochuang commented on a change in pull request #108: SUBMARINE-302. Split the
submitter-yarnservice from travis into job
URL: https://github.com/apache/submarine/pull/108#discussion_r358598764
##
File path: .travis.yml
##
@@ -167,6 +154,18 @@ matrix:
dist: xenial
env: NAME="Test submarine-server" PROFILE="-Phadoop-2.9"
BUILD_FLAG="clean package install -DskipTests" TEST_FLAG="test -DskipRat -am"
MODULES="-pl ${EXCLUDE_K8S},${EXCLUDE_WORKBENCH},${EXCLUDE_INTERPRETER}"
TEST_MODULES="-pl submarine-server/server-core" TEST_PROJECTS=""
+- name: Submarine on Hadoop YARN 3.1
Review comment:
Are we good to resolve this comment? Looks to me we have agreed to resolve
and will raise new jira for additional improvement.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-65) Make PyTorch support HDFS
[ https://issues.apache.org/jira/browse/SUBMARINE-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997890#comment-16997890 ] Wei-Chiu Chuang commented on SUBMARINE-65: -- Not familiar with PyTorch, but you can use PyArrow to access HDFS from Python code. Hope that helps. > Make PyTorch support HDFS > - > > Key: SUBMARINE-65 > URL: https://issues.apache.org/jira/browse/SUBMARINE-65 > Project: Apache Submarine > Issue Type: Sub-task > Components: TonY >Reporter: Szilard Nemeth >Priority: Major > > PyTorch does not support HDFS as a datasource like TensorFlow. > Found this related issue, but nothing meaningful here: > https://github.com/pytorch/pytorch/issues/5867 > I think we should make PyTorch support HDFS by contributing to PyTorch's > source code. > Here is a reference to the TF implementation of HDFS connector: > https://github.com/tensorflow/tensorflow/tree/17e49b339b2b9a58ed967c69b7acb714dcd9b465/tensorflow/core/platform/hadoop > Any other ideas to approach this problem are welcome! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-29) [Submarine] Improve job state check
[
https://issues.apache.org/jira/browse/SUBMARINE-29?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997878#comment-16997878
]
Hadoop QA commented on SUBMARINE-29:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 13s{color}
| {color:red} SUBMARINE-29 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | SUBMARINE-29 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12950304/YARN-9077-trunk.002.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-SUBMARINE-Build/180/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.10.0 http://yetus.apache.org |
This message was automatically generated.
> [Submarine] Improve job state check
> ---
>
> Key: SUBMARINE-29
> URL: https://issues.apache.org/jira/browse/SUBMARINE-29
> Project: Apache Submarine
> Issue Type: New Feature
> Components: YARN-native-service
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>Priority: Major
> Attachments: YARN-9077-trunk.001.patch, YARN-9077-trunk.002.patch
>
>
> Since YARN-8488 gets committed, the service state has "SUCCEEDED" statue. We
> should check it instead of "STOPPED".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-7) [Submarine] Support create models / versions for training result.
[
https://issues.apache.org/jira/browse/SUBMARINE-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997877#comment-16997877
]
Hadoop QA commented on SUBMARINE-7:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 13s{color}
| {color:red} SUBMARINE-7 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | SUBMARINE-7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12974610/SUBMARINE-7.005.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-SUBMARINE-Build/179/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.10.0 http://yetus.apache.org |
This message was automatically generated.
> [Submarine] Support create models / versions for training result.
> --
>
> Key: SUBMARINE-7
> URL: https://issues.apache.org/jira/browse/SUBMARINE-7
> Project: Apache Submarine
> Issue Type: New Feature
> Components: Commons
>Reporter: Wangda Tan
>Assignee: kevin su
>Priority: Major
> Attachments: SUBMARINE-7.005.patch, SUBMARINE-7.v0.patch,
> SUBMARINE-7.v1.patch, SUBMARINE-7.v2.patch, SUBMARINE-7.v3.patch, Screenshot
> from 2019-07-13 00-28-44.png, Screenshot from 2019-07-13 00-29-54.png
>
>
> As mentioned in
> https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#heading=h.vkxp9edl11m7,
> we should be able to have models/versions for models created by training
> algorithm. See design doc for syntax, etc.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-18) Enable local staging directory and clean it up when submarine job is submitted
[
https://issues.apache.org/jira/browse/SUBMARINE-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997876#comment-16997876
]
Hadoop QA commented on SUBMARINE-18:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 12s{color}
| {color:red} SUBMARINE-18 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | SUBMARINE-18 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12941333/YARN-8806.006.patch |
| Console output |
https://builds.apache.org/job/PreCommit-SUBMARINE-Build/178/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.10.0 http://yetus.apache.org |
This message was automatically generated.
> Enable local staging directory and clean it up when submarine job is submitted
> --
>
> Key: SUBMARINE-18
> URL: https://issues.apache.org/jira/browse/SUBMARINE-18
> Project: Apache Submarine
> Issue Type: New Feature
> Components: Commons
> Environment: In the /tmp dir, there are launch scripts which are not
> cleaned up as follows:
> -rw-r--r-- 1 hadoop netease 1100 Sep 18 10:46
> PRIMARY_WORKER-launch-script8635233314077649086.sh
> -rw-r--r-- 1 hadoop netease 1100 Sep 18 10:46
> WORKER-launch-script129488020578466938.sh
> -rw-r--r-- 1 hadoop netease 1028 Sep 18 10:46
> PS-launch-script471092031021738136.sh
>Reporter: Zac Zhou
>Assignee: Zac Zhou
>Priority: Major
> Attachments: YARN-8806.001.patch, YARN-8806.002.patch,
> YARN-8806.003.patch, YARN-8806.004.patch, YARN-8806.005.patch,
> YARN-8806.006.patch
>
>
> YarnServiceJobSubmitter.generateCommandLaunchScript creates container launch
> scripts in the local filesystem. Container launch scripts would be uploaded
> to hdfs staging dir, but would not be not deleted after the job is submitted
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-84) The environment variable TF_CONFIG hard coded port number "8000" will cause distributed training job's worker fail to start
[
https://issues.apache.org/jira/browse/SUBMARINE-84?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997874#comment-16997874
]
Hadoop QA commented on SUBMARINE-84:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
58s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
59s{color} | {color:red} hadoop-submarine-yarnservice-runtime in trunk failed.
{color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 15s{color} | {color:orange} The patch fails to run checkstyle in
hadoop-submarine-yarnservice-runtime {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
16s{color} | {color:red} hadoop-submarine-yarnservice-runtime in trunk failed.
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 22s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
17s{color} | {color:red} hadoop-submarine-yarnservice-runtime in trunk failed.
{color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 14m
57s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
16s{color} | {color:red} hadoop-submarine-yarnservice-runtime in trunk failed.
{color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
10s{color} | {color:red} hadoop-submarine-yarnservice-runtime in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
10s{color} | {color:red} hadoop-submarine-yarnservice-runtime in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 11s{color}
| {color:red} hadoop-submarine-yarnservice-runtime in the patch failed. {color}
|
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 8s{color} | {color:orange} The patch fails to run checkstyle in
hadoop-submarine-yarnservice-runtime {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
10s{color} | {color:red} hadoop-submarine-yarnservice-runtime in the patch
failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 59s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
13s{color} | {color:red} hadoop-submarine-yarnservice-runtime in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
13s{color} | {color:red} hadoop-submarine-yarnservice-runtime in the patch
failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 13s{color}
| {color:red} hadoop-submarine-yarnservice-runtime in the patch failed. {color}
|
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 63m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 base:
https://builds.apache.org/job/PreCommit-SUBMARINE-Build/177/artifact/out/Dockerfile
|
| JIRA Issue | SUBMARINE-84 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12976769/SUBMARINE-84.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux fc7bd79fef17 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019
[jira] [Resolved] (SUBMARINE-2) Hadoop {Submarine} Project: Simple and scalable deployment of deep learning training / serving jobs on Hadoop
[
https://issues.apache.org/jira/browse/SUBMARINE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhankun Tang resolved SUBMARINE-2.
--
Resolution: Fixed
> Hadoop {Submarine} Project: Simple and scalable deployment of deep learning
> training / serving jobs on Hadoop
> -
>
> Key: SUBMARINE-2
> URL: https://issues.apache.org/jira/browse/SUBMARINE-2
> Project: Apache Submarine
> Issue Type: New Feature
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Major
>
> Description:
> *Goals:*
> - Allow infra engineer / data scientist to run *unmodified* Tensorflow jobs
> on YARN.
> - Allow jobs easy access data/models in HDFS and other storages.
> - Can launch services to serve Tensorflow/MXNet models.
> - Support run distributed Tensorflow jobs with simple configs.
> - Support run user-specified Docker images.
> - Support specify GPU and other resources.
> - Support launch tensorboard if user specified.
> - Support customized DNS name for roles (like tensorboard.$user.$domain:6006)
> *Why this name?*
> - Because Submarine is the only vehicle can let human to explore deep
> places. B-)
> h3. {color:#ff}Please refer to on-going design doc, and add your
> thoughts:
> [https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#|https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit?usp=sharing]{color}
>
> Submarine all document list in here:
> https://issues.apache.org/jira/browse/SUBMARINE-113
>
> *{color:#33}See Also:{color}*
> * {color:#33}Zeppelin integration with Submarine design:
> [https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#heading=h.4jov859x47qe]{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Resolved] (SUBMARINE-5) [Submarine] Support users to specify Python/TF package/version/dependencies for training job.
[ https://issues.apache.org/jira/browse/SUBMARINE-5?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang resolved SUBMARINE-5. -- Resolution: Later The current "localization" option (SUBMARINE-4) resolves part of the job. And in the new submarine-server, the generic job spec will consider this. > [Submarine] Support users to specify Python/TF package/version/dependencies > for training job. > - > > Key: SUBMARINE-5 > URL: https://issues.apache.org/jira/browse/SUBMARINE-5 > Project: Apache Submarine > Issue Type: New Feature >Reporter: Wangda Tan >Priority: Major > > YARN-8561 assumes all Python / Tensorflow dependencies will be packed to > docker image. In practice, user doesn't want to build docker image. Instead, > user can provide python package / dependencies (like .whl), Python and TF > version. And Submarine can localize specified dependencies to prebuilt base > Docker images. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-7) [Submarine] Support create models / versions for training result.
[ https://issues.apache.org/jira/browse/SUBMARINE-7?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-7: - Component/s: Commons Target Version: 0.4.0 > [Submarine] Support create models / versions for training result. > -- > > Key: SUBMARINE-7 > URL: https://issues.apache.org/jira/browse/SUBMARINE-7 > Project: Apache Submarine > Issue Type: New Feature > Components: Commons >Reporter: Wangda Tan >Assignee: kevin su >Priority: Major > Attachments: SUBMARINE-7.005.patch, SUBMARINE-7.v0.patch, > SUBMARINE-7.v1.patch, SUBMARINE-7.v2.patch, SUBMARINE-7.v3.patch, Screenshot > from 2019-07-13 00-28-44.png, Screenshot from 2019-07-13 00-29-54.png > > > As mentioned in > https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#heading=h.vkxp9edl11m7, > we should be able to have models/versions for models created by training > algorithm. See design doc for syntax, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Resolved] (SUBMARINE-8) [Submarine] Support deploy model serving for existing models
[
https://issues.apache.org/jira/browse/SUBMARINE-8?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhankun Tang resolved SUBMARINE-8.
--
Resolution: Later
Since the design of the submarine changed a lot after the spin-off. Let's
discuss model serving when needed but close this for now.
> [Submarine] Support deploy model serving for existing models
>
>
> Key: SUBMARINE-8
> URL: https://issues.apache.org/jira/browse/SUBMARINE-8
> Project: Apache Submarine
> Issue Type: New Feature
>Reporter: Wangda Tan
>Priority: Major
>
> See
> https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#heading=h.vkxp9edl11m7
> {{model deploy}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Resolved] (SUBMARINE-10) Submarine job staging directory has a lot of useless PRIMARY_WORKER-launch-script-***.sh scripts when submitting a job multiple times
[
https://issues.apache.org/jira/browse/SUBMARINE-10?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhankun Tang resolved SUBMARINE-10.
---
Resolution: Duplicate
duplicated with SUBMARINE-18
> Submarine job staging directory has a lot of useless
> PRIMARY_WORKER-launch-script-***.sh scripts when submitting a job multiple
> times
> --
>
> Key: SUBMARINE-10
> URL: https://issues.apache.org/jira/browse/SUBMARINE-10
> Project: Apache Submarine
> Issue Type: New Feature
>Reporter: Zac Zhou
>Assignee: Zhankun Tang
>Priority: Major
> Attachments: YARN-8725-trunk.001.patch
>
>
> Submarine jobs upload core-site.xml, hdfs-site.xml, job.info and
> PRIMARY_WORKER-launch-script.sh to staging dir.
> The core-site.xml, hdfs-site.xml and job.info would be overwritten if a job
> is submitted multiple times.
> But PRIMARY_WORKER-launch-script.sh would not be overwritten, as it has
> random numbers in its name.
> The files in the staging dir are as follows：
> {code:java}
> -rw-r- 2 hadoop hdfs 580 2018-08-17 10:11
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script6954941665090337726.sh
> -rw-r- 2 hadoop hdfs 580 2018-08-17 10:02
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script7037369696166769734.sh
> -rw-r- 2 hadoop hdfs 580 2018-08-17 10:06
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script8047707294763488040.sh
> -rw-r- 2 hadoop hdfs 15225 2018-08-17 18:46
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script8122565781159446375.sh
> -rw-r- 2 hadoop hdfs 580 2018-08-16 20:48
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script8598604480700049845.sh
> -rw-r- 2 hadoop hdfs 580 2018-08-17 14:53
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script971703616848859353.sh
> -rw-r- 2 hadoop hdfs 580 2018-08-17 10:16
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script990214235580089093.sh
> -rw-r- 2 hadoop hdfs 8815 2018-08-27 15:54
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/core-site.xml
> -rw-r- 2 hadoop hdfs 11583 2018-08-27 15:54
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/hdfs-site.xml
> -rw-rw-rw- 2 hadoop hdfs 846 2018-08-22 10:56
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/job.info
> {code}
>
> We should stop the staging dir from growing or have a way to clean it up
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-14) [Umbrella] Support using Submarine to submit Pytorch job
[ https://issues.apache.org/jira/browse/SUBMARINE-14?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-14: -- Component/s: Commons Target Version: 0.4.0 > [Umbrella] Support using Submarine to submit Pytorch job > > > Key: SUBMARINE-14 > URL: https://issues.apache.org/jira/browse/SUBMARINE-14 > Project: Apache Submarine > Issue Type: New Feature > Components: Commons >Reporter: Wangda Tan >Assignee: Szilard Nemeth >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-14) [Umbrella] Support using Submarine to submit Pytorch job
[ https://issues.apache.org/jira/browse/SUBMARINE-14?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-14: -- Component/s: Backend Server > [Umbrella] Support using Submarine to submit Pytorch job > > > Key: SUBMARINE-14 > URL: https://issues.apache.org/jira/browse/SUBMARINE-14 > Project: Apache Submarine > Issue Type: New Feature > Components: Backend Server, Commons >Reporter: Wangda Tan >Assignee: Szilard Nemeth >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-18) Enable local staging directory and clean it up when submarine job is submitted
[ https://issues.apache.org/jira/browse/SUBMARINE-18?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-18: -- Component/s: Commons Target Version: 0.4.0 > Enable local staging directory and clean it up when submarine job is submitted > -- > > Key: SUBMARINE-18 > URL: https://issues.apache.org/jira/browse/SUBMARINE-18 > Project: Apache Submarine > Issue Type: New Feature > Components: Commons > Environment: In the /tmp dir, there are launch scripts which are not > cleaned up as follows: > -rw-r--r-- 1 hadoop netease 1100 Sep 18 10:46 > PRIMARY_WORKER-launch-script8635233314077649086.sh > -rw-r--r-- 1 hadoop netease 1100 Sep 18 10:46 > WORKER-launch-script129488020578466938.sh > -rw-r--r-- 1 hadoop netease 1028 Sep 18 10:46 > PS-launch-script471092031021738136.sh >Reporter: Zac Zhou >Assignee: Zac Zhou >Priority: Major > Attachments: YARN-8806.001.patch, YARN-8806.002.patch, > YARN-8806.003.patch, YARN-8806.004.patch, YARN-8806.005.patch, > YARN-8806.006.patch > > > YarnServiceJobSubmitter.generateCommandLaunchScript creates container launch > scripts in the local filesystem. Container launch scripts would be uploaded > to hdfs staging dir, but would not be not deleted after the job is submitted -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Resolved] (SUBMARINE-23) [Submarine] Job monitor long-running service of submarine
[ https://issues.apache.org/jira/browse/SUBMARINE-23?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang resolved SUBMARINE-23. --- Resolution: Won't Fix > [Submarine] Job monitor long-running service of submarine > - > > Key: SUBMARINE-23 > URL: https://issues.apache.org/jira/browse/SUBMARINE-23 > Project: Apache Submarine > Issue Type: New Feature >Reporter: Xun Liu >Assignee: Xun Liu >Priority: Major > > h1. Job monitor long-running service of submarine > After training, the monitoring program need auto close PS service. It is > possible that other deep learning frameworks also have some custom processing > when the tasks are in different states. > The submarine needs to provide a long-term resident service that monitors > each JOB mission. > This monitoring service can be processed differently according to the > training tasks of different depth learning framework types. > For example: Tensorflow performs distributed training, when the training is > completed, > The PS service cannot be automatically stopped. At this time, the PS needs to > be actively stopped by the monitoring service. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Resolved] (SUBMARINE-27) [Submarine] Support for executing tensorflow algorithm files on hdfs
[
https://issues.apache.org/jira/browse/SUBMARINE-27?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhankun Tang resolved SUBMARINE-27.
---
Resolution: Duplicate
> [Submarine] Support for executing tensorflow algorithm files on hdfs
>
>
> Key: SUBMARINE-27
> URL: https://issues.apache.org/jira/browse/SUBMARINE-27
> Project: Apache Submarine
> Issue Type: New Feature
>Reporter: Xun Liu
>Assignee: Xun Liu
>Priority: Major
>
> Now the submarine project can only execute the algorithm files in the docker
> image.
> Need support for executing tensorflow algorithm files on HDFS.
> 1. The user uploads the algorithm file to the HDFS directory manually or via
> zeppelin.
> 2. In the base image provided by submarine, Increase the path parameters
> according to the specified algorithm, Download the algorithm file from HDFS
> to the `/tmp/` directory.
> {code:java}
> hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/main/docker/base/ubuntu-16.04/Dockerfile.cpu.tf_1.8.0
>
> and
> hadoop-yarn-project/hadoop
> -yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/main/docker/base/ubuntu-16.04/Dockerfile.gpu.tf_1.8.0{code}
> 3. Add parameters to the algorithm file path in submarine project.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-25) [Submarine] Manage submarine's docker image with docker hub
[ https://issues.apache.org/jira/browse/SUBMARINE-25?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-25: -- Component/s: Mini Submarine Target Version: 0.3.0 > [Submarine] Manage submarine's docker image with docker hub > --- > > Key: SUBMARINE-25 > URL: https://issues.apache.org/jira/browse/SUBMARINE-25 > Project: Apache Submarine > Issue Type: New Feature > Components: Mini Submarine >Reporter: Xun Liu >Assignee: Xun Liu >Priority: Major > > Currently the deep learning algorithm file is packaged into the docker image > file. > In the formal use of the deep learning system, the algorithm files are > generally divided into the following ways: > 1. The fixed algorithm will be made into a separate docker image, which is > convenient for maintenance and testing. Image files with different algorithms > built in are managed by mirrored warehouse. > 2. We also need to dynamically pass in algorithm scripts so that algorithm > developers can test and run algorithm scripts without packaging. > 3. In addition, submarine needs to support different deep learning frameworks > in the future. > We need to manage the image that submarine needs to use through the docker > hub. And we need to develop some specifications to manage these images. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-29) [Submarine] Improve job state check
[ https://issues.apache.org/jira/browse/SUBMARINE-29?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-29: -- Component/s: YARN-native-service Target Version: 0.4.0 > [Submarine] Improve job state check > --- > > Key: SUBMARINE-29 > URL: https://issues.apache.org/jira/browse/SUBMARINE-29 > Project: Apache Submarine > Issue Type: New Feature > Components: YARN-native-service >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Attachments: YARN-9077-trunk.001.patch, YARN-9077-trunk.002.patch > > > Since YARN-8488 gets committed, the service state has "SUCCEEDED" statue. We > should check it instead of "STOPPED". -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Resolved] (SUBMARINE-36) [Submarine] Support fault tolerance when Tensorflow worker container fails
[ https://issues.apache.org/jira/browse/SUBMARINE-36?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang resolved SUBMARINE-36. --- Resolution: Won't Fix TonY should have already supported this. > [Submarine] Support fault tolerance when Tensorflow worker container fails > -- > > Key: SUBMARINE-36 > URL: https://issues.apache.org/jira/browse/SUBMARINE-36 > Project: Apache Submarine > Issue Type: New Feature >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > > A long-running Tensorflow job needs to restart failed worker containers when > something unexpected happens. Luckily that TF can restore checkpoints and > continue training in a worker, a restart of the worker container seems enough. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Resolved] (SUBMARINE-37) [Submarine] Support long-running service type services
[
https://issues.apache.org/jira/browse/SUBMARINE-37?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhankun Tang resolved SUBMARINE-37.
---
Resolution: Won't Fix
> [Submarine] Support long-running service type services
> --
>
> Key: SUBMARINE-37
> URL: https://issues.apache.org/jira/browse/SUBMARINE-37
> Project: Apache Submarine
> Issue Type: New Feature
>Reporter: Xun Liu
>Priority: Major
>
> At present, YANR only supports batch type tasks. After the batch is
> completed, the task ends and the container will exit.
> Machine learning needs to provide users with tensorboard, model prediction
> online service is a long-running service type of service, currently can
> increase the "{color:#205081}sleep infinity{color}" in the task's start
> command, so that the task has been sleeping until the container does not
> exit, need to be closed Time is terminated manually.
> But this method is not friendly enough or professional, so the YARN should
> add a task type parameter, the named task is batch processing or long-running
> service.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Resolved] (SUBMARINE-39) [Submarine] Improve submarine site contents
[ https://issues.apache.org/jira/browse/SUBMARINE-39?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang resolved SUBMARINE-39. --- Resolution: Fixed After 0.2 got released, I think this has been resolved. Closing this. > [Submarine] Improve submarine site contents > --- > > Key: SUBMARINE-39 > URL: https://issues.apache.org/jira/browse/SUBMARINE-39 > Project: Apache Submarine > Issue Type: Bug >Reporter: Sunil G >Priority: Major > Attachments: SUBMARINE-39.001.patch > > > basic improvements -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-42) Need to support the task to run failed container does not destroy
[ https://issues.apache.org/jira/browse/SUBMARINE-42?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-42: -- Component/s: YARN-native-service Target Version: 0.4.0 > Need to support the task to run failed container does not destroy > - > > Key: SUBMARINE-42 > URL: https://issues.apache.org/jira/browse/SUBMARINE-42 > Project: Apache Submarine > Issue Type: Improvement > Components: YARN-native-service >Reporter: Xun Liu >Priority: Major > > Now through a Submarine job run a tensorflow algorithm, If the JOB fails, All > containers will be destroyed. > So we have no way to locate the problem in the container. So we need YARN to > support one parameter, The container is not destroyed after the task fails. > If you add this feature, please let Yarn-service also support this interface. > Because YARN's REST interface is relatively lightweight and easy to use. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-43) Add a Kubernetes Runtime to Submarine
[ https://issues.apache.org/jira/browse/SUBMARINE-43?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-43: -- Resolution: Duplicate Status: Resolved (was: Patch Available) Close this due to duplicated with SUBMARINE-154 > Add a Kubernetes Runtime to Submarine > - > > Key: SUBMARINE-43 > URL: https://issues.apache.org/jira/browse/SUBMARINE-43 > Project: Apache Submarine > Issue Type: New Feature >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Attachments: SUBMARINE-43.001.patch, SUBMARINE-43.002.patch, > SUBMARINE-43.003.patch > > > The goal of this experimental work is to be able to use Submarine Cli to be > able to deploy the tensor flow job on kubernetes. > The initial patch adds K8S Job Submitter which creates the following > resources: > 1. Pod for Primary Worker > 2. Pod for Workers > 3. Pod for PS > 4. Following resources are created for Tensorboard: >- Deployment for Tensorboard >- Service >- Ingress controller > This is an experimental implementation for using Submarine interface to run > tensorflow on kubernetes. > It lacks important features/tasks > - the status of a particular job or delete it via the Submarine interface. > - tested manually via minikube locally. Unit tests are completely missing. > - was unable to access HDFS in the worker container > - submarine uses yarn cli and so there is a depenedency on yarn. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-48) Format of env vars job run parameter is not documented
[
https://issues.apache.org/jira/browse/SUBMARINE-48?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhankun Tang updated SUBMARINE-48:
--
Component/s: Commons
> Format of env vars job run parameter is not documented
> --
>
> Key: SUBMARINE-48
> URL: https://issues.apache.org/jira/browse/SUBMARINE-48
> Project: Apache Submarine
> Issue Type: Improvement
> Components: Commons
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Fix For: 0.3.0
>
>
> According to the Submarine design doc
> ([https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#heading=h.klmv5hrbgx9l),]
> the environment variables are passed like this to job run:
> {code:java}
> --envs key1=value1,key2=value2
> {code}
> However, this is not straightforward from reading the code and it's not
> documented in
> {{org.apache.hadoop.yarn.submarine.client.cli.RunJobCli#generateOptions}}, so
> users won't see it straight away when executing the help command.
> We need to fix this.
> Some unit tests are also need to be added, as testing envs is not covered at
> all.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Resolved] (SUBMARINE-48) Format of env vars job run parameter is not documented
[
https://issues.apache.org/jira/browse/SUBMARINE-48?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhankun Tang resolved SUBMARINE-48.
---
Fix Version/s: 0.3.0
Resolution: Fixed
Let's resolve this for now. Submarine now is using below formats although
different with the design doc:
--env DOCKER_JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre/ \
--env DOCKER_HADOOP_HDFS_HOME=/app/hadoop-3.2.1 \
--env HADOOP_HOME=/hadoop-3.2.1 \
--env HADOOP_YARN_HOME=/hadoop-3.2.1 \
--env HADOOP_COMMON_HOME=/hadoop-3.2.1 \
--env HADOOP_HDFS_HOME=/hadoop-3.2.1 \
--env HADOOP_CONF_DIR=/hadoop-3.2.1/etc/hadoop \
> Format of env vars job run parameter is not documented
> --
>
> Key: SUBMARINE-48
> URL: https://issues.apache.org/jira/browse/SUBMARINE-48
> Project: Apache Submarine
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Fix For: 0.3.0
>
>
> According to the Submarine design doc
> ([https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#heading=h.klmv5hrbgx9l),]
> the environment variables are passed like this to job run:
> {code:java}
> --envs key1=value1,key2=value2
> {code}
> However, this is not straightforward from reading the code and it's not
> documented in
> {{org.apache.hadoop.yarn.submarine.client.cli.RunJobCli#generateOptions}}, so
> users won't see it straight away when executing the help command.
> We need to fix this.
> Some unit tests are also need to be added, as testing envs is not covered at
> all.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-51) Print short error messages in case of configuration issues
[
https://issues.apache.org/jira/browse/SUBMARINE-51?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhankun Tang updated SUBMARINE-51:
--
Component/s: Commons
Target Version: 0.4.0
> Print short error messages in case of configuration issues
> --
>
> Key: SUBMARINE-51
> URL: https://issues.apache.org/jira/browse/SUBMARINE-51
> Project: Apache Submarine
> Issue Type: Improvement
> Components: Commons
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
>
> We need to "normalize" exceptions of YAML / CLI parser errors: Only print
> short texts (instead of stacktraces) when any issue happens.
> Some requirements (copied from SUBMARINE-47, quoting [~sunilg]):
> {quote}
> In YamlParseException, mostly it is using super methods to print the cause.
> Since submarine is CLI driven, how bulky these exceptions traces will look
> like in console if any failure happens in parsing. Lets me try to cover some
> cases.
> 1. If file doesn't exist, a simple cleaner message in console is enough to
> state that the file is not present or invalid file.
> 2. If file contents are empty or in not a parsable format, we should show
> some more exception details.
> 3. Now coming to some parsing error due to some sections are not correctly
> indented or wrongly given, will we show the line number in exception stating
> that this line has some issues in parsing.?
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-50) Ensure that null values from CLI / YAML configs handled appropriately
[ https://issues.apache.org/jira/browse/SUBMARINE-50?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-50: -- Component/s: Commons Target Version: 0.4.0 > Ensure that null values from CLI / YAML configs handled appropriately > - > > Key: SUBMARINE-50 > URL: https://issues.apache.org/jira/browse/SUBMARINE-50 > Project: Apache Submarine > Issue Type: Improvement > Components: Commons >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > > This is popped up from SUBMARINE-47 > We have a common validation code in: > RunJobParameters#updateParametersByParsedCommandline > Here, we parse the CMD line (or YAML) arguments. > We need to make sure no NPE is thrown and all null values are handled > properly. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-53) Generate Service spec + launch script for distributed PyTorch learning job
[ https://issues.apache.org/jira/browse/SUBMARINE-53?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-53: -- Component/s: YARN-native-service Target Version: 0.4.0 > Generate Service spec + launch script for distributed PyTorch learning job > -- > > Key: SUBMARINE-53 > URL: https://issues.apache.org/jira/browse/SUBMARINE-53 > Project: Apache Submarine > Issue Type: Sub-task > Components: YARN-native-service >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > > More info here: https://pytorch.org/tutorials/intermediate/dist_tuto.html -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-65) Make PyTorch support HDFS
[ https://issues.apache.org/jira/browse/SUBMARINE-65?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-65: -- Component/s: TonY Target Version: 0.4.0 > Make PyTorch support HDFS > - > > Key: SUBMARINE-65 > URL: https://issues.apache.org/jira/browse/SUBMARINE-65 > Project: Apache Submarine > Issue Type: Sub-task > Components: TonY >Reporter: Szilard Nemeth >Priority: Major > > PyTorch does not support HDFS as a datasource like TensorFlow. > Found this related issue, but nothing meaningful here: > https://github.com/pytorch/pytorch/issues/5867 > I think we should make PyTorch support HDFS by contributing to PyTorch's > source code. > Here is a reference to the TF implementation of HDFS connector: > https://github.com/tensorflow/tensorflow/tree/17e49b339b2b9a58ed967c69b7acb714dcd9b465/tensorflow/core/platform/hadoop > Any other ideas to approach this problem are welcome! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-308) Support PyTorch with TonY runtime
[
https://issues.apache.org/jira/browse/SUBMARINE-308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhankun Tang updated SUBMARINE-308:
---
Parent: SUBMARINE-14
Issue Type: Sub-task (was: New Feature)
> Support PyTorch with TonY runtime
> -
>
> Key: SUBMARINE-308
> URL: https://issues.apache.org/jira/browse/SUBMARINE-308
> Project: Apache Submarine
> Issue Type: Sub-task
> Components: Backend Server, Doc
>Reporter: huiyangjian
>Priority: Major
> Attachments: localization-error.jpg, no-pytorch.jpg,
> yarn-mount-error.jpg
>
>
> 1.Hope to support stand-alone and distributed.
> 2.Hopefully, pytorch case will be available on submarine github.
> 3.Referring to the pytorch case in the documentation, the following script
> was run without success.
> {code:java}
> CLASSPATH=`hadoop classpath
> --glob`:/home/bin/hadoop/share/hadoop/yarn/submarine-all-0.3.0-SNAPSHOT-hadoop-3.2.jar
> \
> java org.apache.submarine.client.cli.Cli \
> job run --name ${APP_NAME} \
> --env DOCKER_JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre/ \
> --env DOCKER_HADOOP_HDFS_HOME=/app/hadoop-3.2.1 \
> --env HADOOP_HOME=/hadoop-3.2.1 \
> --env HADOOP_YARN_HOME=/hadoop-3.2.1 \
> --env HADOOP_COMMON_HOME=/hadoop-3.2.1 \
> --env HADOOP_HDFS_HOME=/hadoop-3.2.1 \
> --env HADOOP_CONF_DIR=/hadoop-3.2.1/etc/hadoop \
> --env PYTHONUNBUFFERED="0" \
> --env TZ="Asia/Shanghai" \
> --env YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS=${LOCAL_PATH}/:/home/test \
> --queue dev \
> --input_path hdfs://cluster/user/work/tensorflow/data/ \
> --docker_image
> jx-bd-hadoop13.zeus.lianjia.com:801/runonce/tf-1.13.1-pytorch-0.4-gpu-base:0.0.1
> \
> --num_workers 1 \
> --worker_resources memory=16G,vcores=2,gpu=1 \
> --worker_launch_cmd "export CLASSPATH=\$(/app/hadoop-3.2.1/bin/hadoop
> classpath --glob) && cd /home/test/pth && python ../pth/train_pth.py" \
> --localization /home/local/test/cifar10_estimator:./submarine_algorithm
> --verbose \
> --conf
> tony.containers.resources=/home/bin/hadoop/share/hadoop/yarn/submarine-all-0.3.0-SNAPSHOT-hadoop-3.2.jar
> \
> --conf tony.application.framework=pytorch
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-225) [Umbrella] Submarine Client
[ https://issues.apache.org/jira/browse/SUBMARINE-225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997829#comment-16997829 ] Wanqiang Ji commented on SUBMARINE-225: --- [~tangzhankun] Sure, you are right. When the submarine server works well, we should slim for client. > [Umbrella] Submarine Client > --- > > Key: SUBMARINE-225 > URL: https://issues.apache.org/jira/browse/SUBMARINE-225 > Project: Apache Submarine > Issue Type: New Feature > Components: Backend Server >Reporter: Wanqiang Ji >Priority: Major > > Maybe we can used the SC in the JIRA summary field to mark the task related > to submarine client. Such as [SC] Create the submarine-client module. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-71) [umbrella] Update submarine site
[
https://issues.apache.org/jira/browse/SUBMARINE-71?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhankun Tang updated SUBMARINE-71:
--
Component/s: Doc
> [umbrella] Update submarine site
>
>
> Key: SUBMARINE-71
> URL: https://issues.apache.org/jira/browse/SUBMARINE-71
> Project: Apache Submarine
> Issue Type: Improvement
> Components: Doc
>Reporter: Xun Liu
>Priority: Major
>
> Manage [http://hadoop.apache.org/submarine/] in this umbrella issue each
> release an update to the site.
> Modify the hadoop site as follows
> 1) git clone [https://github.com/apache/hadoop-site.git]
> 2) Run the hadoop site locally
> {code:java}
> cd hadoop-site/submarine/
> vi submarine/config.toml
> baseurl = "http://localhost:1313;
> publishDir = "content"
> # NOTE: in hadoop-site/submarine
> Hugo
> hugo server --disableFastRender
> http://localhost:1313{code}
> You can view the changes to the page through the browser.
> 3) commit
> Restore these 2 configurations in submarine/config.toml
> {code:java}
> baseurl = "/submarine/"
> publishDir = "../content/submarine"
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Resolved] (SUBMARINE-73) Submarine and Tony engine use the same interface
[ https://issues.apache.org/jira/browse/SUBMARINE-73?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang resolved SUBMARINE-73. --- Resolution: Duplicate > Submarine and Tony engine use the same interface > > > Key: SUBMARINE-73 > URL: https://issues.apache.org/jira/browse/SUBMARINE-73 > Project: Apache Submarine > Issue Type: Improvement >Reporter: Xun Liu >Priority: Major > > Need the exact same command run or kill job, etc. > According to the configuration file, Choose to use the tony engine, Still a > submarine engine. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-75) Add tests to HadoopEnvironmentSetup class
[ https://issues.apache.org/jira/browse/SUBMARINE-75?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-75: -- Component/s: Commons Target Version: 0.4.0 > Add tests to HadoopEnvironmentSetup class > - > > Key: SUBMARINE-75 > URL: https://issues.apache.org/jira/browse/SUBMARINE-75 > Project: Apache Submarine > Issue Type: Improvement > Components: Commons >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-74) Add tests to TensorFlowServiceSpec class
[ https://issues.apache.org/jira/browse/SUBMARINE-74?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-74: -- Component/s: Commons Target Version: 0.4.0 > Add tests to TensorFlowServiceSpec class > > > Key: SUBMARINE-74 > URL: https://issues.apache.org/jira/browse/SUBMARINE-74 > Project: Apache Submarine > Issue Type: Improvement > Components: Commons >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-76) Create factory class to produce either TensorFlow or PyTorch service spec for Native Service
[ https://issues.apache.org/jira/browse/SUBMARINE-76?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-76: -- Component/s: YARN-native-service Target Version: 0.4.0 > Create factory class to produce either TensorFlow or PyTorch service spec for > Native Service > > > Key: SUBMARINE-76 > URL: https://issues.apache.org/jira/browse/SUBMARINE-76 > Project: Apache Submarine > Issue Type: Improvement > Components: YARN-native-service >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > > In YarnServiceJobSubmitter#submitJob, we submit either a TF / PyTorch job > based on the Service spec. > The code could be improved by creating a factory class to produce the service > spec as the 2 types of Service specs are having very similar dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-77) Create a builder class for NS Component
[ https://issues.apache.org/jira/browse/SUBMARINE-77?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-77: -- Component/s: Commons Target Version: 0.4.0 > Create a builder class for NS Component > --- > > Key: SUBMARINE-77 > URL: https://issues.apache.org/jira/browse/SUBMARINE-77 > Project: Apache Submarine > Issue Type: Improvement > Components: Commons >Reporter: Szilard Nemeth >Assignee: Adam Antal >Priority: Major > > In TensorFlowPsComponent#createComponent, we create a component object. > According to [~adam.antal]'s comment given on one of my resolved jiras, this > method shouts for a builder that builds the Component object. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-78) Make RemoteDirectoryManager interface more consistent
[ https://issues.apache.org/jira/browse/SUBMARINE-78?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-78: -- Component/s: Commons Target Version: 0.4.0 > Make RemoteDirectoryManager interface more consistent > - > > Key: SUBMARINE-78 > URL: https://issues.apache.org/jira/browse/SUBMARINE-78 > Project: Apache Submarine > Issue Type: Improvement > Components: Commons >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > > RemoteDirectoryManager contains many methods that receive a URI. > The types are sometimes Strings, sometimes Path. > We need to make this more consistent. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-79) Cleanup MockRemoteDirectoryManager
[
https://issues.apache.org/jira/browse/SUBMARINE-79?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhankun Tang updated SUBMARINE-79:
--
Component/s: Commons
Target Version: 0.4.0
> Cleanup MockRemoteDirectoryManager
> --
>
> Key: SUBMARINE-79
> URL: https://issues.apache.org/jira/browse/SUBMARINE-79
> Project: Apache Submarine
> Issue Type: Improvement
> Components: Commons
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
>
> Many methods in MockRemoteDirectoryManager are declaring unnecessary throws
> clause for IOException.
> Also, there are 2 dangerous NPE candidates:
> 1. MockRemoteDirectoryManager#getJobStagingArea:
> {code:java}
> this.jobDir = new File(jobsParentDir.getAbsolutePath(), jobName);
> {code}
> If jobsParentDir is null, we could have an NPE easily.
> 2. MockRemoteDirectoryManager#getModelDir:
> {code:java}
> File modelDir = new File(modelParentDir.getAbsolutePath(), modelName);
> {code}
> If modelParentDir is null, we could have an NPE easily.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-81) Add documentation about YAML config parser with example config
[ https://issues.apache.org/jira/browse/SUBMARINE-81?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-81: -- Component/s: Doc Target Version: 0.4.0 > Add documentation about YAML config parser with example config > -- > > Key: SUBMARINE-81 > URL: https://issues.apache.org/jira/browse/SUBMARINE-81 > Project: Apache Submarine > Issue Type: Bug > Components: Doc >Reporter: Szilard Nemeth >Assignee: Adam Antal >Priority: Major > > YAML parser was added but there's no example documentation of a valid > configuration. > We need to fix this! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-84) The environment variable TF_CONFIG hard coded port number "8000" will cause distributed training job's worker fail to start
[ https://issues.apache.org/jira/browse/SUBMARINE-84?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-84: -- Component/s: YARN-native-service Target Version: 0.4.0 > The environment variable TF_CONFIG hard coded port number "8000" will cause > distributed training job's worker fail to start > --- > > Key: SUBMARINE-84 > URL: https://issues.apache.org/jira/browse/SUBMARINE-84 > Project: Apache Submarine > Issue Type: Improvement > Components: YARN-native-service >Reporter: Zhankun Tang >Assignee: kevin su >Priority: Critical > Attachments: SUBMARINE-84.001.patch > > > When there's no network virtualization solution like Calico deployed, the > hard-coded 8000 port conflicts may cause worker/ps container gRPC server > fails to start if they were allocated to one host and use host network. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-90) Submarine 0.3.0 website
[ https://issues.apache.org/jira/browse/SUBMARINE-90?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-90: -- Component/s: Doc Target Version: 0.3.0 > Submarine 0.3.0 website > --- > > Key: SUBMARINE-90 > URL: https://issues.apache.org/jira/browse/SUBMARINE-90 > Project: Apache Submarine > Issue Type: Sub-task > Components: Doc >Reporter: Xun Liu >Assignee: Xun Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-95) Add Docker image for developers and beginners to try submarine in one container
[
https://issues.apache.org/jira/browse/SUBMARINE-95?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhankun Tang updated SUBMARINE-95:
--
Component/s: Mini Submarine
Target Version: 0.3.0
> Add Docker image for developers and beginners to try submarine in one
> container
> ---
>
> Key: SUBMARINE-95
> URL: https://issues.apache.org/jira/browse/SUBMARINE-95
> Project: Apache Submarine
> Issue Type: Improvement
> Components: Mini Submarine
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>Priority: Major
> Attachments: SUBMARINE-95.001.patch, SUBMARINE-95.002.patch
>
>
> Our users may be new to Hadoop. And many of our developers are not willing to
> set up hadoop and maintain it. I propose a docker image dedicated for
> try/develop submarine in one container.
> The image contains vanilla Hadoop, spark and submarine. When running, below
> commands can give a sense of how it's like:
>
> {code:java}
> build_submarine_dev_env.sh
> docker run -it -h submarine-dev --net=bridge --privileged
> local/hadoop-docker:submarine /bin/bash
> /tmp/hadoop-config/bootstrap.sh
> su yarn && cd && cd submarine && ./run_submarine_minist_tony.sh{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-95) Add Docker image for developers and beginners to try submarine in one container
[
https://issues.apache.org/jira/browse/SUBMARINE-95?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhankun Tang updated SUBMARINE-95:
--
Fix Version/s: 0.3.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Add Docker image for developers and beginners to try submarine in one
> container
> ---
>
> Key: SUBMARINE-95
> URL: https://issues.apache.org/jira/browse/SUBMARINE-95
> Project: Apache Submarine
> Issue Type: Improvement
> Components: Mini Submarine
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>Priority: Major
> Fix For: 0.3.0
>
> Attachments: SUBMARINE-95.001.patch, SUBMARINE-95.002.patch
>
>
> Our users may be new to Hadoop. And many of our developers are not willing to
> set up hadoop and maintain it. I propose a docker image dedicated for
> try/develop submarine in one container.
> The image contains vanilla Hadoop, spark and submarine. When running, below
> commands can give a sense of how it's like:
>
> {code:java}
> build_submarine_dev_env.sh
> docker run -it -h submarine-dev --net=bridge --privileged
> local/hadoop-docker:submarine /bin/bash
> /tmp/hadoop-config/bootstrap.sh
> su yarn && cd && cd submarine && ./run_submarine_minist_tony.sh{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-105) [WEB] Submarine web demo
[ https://issues.apache.org/jira/browse/SUBMARINE-105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-105: --- Component/s: Workbench Web Target Version: 0.4.0 > [WEB] Submarine web demo > > > Key: SUBMARINE-105 > URL: https://issues.apache.org/jira/browse/SUBMARINE-105 > Project: Apache Submarine > Issue Type: Sub-task > Components: Workbench Web >Reporter: Xun Liu >Assignee: Xun Liu >Priority: Major > > Operation flow demo of the entire submarine web -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-113) [Document] Submarine all document in here
[
https://issues.apache.org/jira/browse/SUBMARINE-113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhankun Tang updated SUBMARINE-113:
---
Component/s: Doc
> [Document] Submarine all document in here
> -
>
> Key: SUBMARINE-113
> URL: https://issues.apache.org/jira/browse/SUBMARINE-113
> Project: Apache Submarine
> Issue Type: New Feature
> Components: Doc
>Reporter: Xun Liu
>Assignee: Xun Liu
>Priority: Major
>
> {color:#d04437}Long-term update{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-115) [WEB] Submarine web Model page
[ https://issues.apache.org/jira/browse/SUBMARINE-115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-115: --- Component/s: Workbench Web Target Version: 0.4.0 > [WEB] Submarine web Model page > -- > > Key: SUBMARINE-115 > URL: https://issues.apache.org/jira/browse/SUBMARINE-115 > Project: Apache Submarine > Issue Type: Sub-task > Components: Workbench Web >Reporter: Xun Liu >Assignee: Xun Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-116) [WEB] Submarine web Model serving page
[ https://issues.apache.org/jira/browse/SUBMARINE-116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-116: --- Component/s: Workbench Web Target Version: 0.4.0 > [WEB] Submarine web Model serving page > -- > > Key: SUBMARINE-116 > URL: https://issues.apache.org/jira/browse/SUBMARINE-116 > Project: Apache Submarine > Issue Type: Sub-task > Components: Workbench Web >Reporter: Xun Liu >Assignee: Xun Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-98) [Umbrella] Submarine web page
[ https://issues.apache.org/jira/browse/SUBMARINE-98?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-98: -- Target Version: 0.4.0 > [Umbrella] Submarine web page > - > > Key: SUBMARINE-98 > URL: https://issues.apache.org/jira/browse/SUBMARINE-98 > Project: Apache Submarine > Issue Type: New Feature > Components: Workbench Web >Reporter: Xun Liu >Assignee: Hao Zhou >Priority: Major > Labels: submarine-external-repo > > Submarine can be managed via submarine web: > 1. Support multi-user and multi-role > 2. Support for allocation management of server resources > 3. Support data development, algorithm development, and model launch through > notebook > 4. Support workflow workflow orchestration > 5. Support service monitoring after the model goes online -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-321) Add JobManager and SubmitterManager components
[ https://issues.apache.org/jira/browse/SUBMARINE-321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wanqiang Ji updated SUBMARINE-321: -- Status: Patch Available (was: Open) > Add JobManager and SubmitterManager components > -- > > Key: SUBMARINE-321 > URL: https://issues.apache.org/jira/browse/SUBMARINE-321 > Project: Apache Submarine > Issue Type: Sub-task > Components: Backend Server >Reporter: Wanqiang Ji >Assignee: Wanqiang Ji >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The JobManager is responsible for cache and schedule the job to submitter. > The SubmitterManager help to load the different submitter plugins. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Resolved] (SUBMARINE-122) Fixed package name `org.apache.hadoop` error in the documentation
[ https://issues.apache.org/jira/browse/SUBMARINE-122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang resolved SUBMARINE-122. Fix Version/s: 0.3.0 Resolution: Fixed > Fixed package name `org.apache.hadoop` error in the documentation > - > > Key: SUBMARINE-122 > URL: https://issues.apache.org/jira/browse/SUBMARINE-122 > Project: Apache Submarine > Issue Type: Bug > Components: Doc >Reporter: Xun Liu >Assignee: Xun Liu >Priority: Major > Labels: submarine-external-repo > Fix For: 0.3.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-122) Fixed package name `org.apache.hadoop` error in the documentation
[ https://issues.apache.org/jira/browse/SUBMARINE-122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-122: --- Component/s: Doc Target Version: 0.3.0 > Fixed package name `org.apache.hadoop` error in the documentation > - > > Key: SUBMARINE-122 > URL: https://issues.apache.org/jira/browse/SUBMARINE-122 > Project: Apache Submarine > Issue Type: Bug > Components: Doc >Reporter: Xun Liu >Assignee: Xun Liu >Priority: Major > Labels: submarine-external-repo > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-132) [DB] Submarine Database Design
[ https://issues.apache.org/jira/browse/SUBMARINE-132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-132: --- Component/s: Workbench Target Version: 0.3.0 > [DB] Submarine Database Design > -- > > Key: SUBMARINE-132 > URL: https://issues.apache.org/jira/browse/SUBMARINE-132 > Project: Apache Submarine > Issue Type: Sub-task > Components: Workbench >Reporter: Xun Liu >Assignee: Xun Liu >Priority: Major > Labels: submarine-server > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-132) [DB] Submarine Database Design
[ https://issues.apache.org/jira/browse/SUBMARINE-132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997824#comment-16997824 ] Zhankun Tang commented on SUBMARINE-132: [~liuxun323], it seems all the tasks in the umbrella are done except this one. Can we resolve it? > [DB] Submarine Database Design > -- > > Key: SUBMARINE-132 > URL: https://issues.apache.org/jira/browse/SUBMARINE-132 > Project: Apache Submarine > Issue Type: Sub-task >Reporter: Xun Liu >Assignee: Xun Liu >Priority: Major > Labels: submarine-server > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-160) Move Restful code to new workbench service class
[
https://issues.apache.org/jira/browse/SUBMARINE-160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhankun Tang updated SUBMARINE-160:
---
Component/s: Workbench
Target Version: 0.3.0
> Move Restful code to new workbench service class
>
>
> Key: SUBMARINE-160
> URL: https://issues.apache.org/jira/browse/SUBMARINE-160
> Project: Apache Submarine
> Issue Type: Sub-task
> Components: Workbench
>Reporter: Xun Liu
>Assignee: zhulinhao
>Priority: Major
> Labels: submarine-external-repo
>
> Move the database operation logic, from Restful to the newly created service
> class.
> e.g.,
> Move
>
> {code:java}
> https://github.com/hadoopsubmarine/submarine/blob/master/submarine-server/src/main/java/org/apache/submarine/rest/SysUserRestApi.java
> {code}
> TO
>
>
> {code:java}
> https://github.com/hadoopsubmarine/submarine/blob/master/submarine-server/src/main/java/org/apache/submarine/database/service/SysUserService.java
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-160) Move Restful code to new workbench service class
[
https://issues.apache.org/jira/browse/SUBMARINE-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997823#comment-16997823
]
Zhankun Tang commented on SUBMARINE-160:
[~liuxun323], is this already implemented in the commons module? Can we resolve
it?
> Move Restful code to new workbench service class
>
>
> Key: SUBMARINE-160
> URL: https://issues.apache.org/jira/browse/SUBMARINE-160
> Project: Apache Submarine
> Issue Type: Sub-task
>Reporter: Xun Liu
>Assignee: zhulinhao
>Priority: Major
> Labels: submarine-external-repo
>
> Move the database operation logic, from Restful to the newly created service
> class.
> e.g.,
> Move
>
> {code:java}
> https://github.com/hadoopsubmarine/submarine/blob/master/submarine-server/src/main/java/org/apache/submarine/rest/SysUserRestApi.java
> {code}
> TO
>
>
> {code:java}
> https://github.com/hadoopsubmarine/submarine/blob/master/submarine-server/src/main/java/org/apache/submarine/database/service/SysUserService.java
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-162) [Umbrella] Submarine Job Spec
[ https://issues.apache.org/jira/browse/SUBMARINE-162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997821#comment-16997821 ] Zhankun Tang commented on SUBMARINE-162: [~jiwq], is thhis duplicated with SUBMARINE-267? > [Umbrella] Submarine Job Spec > - > > Key: SUBMARINE-162 > URL: https://issues.apache.org/jira/browse/SUBMARINE-162 > Project: Apache Submarine > Issue Type: Sub-task >Reporter: Wanqiang Ji >Priority: Major > > As we all know the K8s used the YAML file as the job spec and YARN Services > used JSON. I propose the submarine can supports the two format. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-186) [E2E] Create new module submarine-integration Initialize E2E Test Framework
[ https://issues.apache.org/jira/browse/SUBMARINE-186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-186: --- Component/s: E2E Test Target Version: 0.4.0 > [E2E] Create new module submarine-integration Initialize E2E Test Framework > --- > > Key: SUBMARINE-186 > URL: https://issues.apache.org/jira/browse/SUBMARINE-186 > Project: Apache Submarine > Issue Type: Sub-task > Components: E2E Test >Reporter: Xun Liu >Assignee: HSIN YUAN, CHEN >Priority: Major > > 1. Create submarine-integration module in submarine project. > 2. Refer to the system implementation of > https://github.com/apache/zeppelin/tree/master/zeppelin-integration to > initialize the WEB simulator in submarine-integration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-173) [SC] Create the submarine-client module
[ https://issues.apache.org/jira/browse/SUBMARINE-173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-173: --- Component/s: Backend Server Target Version: 0.3.0 > [SC] Create the submarine-client module > --- > > Key: SUBMARINE-173 > URL: https://issues.apache.org/jira/browse/SUBMARINE-173 > Project: Apache Submarine > Issue Type: Sub-task > Components: Backend Server >Reporter: Wanqiang Ji >Assignee: Wanqiang Ji >Priority: Major > > The client as the default implements for submarine server RESTful API, it > provides all the features which declared in the RESTful API. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-188) [E2E] Submarine integration test workbench user
[ https://issues.apache.org/jira/browse/SUBMARINE-188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-188: --- Component/s: E2E Test Target Version: 0.4.0 > [E2E] Submarine integration test workbench user > --- > > Key: SUBMARINE-188 > URL: https://issues.apache.org/jira/browse/SUBMARINE-188 > Project: Apache Submarine > Issue Type: Sub-task > Components: E2E Test >Reporter: Xun Liu >Priority: Major > > In the submarine-integration module, the correctness of the user front-end > operation in the submarine-workbench is tested by the WEB simulator. > 1. add user > 2. Modify user > 3. delete user > 4. query user -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-223) The Ecosystem page links to the old, non-Apache github repo
[ https://issues.apache.org/jira/browse/SUBMARINE-223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-223: --- Component/s: Doc Target Version: 0.3.0 > The Ecosystem page links to the old, non-Apache github repo > --- > > Key: SUBMARINE-223 > URL: https://issues.apache.org/jira/browse/SUBMARINE-223 > Project: Apache Submarine > Issue Type: Task > Components: Doc >Reporter: Wei-Chiu Chuang >Priority: Major > Labels: newbie > > https://hadoop.apache.org/submarine/ecosystem/ > The links to the Submarine Installer and Zeppelin Submarine Interpreter > points to the non-Apache repo https://github.com/hadoopsubmarine. > We should update the link to point them to > https://github.com/apache/hadoop-submarine -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-225) [Umbrella] Submarine Client
[ https://issues.apache.org/jira/browse/SUBMARINE-225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-225: --- Description: Maybe we can used the SC in the JIRA summary field to mark the task related to submarine client. Such as [SC] Create the submarine-client module. (was: This is a milestone. Maybe we can used the SC in the JIRA summary field to mark the task related to submarine client. Such as [SC] Create the submarine-client module.) > [Umbrella] Submarine Client > --- > > Key: SUBMARINE-225 > URL: https://issues.apache.org/jira/browse/SUBMARINE-225 > Project: Apache Submarine > Issue Type: New Feature > Components: Backend Server >Reporter: Wanqiang Ji >Priority: Major > > Maybe we can used the SC in the JIRA summary field to mark the task related > to submarine client. Such as [SC] Create the submarine-client module. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-225) [Umbrella] Submarine Client
[ https://issues.apache.org/jira/browse/SUBMARINE-225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated SUBMARINE-225: --- Component/s: Backend Server Target Version: 0.3.0 > [Umbrella] Submarine Client > --- > > Key: SUBMARINE-225 > URL: https://issues.apache.org/jira/browse/SUBMARINE-225 > Project: Apache Submarine > Issue Type: New Feature > Components: Backend Server >Reporter: Wanqiang Ji >Priority: Major > > This is a milestone. Maybe we can used the SC in the JIRA summary field to > mark the task related to submarine client. Such as [SC] Create the > submarine-client module. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-244) Submarine Releases Milestones and Planning
[ https://issues.apache.org/jira/browse/SUBMARINE-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997803#comment-16997803 ] Zhankun Tang commented on SUBMARINE-244: [~wangda], Thanks for updating the wiki page. Let's close this JIRA. > Submarine Releases Milestones and Planning > -- > > Key: SUBMARINE-244 > URL: https://issues.apache.org/jira/browse/SUBMARINE-244 > Project: Apache Submarine > Issue Type: Improvement >Reporter: Zhankun Tang >Priority: Major > > This is a long-live thread. > We use a google spreadsheet to define the releases milestones and track the > details. Everybody can see the key tasks, where we are and where we are going. > The yellow color shows upcoming tasks. The green color shows in-progress. The > grey color shows completed tasks. The purple bar marks today. > > [https://docs.google.com/spreadsheets/d/1HXA3JKMw1u3hl8nwfuT62ZtClUcFNPiraA_q5nr6v7Y/edit?usp=sharing] > Generally the contributors needs to insert row, fill in the category, JIRA#, > estimated days and start time. The sheet can calculate the end time and draw > the bar chart automatically. > Note if the start time is in the future, please choose "Upcoming" in the > status dropdown box. If it is finished or in-progress, please choose the > corresponding status. If the the bar chart is red, it means that the time or > status is not right (or maybe the formula in cell has bug). > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Resolved] (SUBMARINE-244) Submarine Releases Milestones and Planning
[ https://issues.apache.org/jira/browse/SUBMARINE-244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang resolved SUBMARINE-244. Resolution: Fixed Let's use the wiki page with Jira filter statistics. > Submarine Releases Milestones and Planning > -- > > Key: SUBMARINE-244 > URL: https://issues.apache.org/jira/browse/SUBMARINE-244 > Project: Apache Submarine > Issue Type: Improvement >Reporter: Zhankun Tang >Priority: Major > > This is a long-live thread. > We use a google spreadsheet to define the releases milestones and track the > details. Everybody can see the key tasks, where we are and where we are going. > The yellow color shows upcoming tasks. The green color shows in-progress. The > grey color shows completed tasks. The purple bar marks today. > > [https://docs.google.com/spreadsheets/d/1HXA3JKMw1u3hl8nwfuT62ZtClUcFNPiraA_q5nr6v7Y/edit?usp=sharing] > Generally the contributors needs to insert row, fill in the category, JIRA#, > estimated days and start time. The sheet can calculate the end time and draw > the bar chart automatically. > Note if the start time is in the future, please choose "Upcoming" in the > status dropdown box. If it is finished or in-progress, please choose the > corresponding status. If the the bar chart is red, it means that the time or > status is not right (or maybe the formula in cell has bug). > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Work logged] (SUBMARINE-321) Add JobManager and SubmitterManager components
[ https://issues.apache.org/jira/browse/SUBMARINE-321?focusedWorklogId=360662=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360662 ] ASF GitHub Bot logged work on SUBMARINE-321: Author: ASF GitHub Bot Created on: 17/Dec/19 02:43 Start Date: 17/Dec/19 02:43 Worklog Time Spent: 10m Work Description: jiwq commented on pull request #130: SUBMARINE-321. Add JobManager and SubmitterManager components URL: https://github.com/apache/submarine/pull/130 ### What is this PR for? The JobManager is responsible for cache and schedule the job to submitter. The SubmitterManager help to load the different submitter plugins. ### What type of PR is it? [ Feature ] ### Todos * [ ] - Refactor submitter ### What is the Jira issue? https://issues.apache.org/jira/browse/SUBMARINE-321 ### How should this be tested? ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 360662) Remaining Estimate: 0h Time Spent: 10m > Add JobManager and SubmitterManager components > -- > > Key: SUBMARINE-321 > URL: https://issues.apache.org/jira/browse/SUBMARINE-321 > Project: Apache Submarine > Issue Type: Sub-task > Components: Backend Server >Reporter: Wanqiang Ji >Assignee: Wanqiang Ji >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The JobManager is responsible for cache and schedule the job to submitter. > The SubmitterManager help to load the different submitter plugins. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-321) Add JobManager and SubmitterManager components
[ https://issues.apache.org/jira/browse/SUBMARINE-321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SUBMARINE-321: - Labels: pull-request-available (was: ) > Add JobManager and SubmitterManager components > -- > > Key: SUBMARINE-321 > URL: https://issues.apache.org/jira/browse/SUBMARINE-321 > Project: Apache Submarine > Issue Type: Sub-task > Components: Backend Server >Reporter: Wanqiang Ji >Assignee: Wanqiang Ji >Priority: Major > Labels: pull-request-available > > The JobManager is responsible for cache and schedule the job to submitter. > The SubmitterManager help to load the different submitter plugins. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[GitHub] [submarine] jiwq opened a new pull request #130: SUBMARINE-321. Add JobManager and SubmitterManager components
jiwq opened a new pull request #130: SUBMARINE-321. Add JobManager and SubmitterManager components URL: https://github.com/apache/submarine/pull/130 ### What is this PR for? The JobManager is responsible for cache and schedule the job to submitter. The SubmitterManager help to load the different submitter plugins. ### What type of PR is it? [ Feature ] ### Todos * [ ] - Refactor submitter ### What is the Jira issue? https://issues.apache.org/jira/browse/SUBMARINE-321 ### How should this be tested? ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Created] (SUBMARINE-321) Add JobManager and SubmitterManager components
Wanqiang Ji created SUBMARINE-321: - Summary: Add JobManager and SubmitterManager components Key: SUBMARINE-321 URL: https://issues.apache.org/jira/browse/SUBMARINE-321 Project: Apache Submarine Issue Type: Sub-task Components: Backend Server Reporter: Wanqiang Ji Assignee: Wanqiang Ji The JobManager is responsible for cache and schedule the job to submitter. The SubmitterManager help to load the different submitter plugins. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Resolved] (SUBMARINE-258) [SS]Add k8s job submitter in submarine server to run TF job on k8s
[ https://issues.apache.org/jira/browse/SUBMARINE-258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wanqiang Ji resolved SUBMARINE-258. --- Resolution: Duplicate > [SS]Add k8s job submitter in submarine server to run TF job on k8s > -- > > Key: SUBMARINE-258 > URL: https://issues.apache.org/jira/browse/SUBMARINE-258 > Project: Apache Submarine > Issue Type: Sub-task > Components: K8s Submitter >Reporter: Zhankun Tang >Assignee: Wanqiang Ji >Priority: Major > > As a PoC, we need several below steps to run the TF job on k8s. > # The submarine server can run as an independent daemon > # tf-operator deployed in a k8s cluster. Use kubeflow/tf-operator for now > # REST API in submarine server to accept a generic TF job spec > # submitter-k8s translate the spec into tf-operator format and use k8s > client API to submit this to the cluster > # REST API to get tf job status from k8s and parse it to a standard format > to be consumed -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-258) [SS]Add k8s job submitter in submarine server to run TF job on k8s
[ https://issues.apache.org/jira/browse/SUBMARINE-258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997405#comment-16997405 ] Wanqiang Ji commented on SUBMARINE-258: --- Thanks [~wangda] reminding, I will close this Jira with duplicate resolution. > [SS]Add k8s job submitter in submarine server to run TF job on k8s > -- > > Key: SUBMARINE-258 > URL: https://issues.apache.org/jira/browse/SUBMARINE-258 > Project: Apache Submarine > Issue Type: Sub-task > Components: K8s Submitter >Reporter: Zhankun Tang >Assignee: Wanqiang Ji >Priority: Major > > As a PoC, we need several below steps to run the TF job on k8s. > # The submarine server can run as an independent daemon > # tf-operator deployed in a k8s cluster. Use kubeflow/tf-operator for now > # REST API in submarine server to accept a generic TF job spec > # submitter-k8s translate the spec into tf-operator format and use k8s > client API to submit this to the cluster > # REST API to get tf job status from k8s and parse it to a standard format > to be consumed -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[GitHub] [submarine] liuxunorg commented on issue #127: SUBMARINE-319. kind-cluster-build.sh doesn't invoke installed kind
liuxunorg commented on issue #127: SUBMARINE-319. kind-cluster-build.sh doesn't invoke installed kind URL: https://github.com/apache/submarine/pull/127#issuecomment-565980009 Will merge if no more comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[GitHub] [submarine] Johnny850807 commented on issue #121: Tracking ticket for folks to be added to slack group
Johnny850807 commented on issue #121: Tracking ticket for folks to be added to slack group URL: https://github.com/apache/submarine/issues/121#issuecomment-565959048 Thank you! johnny850...@gmail.com This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-131) [Umbrella] Submarine workbench server
[ https://issues.apache.org/jira/browse/SUBMARINE-131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-131: - Labels: (was: submarine-external-repo) > [Umbrella] Submarine workbench server > - > > Key: SUBMARINE-131 > URL: https://issues.apache.org/jira/browse/SUBMARINE-131 > Project: Apache Submarine > Issue Type: New Feature >Reporter: Xun Liu >Assignee: zhulinhao >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-131) [Umbrella] Submarine workbench server
[ https://issues.apache.org/jira/browse/SUBMARINE-131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-131: - Component/s: Workbench > [Umbrella] Submarine workbench server > - > > Key: SUBMARINE-131 > URL: https://issues.apache.org/jira/browse/SUBMARINE-131 > Project: Apache Submarine > Issue Type: New Feature > Components: Workbench >Reporter: Xun Liu >Assignee: zhulinhao >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-148) [Umbrella] Submarine-sdk tracking
[ https://issues.apache.org/jira/browse/SUBMARINE-148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-148: - Target Version: 0.4.0 > [Umbrella] Submarine-sdk tracking > - > > Key: SUBMARINE-148 > URL: https://issues.apache.org/jira/browse/SUBMARINE-148 > Project: Apache Submarine > Issue Type: New Feature > Components: SDK >Reporter: kevin su >Assignee: kevin su >Priority: Major > > The submarine-sdk support Python, Scala, R language for algorithm > development, The SDK is provided to help developers use submarine's internal > data caching, data exchange, and task tracking to more efficiently improve > the development and execution of machine learning tasks. > > pysubmarine *tracking* module is a python API for logging parameters, code > versions, metrics > 1. Add the sumbrine-sdk/pysubmarine module. > 2. pysubmarine supports mysql/HDFS in two ways to store metric data. > Priority to do mysql, in order to performance requires python to directly > write mysql data, do not take the submarine-sever RESTful interface. > 3. Add metric, param instance to the submarine-server database and rest > 4. Add Metric tracking ui to submarine-web > 5. Add a metric graphical display to submarine-web -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-148) [Umbrella] Submarine-sdk tracking
[ https://issues.apache.org/jira/browse/SUBMARINE-148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-148: - Labels: (was: submarine-external-repo) > [Umbrella] Submarine-sdk tracking > - > > Key: SUBMARINE-148 > URL: https://issues.apache.org/jira/browse/SUBMARINE-148 > Project: Apache Submarine > Issue Type: New Feature >Reporter: kevin su >Assignee: kevin su >Priority: Major > > The submarine-sdk support Python, Scala, R language for algorithm > development, The SDK is provided to help developers use submarine's internal > data caching, data exchange, and task tracking to more efficiently improve > the development and execution of machine learning tasks. > > pysubmarine *tracking* module is a python API for logging parameters, code > versions, metrics > 1. Add the sumbrine-sdk/pysubmarine module. > 2. pysubmarine supports mysql/HDFS in two ways to store metric data. > Priority to do mysql, in order to performance requires python to directly > write mysql data, do not take the submarine-sever RESTful interface. > 3. Add metric, param instance to the submarine-server database and rest > 4. Add Metric tracking ui to submarine-web > 5. Add a metric graphical display to submarine-web -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Resolved] (SUBMARINE-202) submarine core need to support MXNet
[ https://issues.apache.org/jira/browse/SUBMARINE-202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan resolved SUBMARINE-202. -- Resolution: Duplicate Closing as dup of SUBMARINE-198, [~yuan_zac] please reopen if I misunderstood. > submarine core need to support MXNet > > > Key: SUBMARINE-202 > URL: https://issues.apache.org/jira/browse/SUBMARINE-202 > Project: Apache Submarine > Issue Type: Bug >Reporter: Zac Zhou >Assignee: Zac Zhou >Priority: Major > > Submarine core checks the machine learning framework types which must be > tensorflow or PyTorch, MXNet is supported by tony. Submarine core should > enable MXNet as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-225) [Umbrella] Submarine Client
[ https://issues.apache.org/jira/browse/SUBMARINE-225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997074#comment-16997074 ] Wangda Tan commented on SUBMARINE-225: -- [~jiwq], [~tangzhankun] can you update what does this mean? More description will be helpful. Thanks > [Umbrella] Submarine Client > --- > > Key: SUBMARINE-225 > URL: https://issues.apache.org/jira/browse/SUBMARINE-225 > Project: Apache Submarine > Issue Type: New Feature >Reporter: Wanqiang Ji >Priority: Major > > This is a milestone. Maybe we can used the SC in the JIRA summary field to > mark the task related to submarine client. Such as [SC] Create the > submarine-client module. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-242) [E2E] Submarine integration test workbench's start script
[ https://issues.apache.org/jira/browse/SUBMARINE-242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-242: - Component/s: E2E Test > [E2E] Submarine integration test workbench's start script > - > > Key: SUBMARINE-242 > URL: https://issues.apache.org/jira/browse/SUBMARINE-242 > Project: Apache Submarine > Issue Type: Sub-task > Components: E2E Test >Reporter: luzhonghao >Priority: Major > > In the submarine-integration module, we should test whether in any directory > we can use workbench-daemon.sh to start | stop | restart workbenchServer -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-242) [E2E] Submarine integration test workbench's start script
[ https://issues.apache.org/jira/browse/SUBMARINE-242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-242: - Target Version: 0.4.0 > [E2E] Submarine integration test workbench's start script > - > > Key: SUBMARINE-242 > URL: https://issues.apache.org/jira/browse/SUBMARINE-242 > Project: Apache Submarine > Issue Type: Sub-task > Components: E2E Test >Reporter: luzhonghao >Priority: Major > > In the submarine-integration module, we should test whether in any directory > we can use workbench-daemon.sh to start | stop | restart workbenchServer -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
