[jira] [Commented] (YARN-9137) Get the IP and port of the docker container and display it on WEB UI2
[ https://issues.apache.org/jira/browse/YARN-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938169#comment-16938169 ] Xun Liu commented on YARN-9137: --- [~eyang], Sorry, I am too busy, I don't have time to complete the development of this issue, I am very sorry. > Get the IP and port of the docker container and display it on WEB UI2 > - > > Key: YARN-9137 > URL: https://issues.apache.org/jira/browse/YARN-9137 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xun Liu >Priority: Major > > 1) When using a container network such as Calico, the IP of the container is > not the IP of the host, but is allocated in the private network, and the > different containers can be directly connected. > Exposing the services in the container through a reverse proxy such as Ngxin > makes it easy for users to view the IP and port on WEB UI2 to use the > services in the container, such as Tomcat, TensorBoard, and so on. > 2) When not using a container network such as Calico, the container also has > its own container port. > So you need to display the IP and port of the docker container on WEB UI2. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9137) Get the IP and port of the docker container and display it on WEB UI2
[ https://issues.apache.org/jira/browse/YARN-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu reassigned YARN-9137: - Assignee: (was: Xun Liu) > Get the IP and port of the docker container and display it on WEB UI2 > - > > Key: YARN-9137 > URL: https://issues.apache.org/jira/browse/YARN-9137 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xun Liu >Priority: Major > > 1) When using a container network such as Calico, the IP of the container is > not the IP of the host, but is allocated in the private network, and the > different containers can be directly connected. > Exposing the services in the container through a reverse proxy such as Ngxin > makes it easy for users to view the IP and port on WEB UI2 to use the > services in the container, such as Tomcat, TensorBoard, and so on. > 2) When not using a container network such as Calico, the container also has > its own container port. > So you need to display the IP and port of the docker container on WEB UI2. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-2915) Enable YARN RM scale out via federation using multiple RM's
[ https://issues.apache.org/jira/browse/YARN-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783960#comment-16783960 ] Xun Liu edited comment on YARN-2915 at 3/5/19 1:44 AM: --- [~subru], Great! :D We are going to follow the documentation [https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/Federation.html] , Deployed in multiple of our data centers. Thank you team contribution, Help us solve a lot of problems. was (Author: liuxun323): [~subru], Great! :D We are going to follow the documentation [https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/Federation.html] , Deployed in multiple of our data centers. Thank you team contribution, Help us solve a lot of problems. > Enable YARN RM scale out via federation using multiple RM's > --- > > Key: YARN-2915 > URL: https://issues.apache.org/jira/browse/YARN-2915 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager, resourcemanager >Reporter: Sriram Rao >Assignee: Subru Krishnan >Priority: Major > Labels: federation > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: FEDERATION_CAPACITY_ALLOCATION_JIRA.pdf, > Federation-BoF.pdf, YARN-Federation-Hadoop-Summit_final.pptx, > Yarn_federation_design_v1.pdf, federation-prototype.patch > > > This is an umbrella JIRA that proposes to scale out YARN to support large > clusters comprising of tens of thousands of nodes. That is, rather than > limiting a YARN managed cluster to about 4k in size, the proposal is to > enable the YARN managed cluster to be elastically scalable. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2915) Enable YARN RM scale out via federation using multiple RM's
[ https://issues.apache.org/jira/browse/YARN-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783960#comment-16783960 ] Xun Liu commented on YARN-2915: --- [~subru], Great! :D We are going to follow the documentation [https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/Federation.html] , Deployed in multiple of our data centers. Thank you team contribution, Help us solve a lot of problems. > Enable YARN RM scale out via federation using multiple RM's > --- > > Key: YARN-2915 > URL: https://issues.apache.org/jira/browse/YARN-2915 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager, resourcemanager >Reporter: Sriram Rao >Assignee: Subru Krishnan >Priority: Major > Labels: federation > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: FEDERATION_CAPACITY_ALLOCATION_JIRA.pdf, > Federation-BoF.pdf, YARN-Federation-Hadoop-Summit_final.pptx, > Yarn_federation_design_v1.pdf, federation-prototype.patch > > > This is an umbrella JIRA that proposes to scale out YARN to support large > clusters comprising of tens of thousands of nodes. That is, rather than > limiting a YARN managed cluster to about 4k in size, the proposal is to > enable the YARN managed cluster to be elastically scalable. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2915) Enable YARN RM scale out via federation using multiple RM's
[ https://issues.apache.org/jira/browse/YARN-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783249#comment-16783249 ] Xun Liu commented on YARN-2915: --- Hi [~subru]，Excuse me, which versions are supported by the yarn federation Feature? Is it supported by version 3.0.0+? We intend to upgrade yarn to version 3.2.0+. > Enable YARN RM scale out via federation using multiple RM's > --- > > Key: YARN-2915 > URL: https://issues.apache.org/jira/browse/YARN-2915 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager, resourcemanager >Reporter: Sriram Rao >Assignee: Subru Krishnan >Priority: Major > Labels: federation > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: FEDERATION_CAPACITY_ALLOCATION_JIRA.pdf, > Federation-BoF.pdf, YARN-Federation-Hadoop-Summit_final.pptx, > Yarn_federation_design_v1.pdf, federation-prototype.patch > > > This is an umbrella JIRA that proposes to scale out YARN to support large > clusters comprising of tens of thousands of nodes. That is, rather than > limiting a YARN managed cluster to about 4k in size, the proposal is to > enable the YARN managed cluster to be elastically scalable. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9137) Get the IP and port of the docker container and display it on WEB UI2
[ https://issues.apache.org/jira/browse/YARN-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773577#comment-16773577 ] Xun Liu commented on YARN-9137: --- [~eyang]，Ok, let me finish this work. :D > Get the IP and port of the docker container and display it on WEB UI2 > - > > Key: YARN-9137 > URL: https://issues.apache.org/jira/browse/YARN-9137 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xun Liu >Assignee: Xun Liu >Priority: Major > > 1) When using a container network such as Calico, the IP of the container is > not the IP of the host, but is allocated in the private network, and the > different containers can be directly connected. > Exposing the services in the container through a reverse proxy such as Ngxin > makes it easy for users to view the IP and port on WEB UI2 to use the > services in the container, such as Tomcat, TensorBoard, and so on. > 2) When not using a container network such as Calico, the container also has > its own container port. > So you need to display the IP and port of the docker container on WEB UI2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8135) Hadoop {Submarine} Project: Simple and scalable deployment of deep learning training / serving jobs on Hadoop
[
https://issues.apache.org/jira/browse/YARN-8135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764805#comment-16764805
]
Xun Liu edited comment on YARN-8135 at 2/12/19 2:39 AM:
hi [~sinever], Can you tell me the problem you are experiencing? Generally, you
can use submarine normally according to the following steps.
1) You first need to install the submarine runtime environment installed on
your server.
[https://github.com/hadoopsubmarine/hadoop-submarine-ecosystem/tree/master/submarine-installer]
[https://github.com/hadoopsubmarine/hadoop-submarine-ecosystem/blob/master/submarine-installer/InstallationGuide.md]
2) Then use this instructions to operate:
[https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/site/markdown/QuickStart.md]
If you can't solve your problem yet, you can communicate with me via email
(hadoopsubmar...@gmail.com). :)
was (Author: liuxun323):
hi [~sinever], Can you tell me the problem you are experiencing? Generally, you
can use submarine normally according to the following steps.
1) You first need to install the submarine runtime environment installed on
your server.
[https://github.com/hadoopsubmarine/hadoop-submarine-ecosystem/tree/master/submarine-installer]
[https://github.com/hadoopsubmarine/hadoop-submarine-ecosystem/blob/master/submarine-installer/InstallationGuide.md]
2) Then use this instructions to operate:
[https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/site/markdown/QuickStart.md]
If you can't solve your problem yet, you can communicate with me via email
(liuxun...@gmail.com). :)
> Hadoop {Submarine} Project: Simple and scalable deployment of deep learning
> training / serving jobs on Hadoop
> -
>
> Key: YARN-8135
> URL: https://issues.apache.org/jira/browse/YARN-8135
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Major
>
> Description:
> *Goals:*
> - Allow infra engineer / data scientist to run *unmodified* Tensorflow jobs
> on YARN.
> - Allow jobs easy access data/models in HDFS and other storages.
> - Can launch services to serve Tensorflow/MXNet models.
> - Support run distributed Tensorflow jobs with simple configs.
> - Support run user-specified Docker images.
> - Support specify GPU and other resources.
> - Support launch tensorboard if user specified.
> - Support customized DNS name for roles (like tensorboard.$user.$domain:6006)
> *Why this name?*
> - Because Submarine is the only vehicle can let human to explore deep
> places. B-)
> h3. {color:#ff}Please refer to on-going design doc, and add your
> thoughts:
> [https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#|https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit?usp=sharing]{color}
>
> *{color:#33}See Also:{color}*
> * {color:#33}Zeppelin integration with Submarine design:
> [https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#heading=h.4jov859x47qe]{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8135) Hadoop {Submarine} Project: Simple and scalable deployment of deep learning training / serving jobs on Hadoop
[
https://issues.apache.org/jira/browse/YARN-8135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764805#comment-16764805
]
Xun Liu commented on YARN-8135:
---
hi [~sinever], Can you tell me the problem you are experiencing? Generally, you
can use submarine normally according to the following steps.
1) You first need to install the submarine runtime environment installed on
your server.
[https://github.com/hadoopsubmarine/hadoop-submarine-ecosystem/tree/master/submarine-installer]
[https://github.com/hadoopsubmarine/hadoop-submarine-ecosystem/blob/master/submarine-installer/InstallationGuide.md]
2) Then use this instructions to operate:
[https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/site/markdown/QuickStart.md]
If you can't solve your problem yet, you can communicate with me via email
(liuxun...@gmail.com). :)
> Hadoop {Submarine} Project: Simple and scalable deployment of deep learning
> training / serving jobs on Hadoop
> -
>
> Key: YARN-8135
> URL: https://issues.apache.org/jira/browse/YARN-8135
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Major
>
> Description:
> *Goals:*
> - Allow infra engineer / data scientist to run *unmodified* Tensorflow jobs
> on YARN.
> - Allow jobs easy access data/models in HDFS and other storages.
> - Can launch services to serve Tensorflow/MXNet models.
> - Support run distributed Tensorflow jobs with simple configs.
> - Support run user-specified Docker images.
> - Support specify GPU and other resources.
> - Support launch tensorboard if user specified.
> - Support customized DNS name for roles (like tensorboard.$user.$domain:6006)
> *Why this name?*
> - Because Submarine is the only vehicle can let human to explore deep
> places. B-)
> h3. {color:#ff}Please refer to on-going design doc, and add your
> thoughts:
> [https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#|https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit?usp=sharing]{color}
>
> *{color:#33}See Also:{color}*
> * {color:#33}Zeppelin integration with Submarine design:
> [https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#heading=h.4jov859x47qe]{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9003) Support multi-homed network for docker container
[ https://issues.apache.org/jira/browse/YARN-9003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746273#comment-16746273 ] Xun Liu commented on YARN-9003: --- [~eyang] thanks for the patch. :) I have a question to ask. If docker do not support multiple networks, Is it necessary to limit the port type processing when the network type is bridge? If only one parallel network is specified, is it also possible to have the -P parameter take effect? https://issues.apache.org/jira/browse/YARN-8986?focusedCommentId=16695628=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16695628 > Support multi-homed network for docker container > > > Key: YARN-9003 > URL: https://issues.apache.org/jira/browse/YARN-9003 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Priority: Major > Labels: docker > Attachments: YARN-9003.001.patch, YARN-9003.002.patch, > YARN-9003.003.patch, YARN-9003.004.patch, YARN-9003.005.patch, > YARN-9003.006.patch, YARN-9003.007.patch, YARN-9003.008.patch, > YARN-9003.009.patch > > > Docker network can be defined as configuration properties - docker.network to > setup docker container to connect to a specific network in YARN service. > Docker can run multi-homed network by specifying --net=bridge > --net=private-net. This is useful to expose small number of front end > container and ports, while the rest of the infrastructure runs in private > network. This task is to add support for specifying multiple docker networks > to YARN service and docker support. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9207) {Submarine} Support long-running service type services
[
https://issues.apache.org/jira/browse/YARN-9207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745752#comment-16745752
]
Xun Liu commented on YARN-9207:
---
I saw an article here describing the service of a long service type.
Https://zh.hortonworks.com/blog/first-class-support-long-running-services-apache-hadoop-yarn/
It is tested that the service can be run for a specified time through the JSON
in the article.
{quote}{
"name":"[httpd-service|https://www.json.cn/httpd-service];,
"lifetime":"3600",
"components":[
{
"name":"[httpd|https://www.json.cn/httpd];,
"number_of_containers":2,
"artifact":{
"id":"centos/httpd-24-centos7:latest",
"type":"DOCKER"
},
"launch_command":"/usr/bin/run-httpd",
"resource":{
"cpus":1,
"memory":"1024"
}
}
]
}
{quote}
But there are no updates in the article, there are 2 problems:
1. The version field is missing.
2. When lefetime=-1, there is no time limit to run
Using a long service service can be achieved by the following json content
*_yarn app -launch_* *_my-httpd /path/to/httpd.json_*
{quote}{
"name":"[httpd-service|https://www.json.cn/httpd-service];,
"lifetime":"-1",
"components":[
{
"name":"[httpd|https://www.json.cn/httpd];,
"number_of_containers":2,
"artifact":{
"id":"10.120.196.232:5000/centos/httpd-24-centos7:latest",
"type":"DOCKER"
},
"launch_command":"/usr/bin/run-httpd",
"resource":{
"cpus":1,
"memory":"1024"
}
}
],
"version":"13535262542525424"
}
{quote}
> {Submarine} Support long-running service type services
> --
>
> Key: YARN-9207
> URL: https://issues.apache.org/jira/browse/YARN-9207
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: submarine
>Reporter: Xun Liu
>Priority: Major
>
> At present, YANR only supports batch type tasks. After the batch is
> completed, the task ends and the container will exit.
> Machine learning needs to provide users with tensorboard, model prediction
> online service is a long-running service type of service, currently can
> increase the "{color:#205081}sleep infinity{color}" in the task's start
> command, so that the task has been sleeping until the container does not
> exit, need to be closed Time is terminated manually.
> But this method is not friendly enough or professional, so the YARN should
> add a task type parameter, the named task is batch processing or long-running
> service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9207) {Submarine} Support long-running service type services
Xun Liu created YARN-9207:
-
Summary: {Submarine} Support long-running service type services
Key: YARN-9207
URL: https://issues.apache.org/jira/browse/YARN-9207
Project: Hadoop YARN
Issue Type: Sub-task
Components: submarine
Reporter: Xun Liu
At present, YANR only supports batch type tasks. After the batch is completed,
the task ends and the container will exit.
Machine learning needs to provide users with tensorboard, model prediction
online service is a long-running service type of service, currently can
increase the "{color:#205081}sleep infinity{color}" in the task's start
command, so that the task has been sleeping until the container does not exit,
need to be closed Time is terminated manually.
But this method is not friendly enough or professional, so the YARN should add
a task type parameter, the named task is batch processing or long-running
service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726565#comment-16726565 ] Xun Liu commented on YARN-5168: --- [~eyang], Please help me review the code, thank you! :) > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch, > YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch, > YARN-5168.009.patch, YARN-5168.010.patch, YARN-5168.011.patch, > YARN-5168.012.patch, YARN-5168.013.patch, YARN-5168.014.patch, > YARN-5168.015.patch, YARN-5168.016.patch, YARN-5168.017.patch, > YARN-5168.018.patch, YARN-5168.019.patch, YARN-5168.020.patch, > exposedPorts1.png, exposedPorts2.png > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726387#comment-16726387 ] Xun Liu commented on YARN-5168: --- [~eyang], Sorry, I neglected to check and missed. I will fix this problem right away. > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch, > YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch, > YARN-5168.009.patch, YARN-5168.010.patch, YARN-5168.011.patch, > YARN-5168.012.patch, YARN-5168.013.patch, YARN-5168.014.patch, > YARN-5168.015.patch, YARN-5168.016.patch, YARN-5168.017.patch, > YARN-5168.018.patch, YARN-5168.019.patch, YARN-5168.020.patch, > exposedPorts1.png, exposedPorts2.png > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu updated YARN-5168: -- Attachment: YARN-5168.020.patch > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch, > YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch, > YARN-5168.009.patch, YARN-5168.010.patch, YARN-5168.011.patch, > YARN-5168.012.patch, YARN-5168.013.patch, YARN-5168.014.patch, > YARN-5168.015.patch, YARN-5168.016.patch, YARN-5168.017.patch, > YARN-5168.018.patch, YARN-5168.019.patch, YARN-5168.020.patch, > exposedPorts1.png, exposedPorts2.png > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu updated YARN-5168: -- Attachment: YARN-5168.019.patch > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch, > YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch, > YARN-5168.009.patch, YARN-5168.010.patch, YARN-5168.011.patch, > YARN-5168.012.patch, YARN-5168.013.patch, YARN-5168.014.patch, > YARN-5168.015.patch, YARN-5168.016.patch, YARN-5168.017.patch, > YARN-5168.018.patch, YARN-5168.019.patch, exposedPorts1.png, exposedPorts2.png > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu updated YARN-5168: -- Attachment: (was: YARN-5168.019.patch) > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch, > YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch, > YARN-5168.009.patch, YARN-5168.010.patch, YARN-5168.011.patch, > YARN-5168.012.patch, YARN-5168.013.patch, YARN-5168.014.patch, > YARN-5168.015.patch, YARN-5168.016.patch, YARN-5168.017.patch, > YARN-5168.018.patch, exposedPorts1.png, exposedPorts2.png > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu updated YARN-5168: -- Attachment: YARN-5168.019.patch > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch, > YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch, > YARN-5168.009.patch, YARN-5168.010.patch, YARN-5168.011.patch, > YARN-5168.012.patch, YARN-5168.013.patch, YARN-5168.014.patch, > YARN-5168.015.patch, YARN-5168.016.patch, YARN-5168.017.patch, > YARN-5168.018.patch, YARN-5168.019.patch, exposedPorts1.png, exposedPorts2.png > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5168) Add port mapping handling when docker container use bridge network
[
https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16725513#comment-16725513
]
Xun Liu commented on YARN-5168:
---
[~eyang] , I checked the code carefully and I feel that the 3 errors reported
by Jenkins have nothing to do with my code.
{quote}[https://builds.apache.org/job/PreCommit-YARN-Build/22923/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt]
[https://builds.apache.org/job/PreCommit-YARN-Build/22923/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt]
[https://builds.apache.org/job/PreCommit-YARN-Build/22923/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt]
{quote}
I leave the container newInstance with 7 parameters, which are set by the
setExposedPort function.
Please help me review the code, thank you!
> Add port mapping handling when docker container use bridge network
> --
>
> Key: YARN-5168
> URL: https://issues.apache.org/jira/browse/YARN-5168
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jun Gong
>Assignee: Xun Liu
>Priority: Major
> Labels: Docker
> Attachments: YARN-5168.001.patch, YARN-5168.002.patch,
> YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch,
> YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch,
> YARN-5168.009.patch, YARN-5168.010.patch, YARN-5168.011.patch,
> YARN-5168.012.patch, YARN-5168.013.patch, YARN-5168.014.patch,
> YARN-5168.015.patch, YARN-5168.016.patch, YARN-5168.017.patch,
> YARN-5168.018.patch, exposedPorts1.png, exposedPorts2.png
>
>
> YARN-4007 addresses different network setups when launching the docker
> container. We need support port mapping when docker container uses bridge
> network.
> The following problems are what we faced:
> 1. Add "-P" to map docker container's exposed ports to automatically.
> 2. Add "-p" to let user specify specific ports to map.
> 3. Add service registry support for bridge network case, then app could find
> each other. It could be done out of YARN, however it might be more convenient
> to support it natively in YARN.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu updated YARN-5168: -- Attachment: YARN-5168.018.patch > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch, > YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch, > YARN-5168.009.patch, YARN-5168.010.patch, YARN-5168.011.patch, > YARN-5168.012.patch, YARN-5168.013.patch, YARN-5168.014.patch, > YARN-5168.015.patch, YARN-5168.016.patch, YARN-5168.017.patch, > YARN-5168.018.patch, exposedPorts1.png, exposedPorts2.png > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu updated YARN-5168: -- Attachment: YARN-5168.017.patch > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch, > YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch, > YARN-5168.009.patch, YARN-5168.010.patch, YARN-5168.011.patch, > YARN-5168.012.patch, YARN-5168.013.patch, YARN-5168.014.patch, > YARN-5168.015.patch, YARN-5168.016.patch, YARN-5168.017.patch, > exposedPorts1.png, exposedPorts2.png > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16724616#comment-16724616 ] Xun Liu commented on YARN-5168: --- [~eyang], Ok, according to your opinion, I leave the container newInstance with 7 parameters, which are set by the setExposedPort function. I will finish it.:P > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch, > YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch, > YARN-5168.009.patch, YARN-5168.010.patch, YARN-5168.011.patch, > YARN-5168.012.patch, YARN-5168.013.patch, YARN-5168.014.patch, > YARN-5168.015.patch, YARN-5168.016.patch, exposedPorts1.png, exposedPorts2.png > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5168) Add port mapping handling when docker container use bridge network
[
https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723797#comment-16723797
]
Xun Liu commented on YARN-5168:
---
[~eyang] , I checked the code carefully and I feel that the 3 errors reported
by Jenkins have nothing to do with my code.
{quote}[https://builds.apache.org/job/PreCommit-YARN-Build/22907/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt]
[https://builds.apache.org/job/PreCommit-YARN-Build/22907/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt]
[https://builds.apache.org/job/PreCommit-YARN-Build/22907/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt]
{quote}
a warning for checkstyle
{quote}./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Container.java:81:
public static Container newInstance(ContainerId containerId, NodeId
nodeId,:27: More than 7 parameters (found 8). [ParameterNumber]{quote}
I think that the constructor parameter of the Container has more than 7
problems. It is necessary to open a new JIRA code reconstruction of the
Container through the builder mode.
> Add port mapping handling when docker container use bridge network
> --
>
> Key: YARN-5168
> URL: https://issues.apache.org/jira/browse/YARN-5168
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jun Gong
>Assignee: Xun Liu
>Priority: Major
> Labels: Docker
> Attachments: YARN-5168.001.patch, YARN-5168.002.patch,
> YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch,
> YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch,
> YARN-5168.009.patch, YARN-5168.010.patch, YARN-5168.011.patch,
> YARN-5168.012.patch, YARN-5168.013.patch, YARN-5168.014.patch,
> YARN-5168.015.patch, YARN-5168.016.patch, exposedPorts1.png, exposedPorts2.png
>
>
> YARN-4007 addresses different network setups when launching the docker
> container. We need support port mapping when docker container uses bridge
> network.
> The following problems are what we faced:
> 1. Add "-P" to map docker container's exposed ports to automatically.
> 2. Add "-p" to let user specify specific ports to map.
> 3. Add service registry support for bridge network case, then app could find
> each other. It could be done out of YARN, however it might be more convenient
> to support it natively in YARN.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu updated YARN-5168: -- Attachment: YARN-5168.016.patch > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch, > YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch, > YARN-5168.009.patch, YARN-5168.010.patch, YARN-5168.011.patch, > YARN-5168.012.patch, YARN-5168.013.patch, YARN-5168.014.patch, > YARN-5168.015.patch, YARN-5168.016.patch, exposedPorts1.png, exposedPorts2.png > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu updated YARN-5168: -- Attachment: YARN-5168.015.patch > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch, > YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch, > YARN-5168.009.patch, YARN-5168.010.patch, YARN-5168.011.patch, > YARN-5168.012.patch, YARN-5168.013.patch, YARN-5168.014.patch, > YARN-5168.015.patch, exposedPorts1.png, exposedPorts2.png > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu updated YARN-5168: -- Attachment: YARN-5168.014.patch > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch, > YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch, > YARN-5168.009.patch, YARN-5168.010.patch, YARN-5168.011.patch, > YARN-5168.012.patch, YARN-5168.013.patch, YARN-5168.014.patch, > exposedPorts1.png, exposedPorts2.png > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu updated YARN-5168: -- Attachment: YARN-5168.013.patch > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch, > YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch, > YARN-5168.009.patch, YARN-5168.010.patch, YARN-5168.011.patch, > YARN-5168.012.patch, YARN-5168.013.patch, exposedPorts1.png, exposedPorts2.png > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9137) Get the IP and port of the docker container and display it on WEB UI2
[ https://issues.apache.org/jira/browse/YARN-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722340#comment-16722340 ] Xun Liu commented on YARN-9137: --- [~eyang], I want to implement such a function. What is your opinion? > Get the IP and port of the docker container and display it on WEB UI2 > - > > Key: YARN-9137 > URL: https://issues.apache.org/jira/browse/YARN-9137 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xun Liu >Assignee: Xun Liu >Priority: Major > > 1) When using a container network such as Calico, the IP of the container is > not the IP of the host, but is allocated in the private network, and the > different containers can be directly connected. > Exposing the services in the container through a reverse proxy such as Ngxin > makes it easy for users to view the IP and port on WEB UI2 to use the > services in the container, such as Tomcat, TensorBoard, and so on. > 2) When not using a container network such as Calico, the container also has > its own container port. > So you need to display the IP and port of the docker container on WEB UI2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9137) Get the IP and port of the docker container and display it on WEB UI2
[ https://issues.apache.org/jira/browse/YARN-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu updated YARN-9137: -- Description: 1) When using a container network such as Calico, the IP of the container is not the IP of the host, but is allocated in the private network, and the different containers can be directly connected. Exposing the services in the container through a reverse proxy such as Ngxin makes it easy for users to view the IP and port on WEB UI2 to use the services in the container, such as Tomcat, TensorBoard, and so on. 2) When not using a container network such as Calico, the container also has its own container port. So you need to display the IP and port of the docker container on WEB UI2. was: 1. When using a container network such as Calico, the IP of the container is not the IP of the host, but is allocated in the private network, and the different containers can be directly connected. Exposing the services in the container through a reverse proxy such as Ngxin makes it easy for users to view the IP and port on WEB UI2 to use the services in the container, such as Tomcat, TensorBoard, and so on. 2. When not using a container network such as Calico, the container also has its own container port. So you need to display the IP and port of the docker container on WEB UI2. > Get the IP and port of the docker container and display it on WEB UI2 > - > > Key: YARN-9137 > URL: https://issues.apache.org/jira/browse/YARN-9137 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xun Liu >Assignee: Xun Liu >Priority: Major > > 1) When using a container network such as Calico, the IP of the container is > not the IP of the host, but is allocated in the private network, and the > different containers can be directly connected. > Exposing the services in the container through a reverse proxy such as Ngxin > makes it easy for users to view the IP and port on WEB UI2 to use the > services in the container, such as Tomcat, TensorBoard, and so on. > 2) When not using a container network such as Calico, the container also has > its own container port. > So you need to display the IP and port of the docker container on WEB UI2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9137) Get the IP and port of the docker container and display it on WEB UI2
[ https://issues.apache.org/jira/browse/YARN-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu updated YARN-9137: -- Description: 1. When using a container network such as Calico, the IP of the container is not the IP of the host, but is allocated in the private network, and the different containers can be directly connected. Exposing the services in the container through a reverse proxy such as Ngxin makes it easy for users to view the IP and port on WEB UI2 to use the services in the container, such as Tomcat, TensorBoard, and so on. 2. When not using a container network such as Calico, the container also has its own container port. So you need to display the IP and port of the docker container on WEB UI2. was: 1. When using a container network such as Calico, the IP of the container is not the IP of the host, but is allocated in the private network, and the different containers can be directly connected. Exposing the services in the container through a reverse proxy such as Ngxin makes it easy for users to view the IP and port on WEB UI2 to use the services in the container, such as Tomcat, TensorBoard, and so on. 2. When not using a container network such as Calico, the container also has its own container port. So you need to display the IP and port of the docker container on WEB UI2. > Get the IP and port of the docker container and display it on WEB UI2 > - > > Key: YARN-9137 > URL: https://issues.apache.org/jira/browse/YARN-9137 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xun Liu >Assignee: Xun Liu >Priority: Major > > 1. When using a container network such as Calico, the IP of the container is > not the IP of the host, but is allocated in the private network, and the > different containers can be directly connected. > Exposing the services in the container through a reverse proxy such as Ngxin > makes it easy for users to view the IP and port on WEB UI2 to use the > services in the container, such as Tomcat, TensorBoard, and so on. > 2. When not using a container network such as Calico, the container also has > its own container port. > So you need to display the IP and port of the docker container on WEB UI2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9137) Get the IP and port of the docker container and display it on WEB UI2
Xun Liu created YARN-9137: - Summary: Get the IP and port of the docker container and display it on WEB UI2 Key: YARN-9137 URL: https://issues.apache.org/jira/browse/YARN-9137 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xun Liu Assignee: Xun Liu 1. When using a container network such as Calico, the IP of the container is not the IP of the host, but is allocated in the private network, and the different containers can be directly connected. Exposing the services in the container through a reverse proxy such as Ngxin makes it easy for users to view the IP and port on WEB UI2 to use the services in the container, such as Tomcat, TensorBoard, and so on. 2. When not using a container network such as Calico, the container also has its own container port. So you need to display the IP and port of the docker container on WEB UI2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu updated YARN-5168: -- Attachment: YARN-5168.012.patch > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch, > YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch, > YARN-5168.009.patch, YARN-5168.010.patch, YARN-5168.011.patch, > YARN-5168.012.patch, exposedPorts1.png, exposedPorts2.png > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1670#comment-1670 ] Xun Liu commented on YARN-5168: --- [~eyang], Exposed Ports can be displayed on UI2. :D !exposedPorts1.png|width=788,height=376! !exposedPorts2.png|width=788,height=376! > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch, > YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch, > YARN-5168.009.patch, YARN-5168.010.patch, YARN-5168.011.patch, > exposedPorts1.png, exposedPorts2.png > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu updated YARN-5168: -- Attachment: exposedPorts1.png exposedPorts2.png > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch, > YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch, > YARN-5168.009.patch, YARN-5168.010.patch, YARN-5168.011.patch, > exposedPorts1.png, exposedPorts2.png > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu updated YARN-5168: -- Attachment: YARN-5168.011.patch > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch, > YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch, > YARN-5168.009.patch, YARN-5168.010.patch, YARN-5168.011.patch > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722058#comment-16722058 ] Xun Liu commented on YARN-5168: --- [~eyang], I haven't done any development on WEB UI2 before. This time, it took 2 days to learn Ember.js, so I missed the time. I will complete this task as soon as possible, please wait a moment. :) > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch, > YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch, > YARN-5168.009.patch, YARN-5168.010.patch > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu updated YARN-5168: -- Comment: was deleted (was: [~eyang], Testing, ready to submit patch, please wait a moment. :)) > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch, > YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch, > YARN-5168.009.patch, YARN-5168.010.patch > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722054#comment-16722054 ] Xun Liu commented on YARN-5168: --- [~eyang], Testing, ready to submit patch, please wait a moment. :) > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch, > YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch, > YARN-5168.009.patch, YARN-5168.010.patch > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9090) [Submarine] Adjust the submarine installation script document
[ https://issues.apache.org/jira/browse/YARN-9090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713504#comment-16713504 ] Xun Liu commented on YARN-9090: --- [~leftnoteasy] , [~sunilg] Please help me review the code, thank! > [Submarine] Adjust the submarine installation script document > - > > Key: YARN-9090 > URL: https://issues.apache.org/jira/browse/YARN-9090 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xun Liu >Assignee: Xun Liu >Priority: Blocker > Attachments: YARN-9090.001.patch > > > Migrate the submarine installation script document from the hadoop-yarn > project. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9090) [Submarine] Adjust the submarine installation script document
Xun Liu created YARN-9090: - Summary: [Submarine] Adjust the submarine installation script document Key: YARN-9090 URL: https://issues.apache.org/jira/browse/YARN-9090 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xun Liu Assignee: Xun Liu Migrate the submarine installation script document from the hadoop-yarn project. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709512#comment-16709512 ] Xun Liu commented on YARN-5168: --- [~eyang], Thanks for your tips, I deal with it immediately. :) > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch, > YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch, > YARN-5168.009.patch, YARN-5168.010.patch > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16708224#comment-16708224 ] Xun Liu commented on YARN-5168: --- [~eyang], I have completed this task, please help me review, thank you! > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch, > YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch, > YARN-5168.009.patch, YARN-5168.010.patch > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu updated YARN-5168: -- Attachment: YARN-5168.010.patch > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch, > YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch, > YARN-5168.009.patch, YARN-5168.010.patch > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu updated YARN-5168: -- Attachment: YARN-5168.009.patch > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch, > YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch, > YARN-5168.009.patch > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8870) [Submarine] Add submarine installation scripts
[
https://issues.apache.org/jira/browse/YARN-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706870#comment-16706870
]
Xun Liu commented on YARN-8870:
---
Hi [~leftnoteasy], [~sunilg] I have fixed 227 shell style warnings. And passed
my installation and unstallation test on the GPU server.
Some of the remaining warnings cannot be modified as suggested by the
shellCheck. The actual test found that it will cause a program error.
> [Submarine] Add submarine installation scripts
> --
>
> Key: YARN-8870
> URL: https://issues.apache.org/jira/browse/YARN-8870
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xun Liu
>Assignee: Xun Liu
>Priority: Critical
> Fix For: 3.2.0
>
> Attachments: YARN-8870-addendum.008.patch, YARN-8870.001.patch,
> YARN-8870.004.patch, YARN-8870.005.patch, YARN-8870.006.patch,
> YARN-8870.007.patch, YARN-8870.009.patch, YARN-8870.010.patch,
> YARN-8870.011.patch, YARN-8870.012.patch
>
>
> In order to reduce the deployment difficulty of Hadoop
> {Submarine} DNS, Docker, GPU, Network, graphics card, operating system kernel
> modification and other components, I specially developed this installation
> script to deploy Hadoop \{Submarine}
> runtime environment, providing one-click installation Scripts, which can also
> be used to install, uninstall, start, and stop individual components step by
> step.
>
> {color:#ff}design d{color}{color:#FF}ocument：{color}
> [https://docs.google.com/document/d/1muCTGFuUXUvM4JaDYjKqX5liQEg-AsNgkxfLMIFxYHU/edit?usp=sharing]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8870) [Submarine] Add submarine installation scripts
[
https://issues.apache.org/jira/browse/YARN-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xun Liu updated YARN-8870:
--
Attachment: YARN-8870.012.patch
> [Submarine] Add submarine installation scripts
> --
>
> Key: YARN-8870
> URL: https://issues.apache.org/jira/browse/YARN-8870
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xun Liu
>Assignee: Xun Liu
>Priority: Critical
> Fix For: 3.2.0
>
> Attachments: YARN-8870-addendum.008.patch, YARN-8870.001.patch,
> YARN-8870.004.patch, YARN-8870.005.patch, YARN-8870.006.patch,
> YARN-8870.007.patch, YARN-8870.009.patch, YARN-8870.010.patch,
> YARN-8870.011.patch, YARN-8870.012.patch
>
>
> In order to reduce the deployment difficulty of Hadoop
> {Submarine} DNS, Docker, GPU, Network, graphics card, operating system kernel
> modification and other components, I specially developed this installation
> script to deploy Hadoop \{Submarine}
> runtime environment, providing one-click installation Scripts, which can also
> be used to install, uninstall, start, and stop individual components step by
> step.
>
> {color:#ff}design d{color}{color:#FF}ocument：{color}
> [https://docs.google.com/document/d/1muCTGFuUXUvM4JaDYjKqX5liQEg-AsNgkxfLMIFxYHU/edit?usp=sharing]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8770) [Submarine] Support using Submarine to submit Pytorch job
[ https://issues.apache.org/jira/browse/YARN-8770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu reassigned YARN-8770: - Assignee: Xun Liu (was: Sunil Govindan) > [Submarine] Support using Submarine to submit Pytorch job > - > > Key: YARN-8770 > URL: https://issues.apache.org/jira/browse/YARN-8770 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Xun Liu >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8870) [Submarine] Add submarine installation scripts
[
https://issues.apache.org/jira/browse/YARN-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xun Liu updated YARN-8870:
--
Attachment: YARN-8870.011.patch
> [Submarine] Add submarine installation scripts
> --
>
> Key: YARN-8870
> URL: https://issues.apache.org/jira/browse/YARN-8870
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xun Liu
>Assignee: Xun Liu
>Priority: Critical
> Fix For: 3.2.0
>
> Attachments: YARN-8870-addendum.008.patch, YARN-8870.001.patch,
> YARN-8870.004.patch, YARN-8870.005.patch, YARN-8870.006.patch,
> YARN-8870.007.patch, YARN-8870.009.patch, YARN-8870.010.patch,
> YARN-8870.011.patch
>
>
> In order to reduce the deployment difficulty of Hadoop
> {Submarine} DNS, Docker, GPU, Network, graphics card, operating system kernel
> modification and other components, I specially developed this installation
> script to deploy Hadoop \{Submarine}
> runtime environment, providing one-click installation Scripts, which can also
> be used to install, uninstall, start, and stop individual components step by
> step.
>
> {color:#ff}design d{color}{color:#FF}ocument：{color}
> [https://docs.google.com/document/d/1muCTGFuUXUvM4JaDYjKqX5liQEg-AsNgkxfLMIFxYHU/edit?usp=sharing]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8870) [Submarine] Add submarine installation scripts
[
https://issues.apache.org/jira/browse/YARN-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704799#comment-16704799
]
Xun Liu commented on YARN-8870:
---
Hi [~sunilg], Yes, I want to fix all shell checks problems, But the old
shellcheck.txt has been cleaned up.
I am trying to find a way to recover the old checks problems.
> [Submarine] Add submarine installation scripts
> --
>
> Key: YARN-8870
> URL: https://issues.apache.org/jira/browse/YARN-8870
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xun Liu
>Assignee: Xun Liu
>Priority: Critical
> Fix For: 3.2.0
>
> Attachments: YARN-8870-addendum.008.patch, YARN-8870.001.patch,
> YARN-8870.004.patch, YARN-8870.005.patch, YARN-8870.006.patch,
> YARN-8870.007.patch, YARN-8870.009.patch, YARN-8870.010.patch
>
>
> In order to reduce the deployment difficulty of Hadoop
> {Submarine} DNS, Docker, GPU, Network, graphics card, operating system kernel
> modification and other components, I specially developed this installation
> script to deploy Hadoop \{Submarine}
> runtime environment, providing one-click installation Scripts, which can also
> be used to install, uninstall, start, and stop individual components step by
> step.
>
> {color:#ff}design d{color}{color:#FF}ocument：{color}
> [https://docs.google.com/document/d/1muCTGFuUXUvM4JaDYjKqX5liQEg-AsNgkxfLMIFxYHU/edit?usp=sharing]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8714) [Submarine] Support files/tarballs to be localized for a training job.
[
https://issues.apache.org/jira/browse/YARN-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704342#comment-16704342
]
Xun Liu commented on YARN-8714:
---
hi [~leftnoteasy]，[~tangzhankun]
Need to support the functionality of the directory because:
1. The user will only write or upload files in the directory, and the user will
only see the directories and files.
2. If the user operates the directory and files, zeppelin privately becomes a
TAR package, and yarn decompresses the tar package into a directory. The two
processes of packaging and unpacking are implemented in half in each of the two
systems, and the logic is very strange.
In submarine, you can check that the directory size of the user-specified
directory cannot exceed the limit. For example: the default value is 1G, the
file size fails, and the error log is output to YARN.
> [Submarine] Support files/tarballs to be localized for a training job.
> --
>
> Key: YARN-8714
> URL: https://issues.apache.org/jira/browse/YARN-8714
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Zhankun Tang
>Priority: Major
> Attachments: YARN-8714-WIP1-trunk-001.patch,
> YARN-8714-WIP1-trunk-002.patch, YARN-8714-trunk.001.patch,
> YARN-8714-trunk.002.patch
>
>
> See
> [https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#heading=h.vkxp9edl11m7],
> {{job run --localization ...}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8870) [Submarine] Add submarine installation scripts
[
https://issues.apache.org/jira/browse/YARN-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xun Liu updated YARN-8870:
--
Attachment: YARN-8870.010.patch
> [Submarine] Add submarine installation scripts
> --
>
> Key: YARN-8870
> URL: https://issues.apache.org/jira/browse/YARN-8870
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xun Liu
>Assignee: Xun Liu
>Priority: Critical
> Fix For: 3.2.0
>
> Attachments: YARN-8870-addendum.008.patch, YARN-8870.001.patch,
> YARN-8870.004.patch, YARN-8870.005.patch, YARN-8870.006.patch,
> YARN-8870.007.patch, YARN-8870.009.patch, YARN-8870.010.patch
>
>
> In order to reduce the deployment difficulty of Hadoop
> {Submarine} DNS, Docker, GPU, Network, graphics card, operating system kernel
> modification and other components, I specially developed this installation
> script to deploy Hadoop \{Submarine}
> runtime environment, providing one-click installation Scripts, which can also
> be used to install, uninstall, start, and stop individual components step by
> step.
>
> {color:#ff}design d{color}{color:#FF}ocument：{color}
> [https://docs.google.com/document/d/1muCTGFuUXUvM4JaDYjKqX5liQEg-AsNgkxfLMIFxYHU/edit?usp=sharing]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8870) [Submarine] Add submarine installation scripts
[
https://issues.apache.org/jira/browse/YARN-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xun Liu updated YARN-8870:
--
Attachment: YARN-8870.009.patch
> [Submarine] Add submarine installation scripts
> --
>
> Key: YARN-8870
> URL: https://issues.apache.org/jira/browse/YARN-8870
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xun Liu
>Assignee: Xun Liu
>Priority: Critical
> Fix For: 3.2.0
>
> Attachments: YARN-8870-addendum.008.patch, YARN-8870.001.patch,
> YARN-8870.004.patch, YARN-8870.005.patch, YARN-8870.006.patch,
> YARN-8870.007.patch, YARN-8870.009.patch
>
>
> In order to reduce the deployment difficulty of Hadoop
> {Submarine} DNS, Docker, GPU, Network, graphics card, operating system kernel
> modification and other components, I specially developed this installation
> script to deploy Hadoop \{Submarine}
> runtime environment, providing one-click installation Scripts, which can also
> be used to install, uninstall, start, and stop individual components step by
> step.
>
> {color:#ff}design d{color}{color:#FF}ocument：{color}
> [https://docs.google.com/document/d/1muCTGFuUXUvM4JaDYjKqX5liQEg-AsNgkxfLMIFxYHU/edit?usp=sharing]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8870) [Submarine] Add submarine installation scripts
[
https://issues.apache.org/jira/browse/YARN-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xun Liu updated YARN-8870:
--
Attachment: YARN-8870-addendum.008.patch
> [Submarine] Add submarine installation scripts
> --
>
> Key: YARN-8870
> URL: https://issues.apache.org/jira/browse/YARN-8870
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xun Liu
>Assignee: Xun Liu
>Priority: Critical
> Fix For: 3.2.0
>
> Attachments: YARN-8870-addendum.008.patch, YARN-8870.001.patch,
> YARN-8870.004.patch, YARN-8870.005.patch, YARN-8870.006.patch,
> YARN-8870.007.patch
>
>
> In order to reduce the deployment difficulty of Hadoop
> {Submarine} DNS, Docker, GPU, Network, graphics card, operating system kernel
> modification and other components, I specially developed this installation
> script to deploy Hadoop \{Submarine}
> runtime environment, providing one-click installation Scripts, which can also
> be used to install, uninstall, start, and stop individual components step by
> step.
>
> {color:#ff}design d{color}{color:#FF}ocument：{color}
> [https://docs.google.com/document/d/1muCTGFuUXUvM4JaDYjKqX5liQEg-AsNgkxfLMIFxYHU/edit?usp=sharing]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8770) [Submarine] Support using Submarine to submit Pytorch job
[ https://issues.apache.org/jira/browse/YARN-8770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704136#comment-16704136 ] Xun Liu commented on YARN-8770: --- Hi [~sunilg]，[~leftnoteasy]，Can you let me implement this feature? > [Submarine] Support using Submarine to submit Pytorch job > - > > Key: YARN-8770 > URL: https://issues.apache.org/jira/browse/YARN-8770 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Sunil Govindan >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8870) [Submarine] Add submarine installation scripts
[
https://issues.apache.org/jira/browse/YARN-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704121#comment-16704121
]
Xun Liu commented on YARN-8870:
---
[~leftnoteasy]，[~vinodkv]，Ok, let me finish the work.
> [Submarine] Add submarine installation scripts
> --
>
> Key: YARN-8870
> URL: https://issues.apache.org/jira/browse/YARN-8870
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xun Liu
>Assignee: Xun Liu
>Priority: Critical
> Fix For: 3.2.0
>
> Attachments: YARN-8870.001.patch, YARN-8870.004.patch,
> YARN-8870.005.patch, YARN-8870.006.patch, YARN-8870.007.patch
>
>
> In order to reduce the deployment difficulty of Hadoop
> {Submarine} DNS, Docker, GPU, Network, graphics card, operating system kernel
> modification and other components, I specially developed this installation
> script to deploy Hadoop \{Submarine}
> runtime environment, providing one-click installation Scripts, which can also
> be used to install, uninstall, start, and stop individual components step by
> step.
>
> {color:#ff}design d{color}{color:#FF}ocument：{color}
> [https://docs.google.com/document/d/1muCTGFuUXUvM4JaDYjKqX5liQEg-AsNgkxfLMIFxYHU/edit?usp=sharing]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702626#comment-16702626 ] Xun Liu commented on YARN-5168: --- [~eyang], Thank you for your tips, I will finish this work. > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch, > YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700595#comment-16700595 ] Xun Liu commented on YARN-5168: --- Hi [~akhilpb]，My thoughts are the same as what you said. I will do this work myself. If there is a problem, I will ask for your help, thank you! :) > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch, > YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698797#comment-16698797 ] Xun Liu commented on YARN-5168: --- [~eyang]，Thank you for your tips, I will finish the rest of the work. > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch, > YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5168) Add port mapping handling when docker container use bridge network
[
https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698084#comment-16698084
]
Xun Liu commented on YARN-5168:
---
hi [~eyang]
{quote}1.add the new bits of json output to YARN UI 2
{quote}
I don't know where to display the port information in YARN UI 2.
{quote}2.make sure unit test and check style can pass.
{quote}
Done
{quote}3.Remove the container-executor.c hack of hard coding -P
{quote}
Done
> Add port mapping handling when docker container use bridge network
> --
>
> Key: YARN-5168
> URL: https://issues.apache.org/jira/browse/YARN-5168
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jun Gong
>Assignee: Xun Liu
>Priority: Major
> Labels: Docker
> Attachments: YARN-5168.001.patch, YARN-5168.002.patch,
> YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch,
> YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch
>
>
> YARN-4007 addresses different network setups when launching the docker
> container. We need support port mapping when docker container uses bridge
> network.
> The following problems are what we faced:
> 1. Add "-P" to map docker container's exposed ports to automatically.
> 2. Add "-p" to let user specify specific ports to map.
> 3. Add service registry support for bridge network case, then app could find
> each other. It could be done out of YARN, however it might be more convenient
> to support it natively in YARN.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu updated YARN-5168: -- Attachment: YARN-5168.008.patch > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch, > YARN-5168.006.patch, YARN-5168.007.patch, YARN-5168.008.patch > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu updated YARN-5168: -- Attachment: YARN-5168.007.patch > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch, > YARN-5168.006.patch, YARN-5168.007.patch > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu updated YARN-5168: -- Attachment: YARN-5168.006.patch > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch, > YARN-5168.006.patch > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu updated YARN-5168: -- Attachment: YARN-5168.005.patch > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch, YARN-5168.005.patch > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu updated YARN-5168: -- Attachment: YARN-5168.004.patch > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch, YARN-5168.004.patch > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu updated YARN-5168: -- Attachment: YARN-5168.003.patch > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch, YARN-5168.002.patch, > YARN-5168.003.patch > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8986) publish all exposed ports to random ports when using bridge network
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695770#comment-16695770 ] Xun Liu commented on YARN-8986: --- hi, [~Charo Zhang], [~eyang] Configure network plugins using calico, flannel + EDCD in docker. Created a parallel network, How to access the service of the specified port in the container? Is there a new JIRA that will solve this problem? > publish all exposed ports to random ports when using bridge network > --- > > Key: YARN-8986 > URL: https://issues.apache.org/jira/browse/YARN-8986 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.1.1 >Reporter: Charo Zhang >Assignee: Charo Zhang >Priority: Minor > Labels: Docker > Attachments: YARN-8986.001.patch, YARN-8986.002.patch, > YARN-8986.003.patch, YARN-8986.004.patch, YARN-8986.005.patch, > YARN-8986.006.patch, YARN-8986.007.patch, YARN-8986.008.patch > > > it's better to publish all exposed ports to random ports（-P） or support port > mapping（-p） for bridge network when using bridge network for docker container. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8986) publish all exposed ports to random ports when using bridge network
[
https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695628#comment-16695628
]
Xun Liu edited comment on YARN-8986 at 11/22/18 7:49 AM:
-
hi, [~Charo Zhang], [~eyang]
{code:java}
+if (strcasecmp(network_type, "bridge") != 0) {
+ ret = 0;
+ goto cleanup;
+}{code}
Not a bridge network, should it be able to perform port mapping as well?
We now use docker to create a parallel network via calico.
In the Hadoop submarine project, we still need to expose the services in the
container (for example: tensorboard, zeppelin, etc.).
was (Author: liuxun323):
hi, [~Charo Zhang], [~eyang]
{code:java}
+ if (network_name != NULL) {
+char *docker_network_command = make_string("%s network inspect %s
--format='{{.Driver}}'", docker_binary, network_name);
+FILE* docker_network = popen(docker_network_command, "r");
+ret = fscanf(docker_network, "%s", network_type);
+if (pclose (docker_network) != 0 || ret <= 0) {
+ fprintf (ERRORFILE, "Could not inspect docker network to get type
%s.\n", docker_network_command);
+ goto cleanup;
+}
+// other network type exit successfully without ports mapping
+if (strcasecmp(network_type, "bridge") != 0) {
+ ret = 0;
+ goto cleanup;
+}
+ }{code}
Not a bridge network, should it be able to perform port mapping as well?
We now use docker to create a parallel network via calico.
In the Hadoop submarine project, we still need to expose the services in the
container (for example: tensorboard, zeppelin, etc.).
> publish all exposed ports to random ports when using bridge network
> ---
>
> Key: YARN-8986
> URL: https://issues.apache.org/jira/browse/YARN-8986
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.1.1
>Reporter: Charo Zhang
>Assignee: Charo Zhang
>Priority: Minor
> Labels: Docker
> Attachments: YARN-8986.001.patch, YARN-8986.002.patch,
> YARN-8986.003.patch, YARN-8986.004.patch, YARN-8986.005.patch,
> YARN-8986.006.patch, YARN-8986.007.patch, YARN-8986.008.patch
>
>
> it's better to publish all exposed ports to random ports（-P） or support port
> mapping（-p） for bridge network when using bridge network for docker container.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8986) publish all exposed ports to random ports when using bridge network
[
https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695628#comment-16695628
]
Xun Liu commented on YARN-8986:
---
hi, [~Charo Zhang], [~eyang]
{code:java}
+ if (network_name != NULL) {
+char *docker_network_command = make_string("%s network inspect %s
--format='{{.Driver}}'", docker_binary, network_name);
+FILE* docker_network = popen(docker_network_command, "r");
+ret = fscanf(docker_network, "%s", network_type);
+if (pclose (docker_network) != 0 || ret <= 0) {
+ fprintf (ERRORFILE, "Could not inspect docker network to get type
%s.\n", docker_network_command);
+ goto cleanup;
+}
+// other network type exit successfully without ports mapping
+if (strcasecmp(network_type, "bridge") != 0) {
+ ret = 0;
+ goto cleanup;
+}
+ }{code}
Not a bridge network, should it be able to perform port mapping as well?
We now use docker to create a parallel network via calico.
In the Hadoop submarine project, we still need to expose the services in the
container (for example: tensorboard, zeppelin, etc.).
> publish all exposed ports to random ports when using bridge network
> ---
>
> Key: YARN-8986
> URL: https://issues.apache.org/jira/browse/YARN-8986
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.1.1
>Reporter: Charo Zhang
>Assignee: Charo Zhang
>Priority: Minor
> Labels: Docker
> Attachments: YARN-8986.001.patch, YARN-8986.002.patch,
> YARN-8986.003.patch, YARN-8986.004.patch, YARN-8986.005.patch,
> YARN-8986.006.patch, YARN-8986.007.patch, YARN-8986.008.patch
>
>
> it's better to publish all exposed ports to random ports（-P） or support port
> mapping（-p） for bridge network when using bridge network for docker container.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8714) [Submarine] Support files/tarballs to be localized for a training job.
[
https://issues.apache.org/jira/browse/YARN-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695501#comment-16695501
]
Xun Liu commented on YARN-8714:
---
Now when submarine uses an external algorithm file, Only need to have read and
execute permissions to meet the needs. Either submit your current
implementation first.
I feel , From the perspective of YARN, the requirements for file permissions
after mount, It should be that the original file and the file permissions after
mount should be consistent. Still ask [~leftnoteasy] for advice.
> [Submarine] Support files/tarballs to be localized for a training job.
> --
>
> Key: YARN-8714
> URL: https://issues.apache.org/jira/browse/YARN-8714
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Zhankun Tang
>Priority: Major
> Attachments: YARN-8714-WIP1-trunk-001.patch,
> YARN-8714-WIP1-trunk-002.patch
>
>
> See
> https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#heading=h.vkxp9edl11m7,
> {{job run --localizations ...}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8714) [Submarine] Support files/tarballs to be localized for a training job.
[
https://issues.apache.org/jira/browse/YARN-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692813#comment-16692813
]
Xun Liu edited comment on YARN-8714 at 11/20/18 8:22 AM:
-
[~tangzhankun] , I think there is no problem.
[~leftnoteasy]，This {color:#FF}--localizations{color} parameter name, is it
called {color:#FF}--mount{color} more suitable?
was (Author: liuxun323):
[~tangzhankun] , I think there is no problem.
> [Submarine] Support files/tarballs to be localized for a training job.
> --
>
> Key: YARN-8714
> URL: https://issues.apache.org/jira/browse/YARN-8714
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Zhankun Tang
>Priority: Major
> Attachments: YARN-8714-WIP1-trunk-001.patch,
> YARN-8714-WIP1-trunk-002.patch
>
>
> See
> https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#heading=h.vkxp9edl11m7,
> {{job run --localizations ...}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8714) [Submarine] Support files/tarballs to be localized for a training job.
[
https://issues.apache.org/jira/browse/YARN-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692813#comment-16692813
]
Xun Liu commented on YARN-8714:
---
[~tangzhankun] , I think there is no problem.
> [Submarine] Support files/tarballs to be localized for a training job.
> --
>
> Key: YARN-8714
> URL: https://issues.apache.org/jira/browse/YARN-8714
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Zhankun Tang
>Priority: Major
> Attachments: YARN-8714-WIP1-trunk-001.patch,
> YARN-8714-WIP1-trunk-002.patch
>
>
> See
> https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#heading=h.vkxp9edl11m7,
> {{job run --localizations ...}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688912#comment-16688912 ] Xun Liu commented on YARN-5168: --- [~eyang] Thank you for your previous work, I will follow the prompts to complete the rest of the work. thank you very much! > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > Attachments: YARN-5168.001.patch > > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8986) publish all exposed ports to random ports when using bridge network
[
https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16687588#comment-16687588
]
Xun Liu commented on YARN-8986:
---
[~Charo Zhang]，[~eyang]
Because hadoop submarine relies on this feature, it also needs to be submitted
to the {color:#FF}trunk{color} branch.
> publish all exposed ports to random ports when using bridge network
> ---
>
> Key: YARN-8986
> URL: https://issues.apache.org/jira/browse/YARN-8986
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.1.1
>Reporter: Charo Zhang
>Assignee: Charo Zhang
>Priority: Minor
> Labels: Docker
> Fix For: 3.1.2
>
> Attachments: 20181108155450.png, YARN-8986.001.patch,
> YARN-8986.002.patch, YARN-8986.003.patch
>
>
> it's better to publish all exposed ports to random ports（-P） or support port
> mapping（-p） for bridge network when using bridge network for docker container.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5168) Add port mapping handling when docker container use bridge network
[
https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16687527#comment-16687527
]
Xun Liu edited comment on YARN-5168 at 11/15/18 7:01 AM:
-
[~eyang]
Thank you for your reply, remove the support of "-p port1:port2", I think so.
{quote}1. Add "-P" to map docker container's exposed ports to automatically.
{quote}
When the yarn starts docker, will this "-P" parameter be added? This way we can
expose the services inside the container, for example:
So we can add it inside the Dockerfile
{code:java}
EXPOSE 1000
EXPOSE 2000
EXPOSE 3000
{code}
After launching docker via YARN, We can get the external port automatically
assigned by the host.
{code:java}
$ docker run d -P -name exposed-ports-in-dockerfile exposed-ports
63264dae9db85c5d667a37dac77e0da7c8d2d699f49b69ba992485242160ad3a
$ docker port exposed-ports-in-dockerfile
1000/tcp -> 0.0.0.0:49156
2000/tcp -> 0.0.0.0:49157
3000/tcp -> 0.0.0.0:49158
{code}
{quote}1. Add "-P" to map docker container's exposed ports to automatically.
{quote}
When will this feature be added? I feel that waiting for hadoop3.3 to release
is a bit late, Can you assign this JIRA to me and let me contribute the patch?
thank you!
was (Author: liuxun323):
[~eyang]
Thank you for your reply, remove the support of "-p port1:port2", I think so.
{quote}1. Add "-P" to map docker container's exposed ports to automatically.
{quote}
When the yarn starts docker, will this "-P" parameter be added? This way we can
expose the services inside the container, for example:
So we can add it inside the Dockerfile
{code:java}
EXPOSE 1000
EXPOSE 2000
EXPOSE 3000
{code}
After launching docker via YARN, We can get the external port automatically
assigned by the host.
{code:java}
$ docker run d -P -name exposed-ports-in-dockerfile exposed-ports
63264dae9db85c5d667a37dac77e0da7c8d2d699f49b69ba992485242160ad3a
$ docker port exposed-ports-in-dockerfile
1000/tcp -> 0.0.0.0:49156
2000/tcp -> 0.0.0.0:49157
3000/tcp -> 0.0.0.0:49158
{code}
{quote}1. Add "-P" to map docker container's exposed ports to automatically.
{quote}
When will this feature be added? I feel that waiting for hadoop3.3 to release
is a bit late, Can you assign this JIRA to me and let me contribute the patch?
thank you!
> Add port mapping handling when docker container use bridge network
> --
>
> Key: YARN-5168
> URL: https://issues.apache.org/jira/browse/YARN-5168
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jun Gong
>Assignee: Eric Yang
>Priority: Major
> Labels: Docker
>
> YARN-4007 addresses different network setups when launching the docker
> container. We need support port mapping when docker container uses bridge
> network.
> The following problems are what we faced:
> 1. Add "-P" to map docker container's exposed ports to automatically.
> 2. Add "-p" to let user specify specific ports to map.
> 3. Add service registry support for bridge network case, then app could find
> each other. It could be done out of YARN, however it might be more convenient
> to support it natively in YARN.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5168) Add port mapping handling when docker container use bridge network
[
https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16687527#comment-16687527
]
Xun Liu commented on YARN-5168:
---
[~eyang]
Thank you for your reply, remove the support of "-p port1:port2", I think so.
{quote}1. Add "-P" to map docker container's exposed ports to automatically.
{quote}
When the yarn starts docker, will this "-P" parameter be added? This way we can
expose the services inside the container, for example:
So we can add it inside the Dockerfile
EXPOSE 1000
EXPOSE 2000
EXPOSE 3000
After launching docker via YARN, We can get the external port automatically
assigned by the host.
$ docker run -d -P --name exposed-ports-in-dockerfile exposed-ports
63264dae9db85c5d667a37dac77e0da7c8d2d699f49b69ba992485242160ad3a
$ docker port exposed-ports-in-dockerfile
1000/tcp -> 0.0.0.0:49156
2000/tcp -> 0.0.0.0:49157
3000/tcp -> 0.0.0.0:49158
{quote}1. Add "-P" to map docker container's exposed ports to automatically.
{quote}
When will this feature be added? I feel that waiting for hadoop3.3 to release
is a bit late, Can you assign this JIRA to me and let me contribute the patch?
thank you!
> Add port mapping handling when docker container use bridge network
> --
>
> Key: YARN-5168
> URL: https://issues.apache.org/jira/browse/YARN-5168
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jun Gong
>Assignee: Eric Yang
>Priority: Major
> Labels: Docker
>
> YARN-4007 addresses different network setups when launching the docker
> container. We need support port mapping when docker container uses bridge
> network.
> The following problems are what we faced:
> 1. Add "-P" to map docker container's exposed ports to automatically.
> 2. Add "-p" to let user specify specific ports to map.
> 3. Add service registry support for bridge network case, then app could find
> each other. It could be done out of YARN, however it might be more convenient
> to support it natively in YARN.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5168) Add port mapping handling when docker container use bridge network
[
https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16687527#comment-16687527
]
Xun Liu edited comment on YARN-5168 at 11/15/18 6:11 AM:
-
[~eyang]
Thank you for your reply, remove the support of "-p port1:port2", I think so.
{quote}1. Add "-P" to map docker container's exposed ports to automatically.
{quote}
When the yarn starts docker, will this "-P" parameter be added? This way we can
expose the services inside the container, for example:
So we can add it inside the Dockerfile
{code:java}
EXPOSE 1000
EXPOSE 2000
EXPOSE 3000
{code}
After launching docker via YARN, We can get the external port automatically
assigned by the host.
{code:java}
$ docker run d -P -name exposed-ports-in-dockerfile exposed-ports
63264dae9db85c5d667a37dac77e0da7c8d2d699f49b69ba992485242160ad3a
$ docker port exposed-ports-in-dockerfile
1000/tcp -> 0.0.0.0:49156
2000/tcp -> 0.0.0.0:49157
3000/tcp -> 0.0.0.0:49158
{code}
{quote}1. Add "-P" to map docker container's exposed ports to automatically.
{quote}
When will this feature be added? I feel that waiting for hadoop3.3 to release
is a bit late, Can you assign this JIRA to me and let me contribute the patch?
thank you!
was (Author: liuxun323):
[~eyang]
Thank you for your reply, remove the support of "-p port1:port2", I think so.
{quote}1. Add "-P" to map docker container's exposed ports to automatically.
{quote}
When the yarn starts docker, will this "-P" parameter be added? This way we can
expose the services inside the container, for example:
So we can add it inside the Dockerfile
EXPOSE 1000
EXPOSE 2000
EXPOSE 3000
After launching docker via YARN, We can get the external port automatically
assigned by the host.
$ docker run -d -P --name exposed-ports-in-dockerfile exposed-ports
63264dae9db85c5d667a37dac77e0da7c8d2d699f49b69ba992485242160ad3a
$ docker port exposed-ports-in-dockerfile
1000/tcp -> 0.0.0.0:49156
2000/tcp -> 0.0.0.0:49157
3000/tcp -> 0.0.0.0:49158
{quote}1. Add "-P" to map docker container's exposed ports to automatically.
{quote}
When will this feature be added? I feel that waiting for hadoop3.3 to release
is a bit late, Can you assign this JIRA to me and let me contribute the patch?
thank you!
> Add port mapping handling when docker container use bridge network
> --
>
> Key: YARN-5168
> URL: https://issues.apache.org/jira/browse/YARN-5168
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jun Gong
>Assignee: Eric Yang
>Priority: Major
> Labels: Docker
>
> YARN-4007 addresses different network setups when launching the docker
> container. We need support port mapping when docker container uses bridge
> network.
> The following problems are what we faced:
> 1. Add "-P" to map docker container's exposed ports to automatically.
> 2. Add "-p" to let user specify specific ports to map.
> 3. Add service registry support for bridge network case, then app could find
> each other. It could be done out of YARN, however it might be more convenient
> to support it natively in YARN.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5168) Add port mapping handling when docker container use bridge network
[
https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16687426#comment-16687426
]
Xun Liu commented on YARN-5168:
---
[~eyang]
I saw [YARN-8569|https://issues.apache.org/jira/browse/YARN-8569], If I can get
the container internal port and the port mapping information allocated on the
host through service.json, I can also meet my needs.
My new question is:
{quote}We probably want to drop support for adhoc "-p"
{quote}
What command or parameter should I use to tell YARN, Do I need to expose the
service port in my container?
Exposing the specified port of the service should be specified at runtime. If
the port mapping needs to use the EXPOSE command in the Dockerfile, it is too
difficult to use.
> Add port mapping handling when docker container use bridge network
> --
>
> Key: YARN-5168
> URL: https://issues.apache.org/jira/browse/YARN-5168
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jun Gong
>Assignee: Eric Yang
>Priority: Major
> Labels: Docker
>
> YARN-4007 addresses different network setups when launching the docker
> container. We need support port mapping when docker container uses bridge
> network.
> The following problems are what we faced:
> 1. Add "-P" to map docker container's exposed ports to automatically.
> 2. Add "-p" to let user specify specific ports to map.
> 3. Add service registry support for bridge network case, then app could find
> each other. It could be done out of YARN, however it might be more convenient
> to support it natively in YARN.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5168) Add port mapping handling when docker container use bridge network
[
https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686634#comment-16686634
]
Xun Liu commented on YARN-5168:
---
I think [~shaneku...@gmail.com] said that the port conflict is such a result.
{quote}I believe supporting -P will lead to port conflicts. What if two
containers running on the same NM both expose 8080?
{quote}
Both applications use the -p 8080:8080 parameter to create a port mapping for
the container. The second application specifies that the host's 8080 port will
definitely conflict.
However, this is not a problem, because kubernetes will conflict if you specify
the port when creating the pod. What we need to do is to send the conflict
information to the user if the conflict occurs, and let the user create a port.
I think [~eyang] said very important.
{quote}2. Add "-p" to let user specify specific ports to map.
{quote}
The -P parameter is very useful. You can do this -P 8080, let the host pick an
unoccupied port and bind it to the 8080 port of the container. Then the user
can know which physical port is used by obtaining the container information.
{quote}3. Add service registry support for bridge network case, then app could
find each other. It could be done out of YARN, however it might be more
convenient to support it natively in YARN.
{quote}
Just like kubernetes, the container service in the YARN cluster is reversed by
ngxin or traefik, so that it can be used regardless of the container changes.
With these network features supported, YARN can provide better online services
through docker.
[~eyang]，The hadoop submarine project requires a port mapping feature very
much. Can you assign this JIRA to me and let me contribute the patch? thank you!
> Add port mapping handling when docker container use bridge network
> --
>
> Key: YARN-5168
> URL: https://issues.apache.org/jira/browse/YARN-5168
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jun Gong
>Assignee: Eric Yang
>Priority: Major
> Labels: Docker
>
> YARN-4007 addresses different network setups when launching the docker
> container. We need support port mapping when docker container uses bridge
> network.
> The following problems are what we faced:
> 1. Add "-P" to map docker container's exposed ports to automatically.
> 2. Add "-p" to let user specify specific ports to map.
> 3. Add service registry support for bridge network case, then app could find
> each other. It could be done out of YARN, however it might be more convenient
> to support it natively in YARN.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8714) [Submarine] Support files/tarballs to be localized for a training job.
[
https://issues.apache.org/jira/browse/YARN-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684794#comment-16684794
]
Xun Liu edited comment on YARN-8714 at 11/14/18 9:02 AM:
-
1) Support for multiple files or directories.
2) Must maintain the original state, such as folder subdirectory structure, zip
file or normal file, do not make any changes.
3) Mount the specified file or folder to the absolute path in the container.
Because the user can customize the WORKDIR of the container through the
Dockerfile, And the containers are dedicated and custom made, So the user knows
exactly where to mount the file.
4) Keep consistent with the use of docker, use`{color:#ff}:{color}` split
source directory and destination directory.
5) Parameter format: -localizations
hdfs:///user/yarn{color:#ff}:{color}/absolute/path
Requirements document:
[https://docs.google.com/document/d/16YN8Kjmxt1Ym3clx5pDnGNXGajUT36hzQxjaik1cP4A/edit#heading=h.s07ukakieg7q]
was (Author: liuxun323):
1) Support for multiple files or directories.
2) Must maintain the original state, such as folder subdirectory structure, zip
file or normal file, do not make any changes.
3) Mount the specified file or folder to the absolute path in the container.
Because the user can customize the WORKDIR of the container through the
Dockerfile, And the containers are dedicated and custom made, So the user knows
exactly where to mount the file.
4) Keep consistent with the use of docker, use`{color:#FF}:{color}` split
source directory and destination directory.
5) Parameter format: -localizations
hdfs:///user/yarn{color:#FF}:{color}/absolute/path
> [Submarine] Support files/tarballs to be localized for a training job.
> --
>
> Key: YARN-8714
> URL: https://issues.apache.org/jira/browse/YARN-8714
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Zhankun Tang
>Priority: Major
> Attachments: YARN-8714-WIP1-trunk-001.patch
>
>
> See
> https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#heading=h.vkxp9edl11m7,
> {{job run --localizations ...}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8714) [Submarine] Support files/tarballs to be localized for a training job.
[
https://issues.apache.org/jira/browse/YARN-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684794#comment-16684794
]
Xun Liu edited comment on YARN-8714 at 11/14/18 7:52 AM:
-
1) Support for multiple files or directories.
2) Must maintain the original state, such as folder subdirectory structure, zip
file or normal file, do not make any changes.
3) Mount the specified file or folder to the absolute path in the container.
Because the user can customize the WORKDIR of the container through the
Dockerfile, And the containers are dedicated and custom made, So the user knows
exactly where to mount the file.
4) Keep consistent with the use of docker, use`{color:#FF}:{color}` split
source directory and destination directory.
5) Parameter format: -localizations
hdfs:///user/yarn{color:#FF}:{color}/absolute/path
was (Author: liuxun323):
# Support files and folders
# HDFS type and local file type supporting HDFS:// prefix at the same time
# Keep it intact: the uploaded package is compressed, and the submarine is
automatically decompressed into the container, which is not suitable, because
if I rush to save the file that needs to upload the compressed package format,
it will be destroyed. And it also introduces ambiguity.
# Parameter format: {color:#FF}-localizations
hdfs:///user/yarn>{color:#ff}.{color}{color} # indicates the current
execution path of the container
# Parameter format: {color:#FF}-localizations
hdfs:///user/yarn>./abc{color} # Indicates the abc folder under the current
execution path of the container (submarine marks the file under
hdfs:///user/yarn as an abc .tar.gz compression package, extract the abc folder
when pulling up the container, then mount it in it)
> [Submarine] Support files/tarballs to be localized for a training job.
> --
>
> Key: YARN-8714
> URL: https://issues.apache.org/jira/browse/YARN-8714
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Zhankun Tang
>Priority: Major
> Attachments: YARN-8714-WIP1-trunk-001.patch
>
>
> See
> https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#heading=h.vkxp9edl11m7,
> {{job run --localizations ...}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8619) Automate docker network configuration through YARN API
[
https://issues.apache.org/jira/browse/YARN-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684849#comment-16684849
]
Xun Liu commented on YARN-8619:
---
Hadoop-yarn-submarine already provides an installation environment and usage
documentation for the runtime environment (operating system kernel
modifications, parallel networks, etc.)
[https://github.com/apache/hadoop/tree/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/installation]
[https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/site/markdown/InstallationScriptEN.md]
[https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/site/markdown/InstallationScriptCN.md]
This tool has been used to install the runtime environment for deploying docker
and to build parallel networks between docker containers.
> Automate docker network configuration through YARN API
> --
>
> Key: YARN-8619
> URL: https://issues.apache.org/jira/browse/YARN-8619
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-native-services
>Reporter: Eric Yang
>Priority: Major
> Labels: Docker
>
> Docker supports bridge, host, overlay, macvlan networking. It might be
> useful to automate docker network setup through a set of YARN API to improve
> management of docker networks. Each type of network driver requires
> different type of parameters. For Hadoop use case, it seems more useful to
> focus on macvlan networking for ease of use and configuration. It would be
> great addition to support commands like:
> {code}
> yarn network create -d macvlan \
> --subnet=172.16.86.0/24 \
> --gateway=172.16.86.1 \
> -o parent=eth0 \
> my-macvlan-net
> {code}
> This changes docker configuration to hosts managed by YARN.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8714) [Submarine] Support files/tarballs to be localized for a training job.
[
https://issues.apache.org/jira/browse/YARN-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684794#comment-16684794
]
Xun Liu edited comment on YARN-8714 at 11/13/18 7:01 AM:
-
# Support files and folders
# HDFS type and local file type supporting HDFS:// prefix at the same time
# Keep it intact: the uploaded package is compressed, and the submarine is
automatically decompressed into the container, which is not suitable, because
if I rush to save the file that needs to upload the compressed package format,
it will be destroyed. And it also introduces ambiguity.
# Parameter format: {color:#FF}-localizations
hdfs:///user/yarn>{color:#ff}.{color}{color} # indicates the current
execution path of the container
# Parameter format: {color:#FF}-localizations
hdfs:///user/yarn>./abc{color} # Indicates the abc folder under the current
execution path of the container (submarine marks the file under
hdfs:///user/yarn as an abc .tar.gz compression package, extract the abc folder
when pulling up the container, then mount it in it)
was (Author: liuxun323):
# Support files and folders
# HDFS type and local file type supporting HDFS:// prefix at the same time
# Keep it intact: the uploaded package is compressed, and the submarine is
automatically decompressed into the container, which is not suitable, because
if I rush to save the file that needs to upload the compressed package format,
it will be destroyed. And it also introduces ambiguity.
# Parameter format: {color:#FF}--localizations hdfs:///user/yarn->.{color}
# indicates the current execution path of the container
# Parameter format: {color:#FF}--localizations
hdfs:///user/yarn->./abc{color} # Indicates the abc folder under the current
execution path of the container (submarine marks the file under
hdfs:///user/yarn as an abc .tar.gz compression package, extract the abc folder
when pulling up the container, then mount it in it)
> [Submarine] Support files/tarballs to be localized for a training job.
> --
>
> Key: YARN-8714
> URL: https://issues.apache.org/jira/browse/YARN-8714
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Zhankun Tang
>Priority: Major
> Attachments: YARN-8714-WIP1-trunk-001.patch
>
>
> See
> https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#heading=h.vkxp9edl11m7,
> {{job run --localizations ...}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
***UNCHECKED*** [jira] [Commented] (YARN-8714) [Submarine] Support files/tarballs to be localized for a training job.
[
https://issues.apache.org/jira/browse/YARN-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684794#comment-16684794
]
Xun Liu commented on YARN-8714:
---
# Support files and folders
# HDFS type and local file type supporting HDFS:// prefix at the same time
# Keep it intact: the uploaded package is compressed, and the submarine is
automatically decompressed into the container, which is not suitable, because
if I rush to save the file that needs to upload the compressed package format,
it will be destroyed. And it also introduces ambiguity.
# Parameter format: {color:#FF}--localizations hdfs:///user/yarn->.{color}
# indicates the current execution path of the container
# Parameter format: {color:#FF}--localizations
hdfs:///user/yarn->./abc{color} # Indicates the abc folder under the current
execution path of the container (submarine marks the file under
hdfs:///user/yarn as an abc .tar.gz compression package, extract the abc folder
when pulling up the container, then mount it in it)
> [Submarine] Support files/tarballs to be localized for a training job.
> --
>
> Key: YARN-8714
> URL: https://issues.apache.org/jira/browse/YARN-8714
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Zhankun Tang
>Priority: Major
> Attachments: YARN-8714-WIP1-trunk-001.patch
>
>
> See
> https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#heading=h.vkxp9edl11m7,
> {{job run --localizations ...}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684791#comment-16684791 ] Xun Liu commented on YARN-5168: --- [~eyang]，We really need to map this feature to the port. May I ask when this JIRA can be completed, thank you! > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Eric Yang >Priority: Major > Labels: Docker > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8986) publish all exposed ports to random ports when using bridge network
[ https://issues.apache.org/jira/browse/YARN-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683370#comment-16683370 ] Xun Liu commented on YARN-8986: --- The screenshot in your attachment, Isn't the port 80 inside the container mapped to port 32773 on the server? Please describe your problem in more detail. > publish all exposed ports to random ports when using bridge network > --- > > Key: YARN-8986 > URL: https://issues.apache.org/jira/browse/YARN-8986 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.1.1 >Reporter: Charo Zhang >Priority: Minor > Fix For: 3.1.2 > > Attachments: 20181108155450.png > > > it's better to publish all exposed ports to random ports when using bridge > network for docker container. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8981) Virtual IP address support
[ https://issues.apache.org/jira/browse/YARN-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683355#comment-16683355 ] Xun Liu commented on YARN-8981: --- hi, [~Tagar] The virtual IP inside the YARN cluster is already supported by a parallel network created by calico. You can refer to the installation instructions for the Docker environment: [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/site/markdown/InstallationScriptEN.md] And install using the runtime environment installation script; [https://github.com/apache/hadoop/tree/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/installation] It is possible to have the Docker container in YARN run in an internal virtual parallel network. However, after YARN introduced Docker, there are still some features that need to be strengthened in the support of the network. For example, how the services in the internally virtualized Docker container access the network environment outside the cluster. In Kubernetes, the reverse proxy can be performed by the ingress latter Traefik. > Virtual IP address support > -- > > Key: YARN-8981 > URL: https://issues.apache.org/jira/browse/YARN-8981 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 3.0.2, 3.1.1 >Reporter: Ruslan Dautkhanov >Priority: Major > Labels: DNS, Docker, docker, service, service-engine, > service-orchestration, virtual_hosts > > I couldn't find support for virtual IP addresses in YARN framework. > This would be great if we have a docker-on-yarn service and if it for example > has to be failed over to another physical host, clients can still find it. > So the idea is for YARN to bring up that virtual IP address (an > additional/secondary IP address ) on a physical host where that particular > docker container is running, so the clients that use that container's > services don't have to change connection details every time that container > moves around in YARN cluster. > Similarly to virtual IP addresses in Kubernetes world: > [https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies] > One implementation could be through `ip address add` \ `ip address remove`. > Kubernetes uses a more complicated `kube-proxy`, similarly to `docker-proxy` > process in pure dockers / non-kubernetes docker deployments. > Another approach is running a separate DNS service for a DNS subdomain (main > DNS server would have to forward all requests for that DNS subdomain to a > YARN DNS service). In Oracle Clusterware similar process is called GNS: > https://docs.oracle.com/en/database/oracle/oracle-database/12.2/cwsol/about-the-grid-naming-service-vip-address.html#GUID-A4EE0CC6-A5F1-4507-82D6-D5C43E0F1584 > Would be great to have support for either virtual IP addresses managed by > YARN directly or something similar to Oracle's GNS dns service. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8968) [Submarine] Support for executing tensorflow algorithm files on hdfs
[
https://issues.apache.org/jira/browse/YARN-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xun Liu updated YARN-8968:
--
Description:
Now the submarine project can only execute the algorithm files in the docker
image.
Need support for executing tensorflow algorithm files on HDFS.
1. The user uploads the algorithm file to the HDFS directory manually or via
zeppelin.
2. In the base image provided by submarine, Increase the path parameters
according to the specified algorithm, Download the algorithm file from HDFS to
the `/tmp/` directory.
{code:java}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/main/docker/base/ubuntu-16.04/Dockerfile.cpu.tf_1.8.0
and
hadoop-yarn-project/hadoop
-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/main/docker/base/ubuntu-16.04/Dockerfile.gpu.tf_1.8.0{code}
3. Add parameters to the algorithm file path in submarine project.
was:
Now the submarine project can only execute the algorithm files in the docker
image.
Need support for executing tensorflow algorithm files on HDFS.
# The user uploads the algorithm file to the HDFS directory manually or via
zeppelin.
# In the base image provided by submarine, Increase the path parameters
according to the specified algorithm, Download the algorithm file from HDFS to
the `/tmp/` directory.
{code:java}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/main/docker/base/ubuntu-16.04/Dockerfile.cpu.tf_1.8.0
and
hadoop-yarn-project/hadoop
-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/main/docker/base/ubuntu-16.04/Dockerfile.gpu.tf_1.8.0{code}
# Add parameters to the algorithm file path in submarine project.
> [Submarine] Support for executing tensorflow algorithm files on hdfs
>
>
> Key: YARN-8968
> URL: https://issues.apache.org/jira/browse/YARN-8968
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xun Liu
>Assignee: Xun Liu
>Priority: Major
>
> Now the submarine project can only execute the algorithm files in the docker
> image.
> Need support for executing tensorflow algorithm files on HDFS.
> 1. The user uploads the algorithm file to the HDFS directory manually or via
> zeppelin.
> 2. In the base image provided by submarine, Increase the path parameters
> according to the specified algorithm, Download the algorithm file from HDFS
> to the `/tmp/` directory.
> {code:java}
> hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/main/docker/base/ubuntu-16.04/Dockerfile.cpu.tf_1.8.0
>
> and
> hadoop-yarn-project/hadoop
> -yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/main/docker/base/ubuntu-16.04/Dockerfile.gpu.tf_1.8.0{code}
> 3. Add parameters to the algorithm file path in submarine project.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8968) [Submarine] Support for executing tensorflow algorithm files on hdfs
[
https://issues.apache.org/jira/browse/YARN-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xun Liu updated YARN-8968:
--
Description:
Now the submarine project can only execute the algorithm files in the docker
image.
Need support for executing tensorflow algorithm files on HDFS.
# The user uploads the algorithm file to the HDFS directory manually or via
zeppelin.
# In the base image provided by submarine, Increase the path parameters
according to the specified algorithm, Download the algorithm file from HDFS to
the `/tmp/` directory.
{code:java}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/main/docker/base/ubuntu-16.04/Dockerfile.cpu.tf_1.8.0
and
hadoop-yarn-project/hadoop
-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/main/docker/base/ubuntu-16.04/Dockerfile.gpu.tf_1.8.0{code}
# Add parameters to the algorithm file path in submarine project.
was:
Now the submarine project can only execute the algorithm files in the image.
Need support for executing tensorflow algorithm files on HDFS.
# The user uploads the algorithm file to the HDFS directory manually or via
zeppelin.
# In the base image provided by submarine, Increase the path parameters
according to the specified algorithm, Download the algorithm file from HDFS to
the `/tmp/` directory.
{code:java}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/main/docker/base/ubuntu-16.04/Dockerfile.cpu.tf_1.8.0
and
hadoop-yarn-project/hadoop
-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/main/docker/base/ubuntu-16.04/Dockerfile.gpu.tf_1.8.0{code}
# Add parameters to the algorithm file path in submarine project.
> [Submarine] Support for executing tensorflow algorithm files on hdfs
>
>
> Key: YARN-8968
> URL: https://issues.apache.org/jira/browse/YARN-8968
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xun Liu
>Assignee: Xun Liu
>Priority: Major
>
> Now the submarine project can only execute the algorithm files in the docker
> image.
> Need support for executing tensorflow algorithm files on HDFS.
> # The user uploads the algorithm file to the HDFS directory manually or via
> zeppelin.
> # In the base image provided by submarine, Increase the path parameters
> according to the specified algorithm, Download the algorithm file from HDFS
> to the `/tmp/` directory.
> {code:java}
> hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/main/docker/base/ubuntu-16.04/Dockerfile.cpu.tf_1.8.0
>
> and
> hadoop-yarn-project/hadoop
> -yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/main/docker/base/ubuntu-16.04/Dockerfile.gpu.tf_1.8.0{code}
> # Add parameters to the algorithm file path in submarine project.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8968) [Submarine] Support for executing tensorflow algorithm files on hdfs
Xun Liu created YARN-8968:
-
Summary: [Submarine] Support for executing tensorflow algorithm
files on hdfs
Key: YARN-8968
URL: https://issues.apache.org/jira/browse/YARN-8968
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Xun Liu
Assignee: Xun Liu
Now the submarine project can only execute the algorithm files in the image.
Need support for executing tensorflow algorithm files on HDFS.
# The user uploads the algorithm file to the HDFS directory manually or via
zeppelin.
# In the base image provided by submarine, Increase the path parameters
according to the specified algorithm, Download the algorithm file from HDFS to
the `/tmp/` directory.
{code:java}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/main/docker/base/ubuntu-16.04/Dockerfile.cpu.tf_1.8.0
and
hadoop-yarn-project/hadoop
-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/main/docker/base/ubuntu-16.04/Dockerfile.gpu.tf_1.8.0{code}
# Add parameters to the algorithm file path in submarine project.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8926) [Submarine] Manage submarine's docker image with docker hub
Xun Liu created YARN-8926: - Summary: [Submarine] Manage submarine's docker image with docker hub Key: YARN-8926 URL: https://issues.apache.org/jira/browse/YARN-8926 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xun Liu Assignee: Xun Liu Currently the deep learning algorithm file is packaged into the docker image file. In the formal use of the deep learning system, the algorithm files are generally divided into the following ways: 1. The fixed algorithm will be made into a separate docker image, which is convenient for maintenance and testing. Image files with different algorithms built in are managed by mirrored warehouse. 2. We also need to dynamically pass in algorithm scripts so that algorithm developers can test and run algorithm scripts without packaging. 3. In addition, submarine needs to support different deep learning frameworks in the future. We need to manage the image that submarine needs to use through the docker hub. And we need to develop some specifications to manage these images. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8875) [Submarine] Add documentation for submarine installation script details
[ https://issues.apache.org/jira/browse/YARN-8875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu updated YARN-8875: -- Attachment: YARN-8875.004.patch > [Submarine] Add documentation for submarine installation script details > --- > > Key: YARN-8875 > URL: https://issues.apache.org/jira/browse/YARN-8875 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xun Liu >Assignee: Xun Liu >Priority: Critical > Attachments: YARN-8875.001.patch, YARN-8875.002.patch, > YARN-8875.003.patch, YARN-8875.004.patch > > > YARN-8870: submarine installation guide -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8876) [Submarine] Job monitor long-running service of submarine
[ https://issues.apache.org/jira/browse/YARN-8876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu updated YARN-8876: -- Summary: [Submarine] Job monitor long-running service of submarine (was: [Submarine] Job monitor service of submarine) > [Submarine] Job monitor long-running service of submarine > - > > Key: YARN-8876 > URL: https://issues.apache.org/jira/browse/YARN-8876 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xun Liu >Assignee: Xun Liu >Priority: Major > > h1. Job monitor service of submarine > After training, the monitoring program need auto close PS service. It is > possible that other deep learning frameworks also have some custom processing > when the tasks are in different states. > The submarine needs to provide a long-term resident service that monitors > each JOB mission. > This monitoring service can be processed differently according to the > training tasks of different depth learning framework types. > For example: Tensorflow performs distributed training, when the training is > completed, > The PS service cannot be automatically stopped. At this time, the PS needs to > be actively stopped by the monitoring service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8876) [Submarine] Job monitor long-running service of submarine
[ https://issues.apache.org/jira/browse/YARN-8876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu updated YARN-8876: -- Description: h1. Job monitor long-running service of submarine After training, the monitoring program need auto close PS service. It is possible that other deep learning frameworks also have some custom processing when the tasks are in different states. The submarine needs to provide a long-term resident service that monitors each JOB mission. This monitoring service can be processed differently according to the training tasks of different depth learning framework types. For example: Tensorflow performs distributed training, when the training is completed, The PS service cannot be automatically stopped. At this time, the PS needs to be actively stopped by the monitoring service. was: h1. Job monitor service of submarine After training, the monitoring program need auto close PS service. It is possible that other deep learning frameworks also have some custom processing when the tasks are in different states. The submarine needs to provide a long-term resident service that monitors each JOB mission. This monitoring service can be processed differently according to the training tasks of different depth learning framework types. For example: Tensorflow performs distributed training, when the training is completed, The PS service cannot be automatically stopped. At this time, the PS needs to be actively stopped by the monitoring service. > [Submarine] Job monitor long-running service of submarine > - > > Key: YARN-8876 > URL: https://issues.apache.org/jira/browse/YARN-8876 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xun Liu >Assignee: Xun Liu >Priority: Major > > h1. Job monitor long-running service of submarine > After training, the monitoring program need auto close PS service. It is > possible that other deep learning frameworks also have some custom processing > when the tasks are in different states. > The submarine needs to provide a long-term resident service that monitors > each JOB mission. > This monitoring service can be processed differently according to the > training tasks of different depth learning framework types. > For example: Tensorflow performs distributed training, when the training is > completed, > The PS service cannot be automatically stopped. At this time, the PS needs to > be actively stopped by the monitoring service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8876) [Submarine] Job monitor service of submarine
[ https://issues.apache.org/jira/browse/YARN-8876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu updated YARN-8876: -- Description: h1. Job monitor service of submarine After training, the monitoring program need auto close PS service. It is possible that other deep learning frameworks also have some custom processing when the tasks are in different states. The submarine needs to provide a long-term resident service that monitors each JOB mission. This monitoring service can be processed differently according to the training tasks of different depth learning framework types. For example: Tensorflow performs distributed training, when the training is completed, The PS service cannot be automatically stopped. At this time, the PS needs to be actively stopped by the monitoring service. was: h1. Job monitor of submarine After training, the monitoring program need auto close PS service. It is possible that other deep learning frameworks also have some custom processing when the tasks are in different states. The submarine needs to provide a long-term resident service that monitors each JOB mission. This monitoring service can be processed differently according to the training tasks of different depth learning framework types. For example: Tensorflow performs distributed training, when the training is completed, The PS service cannot be automatically stopped. At this time, the PS needs to be actively stopped by the monitoring service. > [Submarine] Job monitor service of submarine > > > Key: YARN-8876 > URL: https://issues.apache.org/jira/browse/YARN-8876 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xun Liu >Assignee: Xun Liu >Priority: Major > > h1. Job monitor service of submarine > After training, the monitoring program need auto close PS service. It is > possible that other deep learning frameworks also have some custom processing > when the tasks are in different states. > The submarine needs to provide a long-term resident service that monitors > each JOB mission. > This monitoring service can be processed differently according to the > training tasks of different depth learning framework types. > For example: Tensorflow performs distributed training, when the training is > completed, > The PS service cannot be automatically stopped. At this time, the PS needs to > be actively stopped by the monitoring service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8876) [Submarine] Job monitor service of submarine
[ https://issues.apache.org/jira/browse/YARN-8876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu updated YARN-8876: -- Summary: [Submarine] Job monitor service of submarine (was: [Submarine] Job monitor of submarine) > [Submarine] Job monitor service of submarine > > > Key: YARN-8876 > URL: https://issues.apache.org/jira/browse/YARN-8876 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xun Liu >Assignee: Xun Liu >Priority: Major > > h1. Job monitor of submarine > After training, the monitoring program need auto close PS service. > It is possible that other deep learning frameworks also have some custom > processing when the tasks are in different states. > The submarine needs to provide a long-term resident service that monitors > each JOB mission. > This monitoring service can be processed differently according to the > training tasks of different depth learning framework types. > For example: Tensorflow performs distributed training, when the training is > completed, > The PS service cannot be automatically stopped. At this time, the PS needs to > be actively stopped by the monitoring service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8876) [Submarine] Job monitor of submarine
[
https://issues.apache.org/jira/browse/YARN-8876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xun Liu updated YARN-8876:
--
Description:
h1. Job monitor of submarine
After training, the monitoring program need auto close PS service.
It is possible that other deep learning frameworks also have some custom
processing when the tasks are in different states.
The submarine needs to provide a long-term resident service that monitors each
JOB mission.
This monitoring service can be processed differently according to the training
tasks of different depth learning framework types.
For example: Tensorflow performs distributed training, when the training is
completed,
The PS service cannot be automatically stopped. At this time, the PS needs to
be actively stopped by the monitoring service.
was:
h1. Job monitor of \{submarine}
After training, the monitoring program need auto close PS service.
The submarine needs to provide a long-term resident service that monitors each
JOB mission.
This monitoring service can be processed differently according to the training
tasks of different depth learning framework types.
For example: Tensorflow performs distributed training, when the training is
completed,
The PS service cannot be automatically stopped. At this time, the PS needs to
be actively stopped by the monitoring service.
> [Submarine] Job monitor of submarine
>
>
> Key: YARN-8876
> URL: https://issues.apache.org/jira/browse/YARN-8876
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xun Liu
>Assignee: Xun Liu
>Priority: Major
>
> h1. Job monitor of submarine
> After training, the monitoring program need auto close PS service.
> It is possible that other deep learning frameworks also have some custom
> processing when the tasks are in different states.
> The submarine needs to provide a long-term resident service that monitors
> each JOB mission.
> This monitoring service can be processed differently according to the
> training tasks of different depth learning framework types.
> For example: Tensorflow performs distributed training, when the training is
> completed,
> The PS service cannot be automatically stopped. At this time, the PS needs to
> be actively stopped by the monitoring service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8876) [Submarine] Job monitor of {submarine}
[
https://issues.apache.org/jira/browse/YARN-8876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xun Liu updated YARN-8876:
--
Description:
h1. Job monitor of \{submarine}
After training, the monitoring program need auto close PS service.
The submarine needs to provide a long-term resident service that monitors each
JOB mission.
This monitoring service can be processed differently according to the training
tasks of different depth learning framework types.
For example: Tensorflow performs distributed training, when the training is
completed,
The PS service cannot be automatically stopped. At this time, the PS needs to
be actively stopped by the monitoring service.
was:
h1. Job monitor of {submarine}
The submarine needs to provide a long-term resident service that monitors each
JOB mission.
This monitoring service can be processed differently according to the training
tasks of different depth learning framework types.
For example: Tensorflow performs distributed training, when the training is
completed,
The PS service cannot be automatically stopped. At this time, the PS needs to
be actively stopped by the monitoring service.
> [Submarine] Job monitor of {submarine}
> --
>
> Key: YARN-8876
> URL: https://issues.apache.org/jira/browse/YARN-8876
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xun Liu
>Assignee: Xun Liu
>Priority: Major
>
> h1. Job monitor of \{submarine}
> After training, the monitoring program need auto close PS service.
> The submarine needs to provide a long-term resident service that monitors
> each JOB mission.
> This monitoring service can be processed differently according to the
> training tasks of different depth learning framework types.
> For example: Tensorflow performs distributed training, when the training is
> completed,
> The PS service cannot be automatically stopped. At this time, the PS needs to
> be actively stopped by the monitoring service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8876) [Submarine] Job monitor of {submarine}
[
https://issues.apache.org/jira/browse/YARN-8876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xun Liu updated YARN-8876:
--
Summary: [Submarine] Job monitor of {submarine} (was: [Submarine] After
training, the monitoring program need auto close PS service)
> [Submarine] Job monitor of {submarine}
> --
>
> Key: YARN-8876
> URL: https://issues.apache.org/jira/browse/YARN-8876
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xun Liu
>Assignee: Xun Liu
>Priority: Major
>
> h1. Job monitor of {submarine}
> The submarine needs to provide a long-term resident service that monitors
> each JOB mission.
> This monitoring service can be processed differently according to the
> training tasks of different depth learning framework types.
> For example: Tensorflow performs distributed training, when the training is
> completed,
> The PS service cannot be automatically stopped. At this time, the PS needs to
> be actively stopped by the monitoring service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8876) [Submarine] Job monitor of submarine
[
https://issues.apache.org/jira/browse/YARN-8876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xun Liu updated YARN-8876:
--
Summary: [Submarine] Job monitor of submarine (was: [Submarine] Job
monitor of {submarine})
> [Submarine] Job monitor of submarine
>
>
> Key: YARN-8876
> URL: https://issues.apache.org/jira/browse/YARN-8876
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xun Liu
>Assignee: Xun Liu
>Priority: Major
>
> h1. Job monitor of \{submarine}
> After training, the monitoring program need auto close PS service.
> The submarine needs to provide a long-term resident service that monitors
> each JOB mission.
> This monitoring service can be processed differently according to the
> training tasks of different depth learning framework types.
> For example: Tensorflow performs distributed training, when the training is
> completed,
> The PS service cannot be automatically stopped. At this time, the PS needs to
> be actively stopped by the monitoring service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8876) [Submarine] After training, the monitoring program need auto close PS service
[
https://issues.apache.org/jira/browse/YARN-8876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xun Liu updated YARN-8876:
--
Description:
h1. Job monitor of {submarine}
The submarine needs to provide a long-term resident service that monitors each
JOB mission.
This monitoring service can be processed differently according to the training
tasks of different depth learning framework types.
For example: Tensorflow performs distributed training, when the training is
completed,
The PS service cannot be automatically stopped. At this time, the PS needs to
be actively stopped by the monitoring service.
was:TensorFlow model after training, TensorFlow can not automatically turn
off the PS service, so it needs to be actively shut down by the service status
monitoring program.
> [Submarine] After training, the monitoring program need auto close PS service
> -
>
> Key: YARN-8876
> URL: https://issues.apache.org/jira/browse/YARN-8876
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xun Liu
>Assignee: Xun Liu
>Priority: Major
>
> h1. Job monitor of {submarine}
> The submarine needs to provide a long-term resident service that monitors
> each JOB mission.
> This monitoring service can be processed differently according to the
> training tasks of different depth learning framework types.
> For example: Tensorflow performs distributed training, when the training is
> completed,
> The PS service cannot be automatically stopped. At this time, the PS needs to
> be actively stopped by the monitoring service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8876) [Submarine] After training, the monitoring program need auto close PS service
[ https://issues.apache.org/jira/browse/YARN-8876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xun Liu updated YARN-8876: -- Summary: [Submarine] After training, the monitoring program need auto close PS service (was: [Submarine] After training, the monitoring program need close PS service) > [Submarine] After training, the monitoring program need auto close PS service > - > > Key: YARN-8876 > URL: https://issues.apache.org/jira/browse/YARN-8876 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xun Liu >Assignee: Xun Liu >Priority: Major > > TensorFlow model after training, TensorFlow can not automatically turn off > the PS service, so it needs to be actively shut down by the service status > monitoring program. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
