[jira] [Comment Edited] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16687575#comment-16687575 ] Eric Yang edited comment on YARN-5168 at 11/15/18 7:12 AM: --- [~liuxun323] This jira is to aggregate the information and save in ContainerStatus. YARN service will automatically include this information in yarn app -status call. [~Charo Zhang] is already working on YARN-8986 to add -P to docker run. I assigned this JIRA to you. was (Author: eyang): [~liuxun323] This jira is to aggregate the information and save in ContainerStatus. YARN service will automatically include this information in yarn app -status call. [~Charo Zhang] is already working on YARN-8986 to add -P to docker run. > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Xun Liu >Priority: Major > Labels: Docker > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5168) Add port mapping handling when docker container use bridge network
[
https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16687527#comment-16687527
]
Xun Liu edited comment on YARN-5168 at 11/15/18 7:01 AM:
-
[~eyang]
Thank you for your reply, remove the support of "-p port1:port2", I think so.
{quote}1. Add "-P" to map docker container's exposed ports to automatically.
{quote}
When the yarn starts docker, will this "-P" parameter be added? This way we can
expose the services inside the container, for example:
So we can add it inside the Dockerfile
{code:java}
EXPOSE 1000
EXPOSE 2000
EXPOSE 3000
{code}
After launching docker via YARN, We can get the external port automatically
assigned by the host.
{code:java}
$ docker run d -P -name exposed-ports-in-dockerfile exposed-ports
63264dae9db85c5d667a37dac77e0da7c8d2d699f49b69ba992485242160ad3a
$ docker port exposed-ports-in-dockerfile
1000/tcp -> 0.0.0.0:49156
2000/tcp -> 0.0.0.0:49157
3000/tcp -> 0.0.0.0:49158
{code}
{quote}1. Add "-P" to map docker container's exposed ports to automatically.
{quote}
When will this feature be added? I feel that waiting for hadoop3.3 to release
is a bit late, Can you assign this JIRA to me and let me contribute the patch?
thank you!
was (Author: liuxun323):
[~eyang]
Thank you for your reply, remove the support of "-p port1:port2", I think so.
{quote}1. Add "-P" to map docker container's exposed ports to automatically.
{quote}
When the yarn starts docker, will this "-P" parameter be added? This way we can
expose the services inside the container, for example:
So we can add it inside the Dockerfile
{code:java}
EXPOSE 1000
EXPOSE 2000
EXPOSE 3000
{code}
After launching docker via YARN, We can get the external port automatically
assigned by the host.
{code:java}
$ docker run d -P -name exposed-ports-in-dockerfile exposed-ports
63264dae9db85c5d667a37dac77e0da7c8d2d699f49b69ba992485242160ad3a
$ docker port exposed-ports-in-dockerfile
1000/tcp -> 0.0.0.0:49156
2000/tcp -> 0.0.0.0:49157
3000/tcp -> 0.0.0.0:49158
{code}
{quote}1. Add "-P" to map docker container's exposed ports to automatically.
{quote}
When will this feature be added? I feel that waiting for hadoop3.3 to release
is a bit late, Can you assign this JIRA to me and let me contribute the patch?
thank you!
> Add port mapping handling when docker container use bridge network
> --
>
> Key: YARN-5168
> URL: https://issues.apache.org/jira/browse/YARN-5168
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jun Gong
>Assignee: Eric Yang
>Priority: Major
> Labels: Docker
>
> YARN-4007 addresses different network setups when launching the docker
> container. We need support port mapping when docker container uses bridge
> network.
> The following problems are what we faced:
> 1. Add "-P" to map docker container's exposed ports to automatically.
> 2. Add "-p" to let user specify specific ports to map.
> 3. Add service registry support for bridge network case, then app could find
> each other. It could be done out of YARN, however it might be more convenient
> to support it natively in YARN.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5168) Add port mapping handling when docker container use bridge network
[
https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16687527#comment-16687527
]
Xun Liu edited comment on YARN-5168 at 11/15/18 6:11 AM:
-
[~eyang]
Thank you for your reply, remove the support of "-p port1:port2", I think so.
{quote}1. Add "-P" to map docker container's exposed ports to automatically.
{quote}
When the yarn starts docker, will this "-P" parameter be added? This way we can
expose the services inside the container, for example:
So we can add it inside the Dockerfile
{code:java}
EXPOSE 1000
EXPOSE 2000
EXPOSE 3000
{code}
After launching docker via YARN, We can get the external port automatically
assigned by the host.
{code:java}
$ docker run d -P -name exposed-ports-in-dockerfile exposed-ports
63264dae9db85c5d667a37dac77e0da7c8d2d699f49b69ba992485242160ad3a
$ docker port exposed-ports-in-dockerfile
1000/tcp -> 0.0.0.0:49156
2000/tcp -> 0.0.0.0:49157
3000/tcp -> 0.0.0.0:49158
{code}
{quote}1. Add "-P" to map docker container's exposed ports to automatically.
{quote}
When will this feature be added? I feel that waiting for hadoop3.3 to release
is a bit late, Can you assign this JIRA to me and let me contribute the patch?
thank you!
was (Author: liuxun323):
[~eyang]
Thank you for your reply, remove the support of "-p port1:port2", I think so.
{quote}1. Add "-P" to map docker container's exposed ports to automatically.
{quote}
When the yarn starts docker, will this "-P" parameter be added? This way we can
expose the services inside the container, for example:
So we can add it inside the Dockerfile
EXPOSE 1000
EXPOSE 2000
EXPOSE 3000
After launching docker via YARN, We can get the external port automatically
assigned by the host.
$ docker run -d -P --name exposed-ports-in-dockerfile exposed-ports
63264dae9db85c5d667a37dac77e0da7c8d2d699f49b69ba992485242160ad3a
$ docker port exposed-ports-in-dockerfile
1000/tcp -> 0.0.0.0:49156
2000/tcp -> 0.0.0.0:49157
3000/tcp -> 0.0.0.0:49158
{quote}1. Add "-P" to map docker container's exposed ports to automatically.
{quote}
When will this feature be added? I feel that waiting for hadoop3.3 to release
is a bit late, Can you assign this JIRA to me and let me contribute the patch?
thank you!
> Add port mapping handling when docker container use bridge network
> --
>
> Key: YARN-5168
> URL: https://issues.apache.org/jira/browse/YARN-5168
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jun Gong
>Assignee: Eric Yang
>Priority: Major
> Labels: Docker
>
> YARN-4007 addresses different network setups when launching the docker
> container. We need support port mapping when docker container uses bridge
> network.
> The following problems are what we faced:
> 1. Add "-P" to map docker container's exposed ports to automatically.
> 2. Add "-p" to let user specify specific ports to map.
> 3. Add service registry support for bridge network case, then app could find
> each other. It could be done out of YARN, however it might be more convenient
> to support it natively in YARN.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5168) Add port mapping handling when docker container use bridge network
[
https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685611#comment-16685611
]
Eric Yang edited comment on YARN-5168 at 11/13/18 8:39 PM:
---
The current plan is to expose this information through YARN service because
YARN service AM can aggregate container information.
{code}
{
"components": [
{
"artifact": {
"id": "hadoop/centos:latest",
"type": "DOCKER"
},
"configuration": {
"env": {
},
"files": [
],
"properties": {
"docker.network": "bridge,hadoop"
}
},
"containers": [
{
"bare_host": "eyang-5.example.com",
"component_instance_name": "ping-2",
"hostname": "ping-2.abc.hbase.ycluster",
"id": "container_1542058458438_0002_01_23",
"ip": "172.17.0.20",
"launch_time": 1542133678396,
"state": "READY",
"exposed_ports": {
"8080/tcp": [ "0.0.0.0:32769" ],
"8443/tcp": [ "0.0.0.0:32768" ]
},
},
{
"bare_host": "eyang-5.example.com",
"component_instance_name": "ping-0",
"hostname": "ping-0.abc.hbase.ycluster",
"id": "container_1542058458438_0002_01_24",
"ip": "172.17.0.20",
"launch_time": 1542133737566,
"state": "READY"
}
],
"decommissioned_instances": [
"ping-1"
],
"dependencies": [],
"launch_command": "sleep,1",
"name": "ping",
"number_of_containers": 2,
"quicklinks": [],
"resource": {
"additional": {},
"cpus": 1,
"memory": "256"
},
"restart_policy": "ALWAYS",
"run_privileged_container": true,
"state": "STABLE"
}
],
"configuration": {
"env": {},
"files": [],
"properties": {}
},
"dependencies": [],
"id": "application_1542058458438_0002",
"kerberos_principal": {
"keytab": "file:///etc/security/keytabs/hbase.service.keytab",
"principal_name": "hbase/_h...@example.com"
},
"lifetime": -1,
"name": "abc",
"quicklinks": {},
"state": "STABLE",
"version": "5"
}
{code}
was (Author: eyang):
The current plan is to expose this information through YARN service because
YARN service AM can aggregate container information.
{code}
{
"components": [
{
"artifact": {
"id": "hadoop/centos:latest",
"type": "DOCKER"
},
"configuration": {
"env": {
},
"files": [
],
"properties": {
"docker.network": "bridge,hadoop"
}
},
"containers": [
{
"bare_host": "eyang-5.example.com",
"component_instance_name": "ping-2",
"hostname": "ping-2.abc.hbase.ycluster",
"id": "container_1542058458438_0002_01_23",
"ip": "172.17.0.20",
"launch_time": 1542133678396,
"state": "READY",
"exposed_ports": {
"8080/tcp": "0.0.0.0:32769",
"8443/tcp": "0.0.0.0:32768"
},
},
{
"bare_host": "eyang-5.example.com",
"component_instance_name": "ping-0",
"hostname": "ping-0.abc.hbase.ycluster",
"id": "container_1542058458438_0002_01_24",
"ip": "172.17.0.20",
"launch_time": 1542133737566,
"state": "READY"
}
],
"decommissioned_instances": [
"ping-1"
],
"dependencies": [],
"launch_command": "sleep,1",
"name": "ping",
"number_of_containers": 2,
"quicklinks": [],
"resource": {
"additional": {},
"cpus": 1,
"memory": "256"
},
"restart_policy": "ALWAYS",
"run_privileged_container": true,
"state": "STABLE"
}
],
"configuration": {
"env": {},
"files": [],
"properties": {}
},
[jira] [Comment Edited] (YARN-5168) Add port mapping handling when docker container use bridge network
[
https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685611#comment-16685611
]
Eric Yang edited comment on YARN-5168 at 11/13/18 8:33 PM:
---
The current plan is to expose this information through YARN service because
YARN service AM can aggregate container information.
{code}
{
"components": [
{
"artifact": {
"id": "hadoop/centos:latest",
"type": "DOCKER"
},
"configuration": {
"env": {
},
"files": [
],
"properties": {
"docker.network": "bridge,hadoop"
}
},
"containers": [
{
"bare_host": "eyang-5.example.com",
"component_instance_name": "ping-2",
"hostname": "ping-2.abc.hbase.ycluster",
"id": "container_1542058458438_0002_01_23",
"ip": "172.17.0.20",
"launch_time": 1542133678396,
"state": "READY",
"exposed_ports": {
"8080/tcp": "0.0.0.0:32769",
"8443/tcp": "0.0.0.0:32768"
},
},
{
"bare_host": "eyang-5.example.com",
"component_instance_name": "ping-0",
"hostname": "ping-0.abc.hbase.ycluster",
"id": "container_1542058458438_0002_01_24",
"ip": "172.17.0.20",
"launch_time": 1542133737566,
"state": "READY"
}
],
"decommissioned_instances": [
"ping-1"
],
"dependencies": [],
"launch_command": "sleep,1",
"name": "ping",
"number_of_containers": 2,
"quicklinks": [],
"resource": {
"additional": {},
"cpus": 1,
"memory": "256"
},
"restart_policy": "ALWAYS",
"run_privileged_container": true,
"state": "STABLE"
}
],
"configuration": {
"env": {},
"files": [],
"properties": {}
},
"dependencies": [],
"id": "application_1542058458438_0002",
"kerberos_principal": {
"keytab": "file:///etc/security/keytabs/hbase.service.keytab",
"principal_name": "hbase/_h...@example.com"
},
"lifetime": -1,
"name": "abc",
"quicklinks": {},
"state": "STABLE",
"version": "5"
}
{code}
was (Author: eyang):
The current plan is to expose this information through YARN service because
YARN service AM can aggregate container information.
{code}
{
"components": [
{
"artifact": {
"id": "hadoop/centos:latest",
"type": "DOCKER"
},
"configuration": {
"env": {
},
"files": [
],
"properties": {
"docker.network": "bridge,hadoop"
}
},
"containers": [
{
"bare_host": "eyang-5.example.com",
"component_instance_name": "ping-2",
"hostname": "ping-2.abc.hbase.ycluster",
"id": "container_1542058458438_0002_01_23",
"ip": "172.17.0.20",
"launch_time": 1542133678396,
"state": "READY",
"exposed_ports": [
"8080/tcp",
"8443/tcp"
],
},
{
"bare_host": "eyang-5.example.com",
"component_instance_name": "ping-0",
"hostname": "ping-0.abc.hbase.ycluster",
"id": "container_1542058458438_0002_01_24",
"ip": "172.17.0.20",
"launch_time": 1542133737566,
"state": "READY"
}
],
"decommissioned_instances": [
"ping-1"
],
"dependencies": [],
"launch_command": "sleep,1",
"name": "ping",
"number_of_containers": 2,
"quicklinks": [],
"resource": {
"additional": {},
"cpus": 1,
"memory": "256"
},
"restart_policy": "ALWAYS",
"run_privileged_container": true,
"state": "STABLE"
}
],
"configuration": {
"env": {},
"files": [],
"properties": {}
},
"dependencies": [],
"id":
[jira] [Comment Edited] (YARN-5168) Add port mapping handling when docker container use bridge network
[ https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685660#comment-16685660 ] Shane Kumpf edited comment on YARN-5168 at 11/13/18 7:46 PM: - I believe supporting -P will lead to port conflicts. What if two containers running on the same NM both expose 8080? An alternative might be to support the "-p 8080" syntax (NOT "-p 8080:8080", only the single port). With this syntax, docker will allocate an ephemeral port on the NM and forward traffic to that port to 8080 in the container (0.0.0.0:32768 -> container_ip:8080). This allows docker to manage the ports and avoid conflict. The user would need to supply the list of ports they want to expose at job submission time, likely via an env variable like we do with networks. was (Author: shaneku...@gmail.com): I believe supporting -P will lead to port conflicts. What if two containers running on the same NM both expose 8080? An alternative might be to support the "-p 8080" syntax (NOT "-p 8080:8080", only the single port). With this syntax, docker will allocate an ephemeral port on the NM and forward traffic to that port to 8080 in the container (0.0.0.0:32768 -> container_ip:8080). The user would need to supply the list of ports they want to expose at job submission time, likely via an env variable like we do with networks. > Add port mapping handling when docker container use bridge network > -- > > Key: YARN-5168 > URL: https://issues.apache.org/jira/browse/YARN-5168 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jun Gong >Assignee: Eric Yang >Priority: Major > Labels: Docker > > YARN-4007 addresses different network setups when launching the docker > container. We need support port mapping when docker container uses bridge > network. > The following problems are what we faced: > 1. Add "-P" to map docker container's exposed ports to automatically. > 2. Add "-p" to let user specify specific ports to map. > 3. Add service registry support for bridge network case, then app could find > each other. It could be done out of YARN, however it might be more convenient > to support it natively in YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5168) Add port mapping handling when docker container use bridge network
[
https://issues.apache.org/jira/browse/YARN-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685611#comment-16685611
]
Eric Yang edited comment on YARN-5168 at 11/13/18 7:33 PM:
---
The current plan is to expose this information through YARN service because
YARN service AM can aggregate container information.
{code}
{
"components": [
{
"artifact": {
"id": "hadoop/centos:latest",
"type": "DOCKER"
},
"configuration": {
"env": {
},
"files": [
],
"properties": {
"docker.network": "bridge,hadoop"
}
},
"containers": [
{
"bare_host": "eyang-5.example.com",
"component_instance_name": "ping-2",
"hostname": "ping-2.abc.hbase.ycluster",
"id": "container_1542058458438_0002_01_23",
"ip": "172.17.0.20",
"launch_time": 1542133678396,
"state": "READY",
"exposed_ports": [
"8080/tcp",
"8443/tcp"
],
},
{
"bare_host": "eyang-5.example.com",
"component_instance_name": "ping-0",
"hostname": "ping-0.abc.hbase.ycluster",
"id": "container_1542058458438_0002_01_24",
"ip": "172.17.0.20",
"launch_time": 1542133737566,
"state": "READY"
}
],
"decommissioned_instances": [
"ping-1"
],
"dependencies": [],
"launch_command": "sleep,1",
"name": "ping",
"number_of_containers": 2,
"quicklinks": [],
"resource": {
"additional": {},
"cpus": 1,
"memory": "256"
},
"restart_policy": "ALWAYS",
"run_privileged_container": true,
"state": "STABLE"
}
],
"configuration": {
"env": {},
"files": [],
"properties": {}
},
"dependencies": [],
"id": "application_1542058458438_0002",
"kerberos_principal": {
"keytab": "file:///etc/security/keytabs/hbase.service.keytab",
"principal_name": "hbase/_h...@example.com"
},
"lifetime": -1,
"name": "abc",
"quicklinks": {},
"state": "STABLE",
"version": "5"
}
{code}
was (Author: eyang):
The current plan is to expose this information through YARN service because
YARN service AM can aggregate container information.
{code}
{
"components": [
{
"artifact": {
"id": "hadoop/centos:latest",
"type": "DOCKER"
},
"configuration": {
"env": {
},
"files": [
],
"properties": {
"docker.network": "bridge,hadoop"
}
},
"containers": [
{
"bare_host": "eyang-5.example.com",
"component_instance_name": "ping-2",
"hostname": "ping-2.abc.hbase.ycluster",
"id": "container_1542058458438_0002_01_23",
"ip": "172.17.0.20",
"launch_time": 1542133678396,
"state": "READY",
"ports": {
"8080/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "8080"
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "8443"
}
]
},
},
{
"bare_host": "eyang-5.example.com",
"component_instance_name": "ping-0",
"hostname": "ping-0.abc.hbase.ycluster",
"id": "container_1542058458438_0002_01_24",
"ip": "172.17.0.20",
"launch_time": 1542133737566,
"state": "READY"
}
],
"decommissioned_instances": [
"ping-1"
],
"dependencies": [],
"launch_command": "sleep,1",
"name": "ping",
"number_of_containers": 2,
"quicklinks": [],
"resource": {
"additional": {},
