[
https://issues.apache.org/jira/browse/MESOS-9031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16536133#comment-16536133
]
Qian Zhang edited comment on MESOS-9031 at 7/9/18 8:06 AM:
-----------------------------------------------------------
[~Kirill P] I tried to reproduce this issue with `mesos-execute`, but no luck,
Here are the detailed steps.
1. Run Mesos master and Mesos agent on a single host (192.168.56.5), I am using
the CNI network `port-mapper-test`:
{code:java}
{
"name" : "port-mapper-test",
"type" : "mesos-cni-port-mapper",
"excludeDevices" : [],
"chain": "MESOS-TEST-PORT-MAPPER",
"delegate": {
"type": "bridge",
"bridge": "mesos-cni0",
"isGateway": true,
"ipMasq": true,
"hairpinMode": true,
"ipam": {
"type": "host-local",
"subnet": "192.168.3.0/24",
"routes": [
{ "dst": "0.0.0.0/0" }
]
}
}
}
{code}
2. Run `mesos-execute --master=192.168.56.5:5050
--task=[file:///home/stack/workspace/config/task_cni_port.json]` to launch an
Nginx container (192.168.3.61) and map 192.168.56.5:8080 to 192.168.3.61:80.
{code:java}
// task_cni_port.json
{
"name": "test1",
"task_id": {"value" : "test1"},
"agent_id": {"value" : ""},
"resources": [
{"name": "cpus", "type": "SCALAR", "scalar": {"value": 0.1}},
{"name": "mem", "type": "SCALAR", "scalar": {"value": 32}}
],
"command": {
"shell": false,
"user": "root"
},
"container": {
"type": "MESOS",
"mesos": {
"image": {
"type": "DOCKER",
"docker": {
"name": "nginx:alpine"
}
}
},
"network_infos": [
{
"name": "port-mapper-test",
"port_mappings" : [
{
"host_port": 8080,
"container_port": 80
}
]
}
]
}
}
{code}
At this moment, the rules in the `nat` table are:
{code:java}
$ sudo iptables -t nat -nvL
Chain PREROUTING (policy ACCEPT 1 packets, 60 bytes)
pkts bytes target prot opt in out source destination
13 636 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0
ADDRTYPE match dst-type LOCAL
4 240 MESOS-TEST-PORT-MAPPER all -- * * 0.0.0.0/0
0.0.0.0/0 ADDRTYPE match dst-type LOCAL
...
Chain POSTROUTING (policy ACCEPT 1 packets, 57 bytes)
pkts bytes target prot opt in out source destination
0 0 MASQUERADE all -- * !docker0 172.17.0.0/16
0.0.0.0/0
3 190 CNI-f66b0ad1eddbe45370f979cd all -- * *
192.168.3.0/24 0.0.0.0/0 /* name: "port-mapper-test" id:
"b2831648-123e-47f5-b46f-b03414dd45c5" */
Chain CNI-f66b0ad1eddbe45370f979cd (1 references)
pkts bytes target prot opt in out source destination
2 133 ACCEPT all -- * * 0.0.0.0/0
192.168.3.0/24 /* name: "port-mapper-test" id:
"b2831648-123e-47f5-b46f-b03414dd45c5" */
0 0 MASQUERADE all -- * * 0.0.0.0/0
!224.0.0.0/4 /* name: "port-mapper-test" id:
"b2831648-123e-47f5-b46f-b03414dd45c5" */
Chain MESOS-TEST-PORT-MAPPER (2 references)
pkts bytes target prot opt in out source destination
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0
tcp dpt:8080 /* container_id: b2831648-123e-47f5-b46f-b03414dd45c5 */
to:192.168.3.61:80
{code}
3. Run `mesos-execute --master=192.168.56.5:5050
--task=[file:///home/stack/workspace/config/task_cni_port1.json|file:///home/stack/workspace/config/task_cni_port.json]`
to launch another container (192.168.3.62) and map 192.168.56.5:8081 to
192.168.3.62:80, but this container will not actually listen on its 80 port, it
will try to access the first container via `192.168.56.5:8080`.
{code:java}
// task_cni_port1.json
{
"name": "test1",
"task_id": {"value" : "test1"},
"agent_id": {"value" : ""},
"resources": [
{"name": "cpus", "type": "SCALAR", "scalar": {"value": 0.1}},
{"name": "mem", "type": "SCALAR", "scalar": {"value": 32}}
],
"command": {
"value": "curl 192.168.56.5:8080"
},
"container": {
"type": "MESOS",
"network_infos": [
{
"name": "port-mapper-test",
"port_mappings" : [
{
"host_port": 8081,
"container_port": 80
}
]
}
]
}
}
{code}
At this moment, the rules in the `nat` table are:
{code:java}
$ sudo iptables -t nat -nvL
Chain PREROUTING (policy ACCEPT 1 packets, 60 bytes)
pkts bytes target prot opt in out source destination
15 756 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0
ADDRTYPE match dst-type LOCAL
6 360 MESOS-TEST-PORT-MAPPER all -- * * 0.0.0.0/0
0.0.0.0/0 ADDRTYPE match dst-type LOCAL
...
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 MASQUERADE all -- * !docker0 172.17.0.0/16
0.0.0.0/0
7 440 CNI-f66b0ad1eddbe45370f979cd all -- * *
192.168.3.0/24 0.0.0.0/0 /* name: "port-mapper-test" id:
"b2831648-123e-47f5-b46f-b03414dd45c5" */
0 0 CNI-2f289d81da29f88d7e4dd23a all -- * *
192.168.3.0/24 0.0.0.0/0 /* name: "port-mapper-test" id:
"aa2230e6-43d0-465b-b5c7-c785abe5ecb4" */
Chain CNI-2f289d81da29f88d7e4dd23a (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0
192.168.3.0/24 /* name: "port-mapper-test" id:
"aa2230e6-43d0-465b-b5c7-c785abe5ecb4" */
0 0 MASQUERADE all -- * * 0.0.0.0/0
!224.0.0.0/4 /* name: "port-mapper-test" id:
"aa2230e6-43d0-465b-b5c7-c785abe5ecb4" */
Chain CNI-f66b0ad1eddbe45370f979cd (1 references)
pkts bytes target prot opt in out source destination
5 326 ACCEPT all -- * * 0.0.0.0/0
192.168.3.0/24 /* name: "port-mapper-test" id:
"b2831648-123e-47f5-b46f-b03414dd45c5" */
0 0 MASQUERADE all -- * * 0.0.0.0/0
!224.0.0.0/4 /* name: "port-mapper-test" id:
"b2831648-123e-47f5-b46f-b03414dd45c5" */
Chain MESOS-TEST-PORT-MAPPER (2 references)
pkts bytes target prot opt in out source destination
1 60 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0
tcp dpt:8080 /* container_id: b2831648-123e-47f5-b46f-b03414dd45c5 */
to:192.168.3.61:80
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0
tcp dpt:8081 /* container_id: aa2230e6-43d0-465b-b5c7-c785abe5ecb4 */
to:192.168.3.62:80
{code}
Based on my test, the second container can successfully access the first
container via `192.168.56.5:8080`. Maybe something missed in my test? And from
the first container's log, I can see the request IP is the second container's
IP (192.168.3.62) rather than the bridge's IP (192.168.3.1), so it seems
MASQUERADE is not needed in this case.
was (Author: qianzhang):
[~Kirill P] I tried to reproduce this issue with `mesos-execute`, but no luck,
Here are the detailed steps.
1. Run Mesos master and Mesos agent on a single host (192.168.56.5), I am using
the CNI network `port-mapper-test`:
{code:java}
{
"name" : "port-mapper-test",
"type" : "mesos-cni-port-mapper",
"excludeDevices" : [],
"chain": "MESOS-TEST-PORT-MAPPER",
"delegate": {
"type": "bridge",
"bridge": "mesos-cni0",
"isGateway": true,
"ipMasq": true,
"hairpinMode": true,
"ipam": {
"type": "host-local",
"subnet": "192.168.3.0/24",
"routes": [
{ "dst": "0.0.0.0/0" }
]
}
}
}
{code}
2. Run `mesos-execute --master=192.168.56.5:5050
--task=[file:///home/stack/workspace/config/task_cni_port.json]` to launch an
Nginx container (192.168.3.61) and map 192.168.56.5:8080 to 192.168.3.61:80.
{code:java}
// task_cni_port.json
{
"name": "test1",
"task_id": {"value" : "test1"},
"agent_id": {"value" : ""},
"resources": [
{"name": "cpus", "type": "SCALAR", "scalar": {"value": 0.1}},
{"name": "mem", "type": "SCALAR", "scalar": {"value": 32}}
],
"command": {
"shell": false,
"user": "root"
},
"container": {
"type": "MESOS",
"mesos": {
"image": {
"type": "DOCKER",
"docker": {
"name": "nginx:alpine"
}
}
},
"network_infos": [
{
"name": "port-mapper-test",
"port_mappings" : [
{
"host_port": 8080,
"container_port": 80
}
]
}
]
}
}
{code}
At this moment, the rules in the `nat` table are:
{code:java}
$ sudo iptables -t nat -nvL
Chain PREROUTING (policy ACCEPT 1 packets, 60 bytes)
pkts bytes target prot opt in out source destination
13 636 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0
ADDRTYPE match dst-type LOCAL
4 240 MESOS-TEST-PORT-MAPPER all -- * * 0.0.0.0/0
0.0.0.0/0 ADDRTYPE match dst-type LOCAL
...
Chain POSTROUTING (policy ACCEPT 1 packets, 57 bytes)
pkts bytes target prot opt in out source destination
0 0 MASQUERADE all -- * !docker0 172.17.0.0/16
0.0.0.0/0
3 190 CNI-f66b0ad1eddbe45370f979cd all -- * *
192.168.3.0/24 0.0.0.0/0 /* name: "port-mapper-test" id:
"b2831648-123e-47f5-b46f-b03414dd45c5" */
Chain CNI-f66b0ad1eddbe45370f979cd (1 references)
pkts bytes target prot opt in out source destination
2 133 ACCEPT all -- * * 0.0.0.0/0
192.168.3.0/24 /* name: "port-mapper-test" id:
"b2831648-123e-47f5-b46f-b03414dd45c5" */
0 0 MASQUERADE all -- * * 0.0.0.0/0
!224.0.0.0/4 /* name: "port-mapper-test" id:
"b2831648-123e-47f5-b46f-b03414dd45c5" */
Chain MESOS-TEST-PORT-MAPPER (2 references)
pkts bytes target prot opt in out source destination
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0
tcp dpt:8080 /* container_id: b2831648-123e-47f5-b46f-b03414dd45c5 */
to:192.168.3.61:80
{code}
3. Run `mesos-execute --master=192.168.56.5:5050
--task=[file:///home/stack/workspace/config/task_cni_port1.json|file:///home/stack/workspace/config/task_cni_port.json]`
to launch another container (192.168.3.62) and map 192.168.56.5:8081 to
192.168.3.62:80, but this container will not actually listen on its 80 port, it
will try to access the first container via `192.168.56.5:8080`.
{code:java}
// task_cni_port1.json
{
"name": "test1",
"task_id": {"value" : "test1"},
"agent_id": {"value" : ""},
"resources": [
{"name": "cpus", "type": "SCALAR", "scalar": {"value": 0.1}},
{"name": "mem", "type": "SCALAR", "scalar": {"value": 32}}
],
"command": {
"value": "curl 192.168.56.5:8080"
},
"container": {
"type": "MESOS",
"network_infos": [
{
"name": "port-mapper-test",
"port_mappings" : [
{
"host_port": 8081,
"container_port": 80
}
]
}
]
}
}
{code}
At this moment, the rules in the `nat` table are:
{code:java}
$ sudo iptables -t nat -nvL
Chain PREROUTING (policy ACCEPT 1 packets, 60 bytes)
pkts bytes target prot opt in out source destination
15 756 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0
ADDRTYPE match dst-type LOCAL
6 360 MESOS-TEST-PORT-MAPPER all -- * * 0.0.0.0/0
0.0.0.0/0 ADDRTYPE match dst-type LOCAL
...
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 MASQUERADE all -- * !docker0 172.17.0.0/16
0.0.0.0/0
7 440 CNI-f66b0ad1eddbe45370f979cd all -- * *
192.168.3.0/24 0.0.0.0/0 /* name: "port-mapper-test" id:
"b2831648-123e-47f5-b46f-b03414dd45c5" */
0 0 CNI-2f289d81da29f88d7e4dd23a all -- * *
192.168.3.0/24 0.0.0.0/0 /* name: "port-mapper-test" id:
"aa2230e6-43d0-465b-b5c7-c785abe5ecb4" */
Chain CNI-2f289d81da29f88d7e4dd23a (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0
192.168.3.0/24 /* name: "port-mapper-test" id:
"aa2230e6-43d0-465b-b5c7-c785abe5ecb4" */
0 0 MASQUERADE all -- * * 0.0.0.0/0
!224.0.0.0/4 /* name: "port-mapper-test" id:
"aa2230e6-43d0-465b-b5c7-c785abe5ecb4" */
Chain CNI-f66b0ad1eddbe45370f979cd (1 references)
pkts bytes target prot opt in out source destination
5 326 ACCEPT all -- * * 0.0.0.0/0
192.168.3.0/24 /* name: "port-mapper-test" id:
"b2831648-123e-47f5-b46f-b03414dd45c5" */
0 0 MASQUERADE all -- * * 0.0.0.0/0
!224.0.0.0/4 /* name: "port-mapper-test" id:
"b2831648-123e-47f5-b46f-b03414dd45c5" */
Chain MESOS-TEST-PORT-MAPPER (2 references)
pkts bytes target prot opt in out source destination
1 60 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0
tcp dpt:8080 /* container_id: b2831648-123e-47f5-b46f-b03414dd45c5 */
to:192.168.3.61:80
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0
tcp dpt:8081 /* container_id: aa2230e6-43d0-465b-b5c7-c785abe5ecb4 */
to:192.168.3.62:80
{code}
Based on my test, the second container can successfully access the first
container via `192.168.56.5:8080`. Maybe something missed in my test?
> Mesos CNI portmap plugins' iptables rules doesn't allow connections via host
> ip and port from the same bridge container network
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: MESOS-9031
> URL: https://issues.apache.org/jira/browse/MESOS-9031
> Project: Mesos
> Issue Type: Bug
> Components: cni, containerization
> Affects Versions: 1.6.0
> Reporter: Kirill Plyashkevich
> Assignee: Qian Zhang
> Priority: Major
>
> using `mesos-cni-port-mapper` with folllowing config:
> {noformat}
> {
> "name" : "dcos",
> "type" : "mesos-cni-port-mapper",
> "excludeDevices" : [],
> "chain": "MESOS-CNI0-PORT-MAPPER",
> "delegate": {
> "type": "bridge",
> "bridge": "mesos-cni0",
> "isGateway": true,
> "ipMasq": true,
> "hairpinMode": true,
> "ipam": {
> "type": "host-local",
> "ranges": [
> [{"subnet": "172.26.0.0/16"}]
> ],
> "routes": [
> {"dst": "0.0.0.0/0"}
> ]
> }
> }
> }
> {noformat}
> - 2 services running on the same mesos-slave using unified containerizer in
> different tasks and communicating via host ip and host port
> - connection timeouts due to iptables rules per container CNI-XXX chain
> - actually timeouts are caused by
> {noformat}
> Chain CNI-XXX (1 references)
> num target prot opt source destination
> 1 ACCEPT all -- anywhere 172.26.0.0/16 /* name:
> "dcos" id: "YYYY" */
> 2 MASQUERADE all -- anywhere !base-address.mcast.net/4 /*
> name: "dcos" id: "YYYY" */
> {noformat}
> rule #1 is executed and no masquerading happens.
> there are multiple solutions:
> - -simpliest and fastest one is not to add that ACCEPT- - NOT A SOLUTION.
> it's happening in `bridge` plugin and `cni/portmap` shows that
> snat/masquerade should be done during portmapping as well.
> - perhaps, there's a better change in iptables rules that can fix it
> - proper one (imho) is to finally implement cni spec 0.3.x in order to be
> able to use chaining of plugins and use cni's `bridge` and `portmap` plugins
> in chain (and get rid of mesos-cni-port-mapper completely eventually).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
