[
https://issues.apache.org/jira/browse/MESOS-9031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16534372#comment-16534372
]
Qian Zhang edited comment on MESOS-9031 at 7/6/18 2:21 AM:
-----------------------------------------------------------
After more investigations, I found there are 4 issues in total, and the root
cause of all these 4 issues is some iptables rules are missed which causes
packets are dropped by the default policy of the FORWARD chain. Here is the
summary:
*Case 1:* Two containers join the `mesos-cni0` bridge network, the first
container listens and serves on its 80 port and map it to host's 8080 port, and
the second container tries to access hostIP:8080.
We need this rule `+iptables -t filter -A FORWARD -o mesos-cni0 -j ACCEPT`+ for
this case, otherwise the packets sent from the second container will be dropped
by the default policy of the FORWARD chain, and once this rule is added, the
packets will be accepted and reaches to the first container successfully.
*Case 2:* One container joins the `mesos-cni0` bridge network, and it tries to
access the world outside the host on which it runs (e.g., it executes the
command `curl www.myip.ch`).
We need this rule `+iptables -t filter -A FORWARD -i mesos-cni0 ! -o mesos-cni0
-j ACCEPT+` for this case, otherwise the packets sent from the container will
be dropped by the default policy of the FORWARD chain, and once this rule is
added, the packets will be accepted and sent out of the host successfully.
*Case 3:* One container joins the `mesos-cni0` bridge network and listens and
serves on its 80 port and map it to host's 8080 port, we try to access
hostIP:8080 from another host.
We need both of two rules mentioned in case 1 & 2, because without rule in case
1 the packets sent from the other host will be dropped by the default policy of
the FORWARD chain, and without rule in case 2 the reply packets sent by the
container will also be dropped by the default policy of the FORWARD chain.
*Case 4:* Two containers join the `mesos-cni0` bridge network, the first
container listens and serves on its 80 port, and the second container tries to
access the first container's IP:80.
We need this rule `+iptables -t filter -A FORWARD -i mesos-cni0 -o mesos-cni0
-j ACCEPT+` for this case, otherwise the packets sent from the second container
will be dropped by the default policy of the FORWARD chain, and once this rule
is added, the packets will be accepted and reaches to the first container
successfully.
Please note that we only have all the above issues when the default policy of
the FORWARD chain is DROP, if it is ACCEPT, then there is no any issues.
So to resolve the above issues, we need to add the 3 rules mentioned in the
above cases into the FORWARD chain, it is actually how Docker handles these
issues as well, you will find the following rules in the host which has Docker
installed.
{code:java}
$ sudo iptables -t filter -nvL
...
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
...
0 0 ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/0{code}
was (Author: qianzhang):
After more investigations, I found there are 4 issues in total, and the root
cause of all these 4 issues is some iptables rules are missed which causes
packets are dropped by the default policy of the FORWARD chain. Here is the
summary:
*Case 1:* Two containers join the `mesos-cni0` bridge network, the first
container listens and serves on its 80 port and map it to host's 8080 port, and
the second container tries to access hostIP:8080.
We need this rule `+iptables -t filter -A FORWARD -o mesos-cni0 -j ACCEPT`+ for
this case, otherwise the packets sent from the second container will be dropped
by the default policy of the FORWARD chain, and once this rule is added, the
packets will be accepted and reaches to the first container successfully.
*Case 2:* One container joins the `mesos-cni0` bridge network, and it tries to
access the world outside the host on which it runs (e.g., it executes the
command `curl www.myip.ch`).
We need this rule `+iptables -t filter -A FORWARD -i mesos-cni0 ! -o mesos-cni0
-j ACCEPT+` for this case, otherwise the packets sent from the container will
be dropped by the default policy of the FORWARD chain, and once this rule is
added, the packets will be accepted and sent out of the host successfully.
*Case 3:* One container joins the `mesos-cni0` bridge network and listens and
serves on its 80 port and map it to host's 8080 port, we try to access
hostIP:8080 from another host.
We need both of two rules mentioned in case 1 & 2, because without rule in case
1 the packets sent from the other host will be dropped by the default policy of
the FORWARD chain, and without rule in case 2 the reply packets sent by the
container will also be dropped by the default policy of the FORWARD chain.
*Case 4:* Two containers join the `mesos-cni0` bridge network, the first
container listens and serves on its 80 port, and the second container tries to
access the first container's IP:80.
We need this rule `+iptables -t filter -A FORWARD -i mesos-cni0 -o mesos-cni0
-j ACCEPT+` for this case, otherwise the packets sent from the second container
will be dropped by the default policy of the FORWARD chain, and once this rule
is added, the packets will be accepted and reaches to the first container
successfully.
Please note that we only have all the above issues when the default policy of
the FORWARD chain is DROP, if it is ACCEPT, then there is no any issues.
> Mesos CNI portmap plugins' iptables rules doesn't allow connections via host
> ip and port from the same bridge container network
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: MESOS-9031
> URL: https://issues.apache.org/jira/browse/MESOS-9031
> Project: Mesos
> Issue Type: Bug
> Components: cni, containerization
> Affects Versions: 1.6.0
> Reporter: Kirill Plyashkevich
> Assignee: Qian Zhang
> Priority: Major
>
> using `mesos-cni-port-mapper` with folllowing config:
> {noformat}
> {
> "name" : "dcos",
> "type" : "mesos-cni-port-mapper",
> "excludeDevices" : [],
> "chain": "MESOS-CNI0-PORT-MAPPER",
> "delegate": {
> "type": "bridge",
> "bridge": "mesos-cni0",
> "isGateway": true,
> "ipMasq": true,
> "hairpinMode": true,
> "ipam": {
> "type": "host-local",
> "ranges": [
> [{"subnet": "172.26.0.0/16"}]
> ],
> "routes": [
> {"dst": "0.0.0.0/0"}
> ]
> }
> }
> }
> {noformat}
> - 2 services running on the same mesos-slave using unified containerizer in
> different tasks and communicating via host ip and host port
> - connection timeouts due to iptables rules per container CNI-XXX chain
> - actually timeouts are caused by
> {noformat}
> Chain CNI-XXX (1 references)
> num target prot opt source destination
> 1 ACCEPT all -- anywhere 172.26.0.0/16 /* name:
> "dcos" id: "YYYY" */
> 2 MASQUERADE all -- anywhere !base-address.mcast.net/4 /*
> name: "dcos" id: "YYYY" */
> {noformat}
> rule #1 is executed and no masquerading happens.
> there are multiple solutions:
> - -simpliest and fastest one is not to add that ACCEPT- - NOT A SOLUTION.
> it's happening in `bridge` plugin and `cni/portmap` shows that
> snat/masquerade should be done during portmapping as well.
> - perhaps, there's a better change in iptables rules that can fix it
> - proper one (imho) is to finally implement cni spec 0.3.x in order to be
> able to use chaining of plugins and use cni's `bridge` and `portmap` plugins
> in chain (and get rid of mesos-cni-port-mapper completely eventually).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
