[
https://issues.apache.org/jira/browse/MESOS-9031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16534840#comment-16534840
]
Kirill Plyashkevich commented on MESOS-9031:
--------------------------------------------
[~qianzhang],
well, that's your POV.
I can only state that the problem is not because of those rules.
for others having such problems, here's the short summary of the problem and
temp solution to it:
the problem setup:
- mesos unified containerizer + akka cluster nodes
- akka cluster nodes are configured with respect to [akka
recommendations|https://doc.akka.io/docs/akka/2.5/remoting.html#remote-configuration-nat]
with external IP/ports being host's IP and ports and internal ones being
container's IP (from bridge) and port.
for simplification:
-- host IP is 192.168.1.13
-- bridge is using 172.26.0.0/16
-- node1 is 172.26.1.1 listening on 2552 (host port 31030)
-- node2 is 172.26.1.2 listening on 2552 (host port 31031)
- connection flow
{quote}
172.26.1.1 -> 192.168.1.13:31031 -> port mapping to 172.26.1.2:2552
{quote}
with bridge config having
{quote}
"ipMasq": true
{quote}
relevant iptables' rules are
{quote}
Chain POSTROUTING (policy ACCEPT)
num target prot opt source destination
1 CNI-XXX all -- 172.26.0.0/16 anywhere /* name:
"dcos" id: "YYYY" */
{quote}
{quote}
Chain CNI-XXX (1 references)
num target prot opt source destination
1 ACCEPT all -- anywhere 172.26.0.0/16 /* name:
"dcos" id: "YYYY" */
2 MASQUERADE all -- anywhere !base-address.mcast.net/4 /*
name: "dcos" id: "YYYY" */
{quote}
it ends in ACCEPT (it's matching 172.26.0.0/16 -> 172.26.0.0/16).
what it actually needs is MASQUERADE.
temp solution:
- set
{quote}
"ipMasq": false
{quote}
- add
{quote}
iptables -t nat -N CNI-MESOS-FORCE-MASQ
iptables -t nat -A CNI-MESOS-FORCE-MASQ -m comment --comment "CNI force
masquerade" -j MASQUERADE
iptables -t nat -I POSTROUTING 1 -s 172.26.0.0/16 -j CNI-MESOS-FORCE-MASQ
{quote}
so all traffic from bridge gets MASQUERADE, it's not perfect, but it works.
I'll check with cni plugins, whether something can be done on the `cni-bridge`
side and/or `cni-portmap`.
> Mesos CNI portmap plugins' iptables rules doesn't allow connections via host
> ip and port from the same bridge container network
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: MESOS-9031
> URL: https://issues.apache.org/jira/browse/MESOS-9031
> Project: Mesos
> Issue Type: Bug
> Components: cni, containerization
> Affects Versions: 1.6.0
> Reporter: Kirill Plyashkevich
> Assignee: Qian Zhang
> Priority: Major
>
> using `mesos-cni-port-mapper` with folllowing config:
> {noformat}
> {
> "name" : "dcos",
> "type" : "mesos-cni-port-mapper",
> "excludeDevices" : [],
> "chain": "MESOS-CNI0-PORT-MAPPER",
> "delegate": {
> "type": "bridge",
> "bridge": "mesos-cni0",
> "isGateway": true,
> "ipMasq": true,
> "hairpinMode": true,
> "ipam": {
> "type": "host-local",
> "ranges": [
> [{"subnet": "172.26.0.0/16"}]
> ],
> "routes": [
> {"dst": "0.0.0.0/0"}
> ]
> }
> }
> }
> {noformat}
> - 2 services running on the same mesos-slave using unified containerizer in
> different tasks and communicating via host ip and host port
> - connection timeouts due to iptables rules per container CNI-XXX chain
> - actually timeouts are caused by
> {noformat}
> Chain CNI-XXX (1 references)
> num target prot opt source destination
> 1 ACCEPT all -- anywhere 172.26.0.0/16 /* name:
> "dcos" id: "YYYY" */
> 2 MASQUERADE all -- anywhere !base-address.mcast.net/4 /*
> name: "dcos" id: "YYYY" */
> {noformat}
> rule #1 is executed and no masquerading happens.
> there are multiple solutions:
> - -simpliest and fastest one is not to add that ACCEPT- - NOT A SOLUTION.
> it's happening in `bridge` plugin and `cni/portmap` shows that
> snat/masquerade should be done during portmapping as well.
> - perhaps, there's a better change in iptables rules that can fix it
> - proper one (imho) is to finally implement cni spec 0.3.x in order to be
> able to use chaining of plugins and use cni's `bridge` and `portmap` plugins
> in chain (and get rid of mesos-cni-port-mapper completely eventually).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
