Hello all, I’m testing with mesos 1.1.0 on aws linux to see if it applies to some of our processes and I ran into a problem with network/port_mapping, maybe this is a known issue?
The agent is running with these flags: export MESOS_isolation=cgroups/cpu,cgroups/mem,network/port_mapping export MESOS_containerizers=mesos export MESOS_resources="ports:[31000-32000];ephemeral_ports:[32768-57344]" export MESOS_ephemeral_ports_per_container=1024 Running spark 2.1.0 with 2 mesos containers on the same host, they can connect to each other’s block manager but can’t send traffic, it stays in their netns send-q. Spark is logging: 7/03/19 16:54:56 INFO TransportClientFactory: Successfully created connection to ip-10-32-20-34.ec2.internal/10.32.20.34:34294 after 12 ms (0 ms spent in bootstraps) 17/03/19 16:56:56 ERROR TransportChannelHandler: Connection to ip-10-32-20-34.ec2.internal/10.32.20.34:34294 has been quiet for 120000 ms while there are outstanding requests. Assuming connection is dead; please adjust spark.network.timeout if this is wrong. I can see connections established between containers but everything stays in the send Qs: [root@ip-10-32-20-34 sysctl.d]# ip netns 4602 (id: 1) 4600 (id: 0) [root@ip-10-32-20-34 sysctl.d]# ip netns exec 4600 netstat -an Connexions Internet actives (serveurs et établies) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 10.32.20.34:32861 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:33003 0.0.0.0:* LISTEN tcp 0 0 10.32.20.34:33003 10.32.20.34:57363 ESTABLISHED tcp 0 0 10.32.20.34:33566 10.32.20.34:34294 ESTABLISHED tcp 0 0 10.32.20.34:33658 10.32.18.185:40600 ESTABLISHED tcp 0 0 10.32.20.34:32832 10.32.18.185:40196 ESTABLISHED tcp 0 0 10.32.20.34:33406 10.32.20.34:5051 ESTABLISHED Sockets du domaine UNIX actives(serveurs et établies) Proto RefCpt Indicatrs Type Etat I-Node Chemin unix 2 [ ] STREAM CONNECTE 21869 unix 2 [ ] STREAM CONNECTE 20339 [root@ip-10-32-20-34 sysctl.d]# ip netns exec 4602 netstat -an Connexions Internet actives (serveurs et établies) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 0.0.0.0:33836 0.0.0.0:* LISTEN tcp 0 0 10.32.20.34:34294 0.0.0.0:* LISTEN tcp 0 24229 10.32.20.34:34294 10.32.20.34:33566 ESTABLISHED tcp 0 0 10.32.20.34:33860 10.32.18.185:40196 ESTABLISHED tcp 0 0 10.32.20.34:34680 10.32.18.185:40600 ESTABLISHED tcp 0 0 10.32.20.34:34434 10.32.20.34:5051 ESTABLISHED tcp 0 0 10.32.20.34:33836 10.32.20.34:58149 ESTABLISHED Sockets du domaine UNIX actives(serveurs et établies) Proto RefCpt Indicatrs Type Etat I-Node Chemin unix 2 [ ] STREAM CONNECTE 20359 unix 2 [ ] STREAM CONNECTE 20373 [root@ip-10-32-20-34 sysctl.d]#
