Hello Cloudstack VM Issue, no outgoing access
There is an issue in cloudstack where a VM cannot gain outgoing access to the internet. This is a blocking issue since Cloudstack users cannot gain Internet access to do patch updates, download software, etc. Environment: Cloudstack version 4.3 System VM from January 2014 CentOS6.5 for all servers 4 servers (management server, 2 compute nodes, and a server that serves both primary and secondary storage) Advanced networking zone with 1 zone, 1 pod, 1 cluster, 2 hosts 2 NICs on the compute node hosts - 1 NIC hosts the management and storage - 1 NIC hosts the public and guest networks Primary and Secondard storage use NFS Management network is 172.16.10.x/24 Guest networks are 10.0.0.x/24 Public network uses a /24 public range Here is how to reproduce the prove the issue. 1. Create a new VM using a new isolated network 2. Create the egress rules to allow 10.0.0.0/24 out for everything 3. Create a port map rule for SSH to the new VM and also allow external acess to SSH 4. Login to the console on the new VM 5. Ping google.com, see 100% packet loss. 6. Login to the console on the virtual router for this network and type this command: Ping results [root@NewTestVM ~]# ping google.com PING google.com (74.125.225.128) 56(84) bytes of data. ^C --- google.com ping statistics --- 7 packets transmitted, 0 received, 100% packet loss, time 6127ms [root@NewTestVM ~]# All the iptables rules look as they should, especially look at the SNAT rule in the nat table for POSTROUTING. But, still can't ping. How do we know there isn't some other problem in the network. Well, first we were able to SSH to the new VM externally from the Internet to do the ping. Second, we found that if we add a second identical SNAT rule, now everything works. Okay so now add the second SNAT rule like this; iptables -t nat -A POSTROUTING -j SNAT –t-source 68.70.175.6 Yes, pinging google.com still does not work, losing 100% of packets.. Now, Pinging from the new VM works like a charm. What is going on?? [root@NewTestVM ~]# ping google.com PING google.com (74.125.225.132) 56(84) bytes of data. 64 bytes from ord08s09-in-f4.1e100.net (74.125.225.132): icmp_seq=1 ttl=52 time=2.12 ms 64 bytes from ord08s09-in-f4.1e100.net (74.125.225.132): icmp_seq=2 ttl=52 time=2.09 ms 64 bytes from ord08s09-in-f4.1e100.net (74.125.225.132): icmp_seq=3 ttl=52 time=2.22 ms 64 bytes from ord08s09-in-f4.1e100.net (74.125.225.132): icmp_seq=4 ttl=52 time=2.24 ms ^C --- google.com ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3556ms rtt min/avg/max/mdev = 2.095/2.172/2.249/0.064 ms [root@NewTestVM ~]# The first SNAT rule should have worked but it doesn't. WE've proven, I think, that everything else is setup correctly by being able to access the VM externally via SSH and by simply adding the SNAT rule to the virtual router. Why doesn't the first rule work. It seems cloudstack is doing exactly what it is supposed to do, adding the SNAT rule. So, it seems the actual problem is with either iptables or with the debian kernel for the system VM. I think we are using the most recent system VM. Note, neither restarting the new network nor cleaning it makes any difference. It just seems having the one SNAT rule will not work. I've done all the tcpdump debugging, that is how I found this issue to begin with. Tcpdump shows that the ping/icmp packets go through the virtual router unaltered using the private IP address of the VM (10.0.0.x). After the second SNAT rule is added tcpdump shows then that the ping/icmp packets do get translated. I am totally floored by these results. How can iptables or the debian kernel not recognize the first SNAT rule and then recognize the second one?? Okay, so before someone tells me that this is not a cloudstack issue and to find a resolution else, cloudstack depends on this technique for isolated networks. If guest users can't make their Vms work (note, they don't have access to the virtual routers like I do), isolated networks are broken and cloudstack is not usable. I have screen shots i can send if it will help.