Hi Dag, Thanks, for your reply which I've missed earlier.
I have done some more digging around and would like to make some corrections to the problem at hand. 1. It seems that the problem only effects VPC networks within cloudstack. 2. Networks which use non-VPC networking can talk to each other. 3. VPC to VPC traffic is not working. 4. VPC traffic can reach non-VPC network 5. non-VPC traffic can't reach VPC network In all cases, if the icmp is allowed in the ACLs, all networks can ping each other. So, the traffic is being routed and reaches the virtual router. Any advice? Thanks ----- Original Message ----- > From: "Dag Sonstebo" <dag.sonst...@shapeblue.com> > To: "users" <users@cloudstack.apache.org> > Sent: Friday, 23 February, 2018 16:45:04 > Subject: Re: VR routing issues in Advanced Mode > Hi Andrei, > > Next step is to do some tcpdumping on the two VRs. Set some ping’s going and > check: > > Private NIC: tcpdump -i eth0 icmp > Public NIC: tcpdump -i eth2 icmp > > That way you should be able to see how far your traffic reaches. > > > Regards, > Dag Sonstebo > Cloud Architect > ShapeBlue > > On 23/02/2018, 16:17, "Andrei Mikhailovsky" <and...@arhont.com.INVALID> wrote: > > Bump. > > Any ideas anyone? This issue is really annoying. > > Thanks > > > dag.sonst...@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > > > > ----- Original Message ----- > > From: "Andrei Mikhailovsky" <and...@arhont.com.INVALID> > > To: "users" <users@cloudstack.apache.org>, "users" > <users@cloudstack.apache.org> > > Sent: Wednesday, 21 February, 2018 22:10:25 > > Subject: Re: VR routing issues in Advanced Mode > > > Dag, > > > > > > > > > > Yeah, we have egress traffic enabled. We also use VPCs on some of the > networks > > and they are also effected by this issue along with None VPC networks. > > > > > > > > > > Any thoughts? > > > > > > > > > > Andrei Mikhailovsky > > > > > > > > > > > > > > > > From: Dag Sonstebo > > > > > > Sent: Wednesday 21 February, 18:30 > > > > > > Subject: Re: VR routing issues in Advanced Mode > > > > > > To: users@cloudstack.apache.org > > > > > > > > > > > > > > Hi Andrei, > > > > > > > > > > Understand. To get all the obvious things out the way – have you allowed > egress > > traffic on the two networks (you mention ACLs which we only use on VPCs > and > > basic networks)? > > > > > > > > > > Regards, > > > > > > Dag Sonstebo > > > > > > Cloud Architect > > > > > > ShapeBlue > > > > > > > > > > On 21/02/2018, 14:51, "Andrei Mikhailovsky" <and...@arhont.com.INVALID> > wrote: > > > > > > > > > > Hi Dag, > > > > > > > > > > Please see my comments below: > > > > > > > > > >> Hi Andrei, > > > > > >> > > > > > >> You’re confusing the matter with your masking of public IP ranges. You > said you > > > > > >> have “2 x Public IP ranges with /26 netmask” – but since you are > masking them > > > > > >> out with X’s your email doesn’t make sense. If all the X’s are the same > then a > > > > > >> .10 and a .20 IP address would be on the same /26 network. > > > > > >> > > > > > >> I will assume that you do in fact have 2 x 26-bit networks, e.g.: > > > > > >> > > > > > >> 192.168.0.0/26 – with default gateway 192.168.0.1 > > > > > >> 192.168.0.64/26 – with default gateway 192.168.0.65 > > > > > >> > > > > > > > > > > > > > > That is correct. I do have two separate /26 networks similar to what > you've > > described above. However, one /26 is used for direct public IP service > > offering, where VRs are not involved in networking at all and the second > /26 is > > used for the service offering where VRs are used to provide the > networking > > function. > > > > > > > > > > > > > >> If your two guest networks have VRs on separate public IP ranges you > will have > > > > > >> e.g. > > > > > >> > > > > > >> VR1: public IP 192.168.0.10 > > > > > >> VR2: public IP 192.168.0.70 > > > > > >> > > > > > > > > > > > > > > Nope, the guest vms with VRs that can't talk to each other are on the > same /26 > > network. (in your example that would be on the same 192.168.0.0/26) > > > > > > > > > > > > > >> For a VM hosted behind VR1 to reach a service NAT’ed on VR2 you need to > set up > > > > > >> routing and possibly firewalling on the data centre device which > handles the > > > > > >> default gateway for the two networks – i.e. the top of rack switch or > router > > > > > >> which hosts default gateways 192.168.0.1 and 192.168.0.65. The fact > that you > > > > > >> can reach services on both networks from outside this range makes sense. > > > > > >> > > > > > > > > > > These has been set up and vms between separate /26 networks CAN talk to > each > > other. The VMs on the same /26 network that doesn't use the VR service > can also > > talk to each other. The problem with VMs on the same /26 that use VRs > can't > > talk to each other using their public IP addresses. > > > > > > > > > > > > > >> So once you have fixed this you will have VM1 > VR1 > > DC_SWITCH_OR_ROUTER > VR2 > > > > > >> > VM2. > > > > > >> > > > > > >> > > > > > >> Regards, > > > > > >> Dag Sonstebo > > > > > >> Cloud Architect > > > > > >> ShapeBlue > > > > > >> > > > > > >> On 21/02/2018, 12:27, "Andrei Mikhailovsky" <and...@arhont.com.INVALID> > wrote: > > > > > >> > > > > > >> Hello > > > > > >> > > > > > >> Could someone help me to identify the routing issues that we have. The > problem > > > > > >> is the traffic from different guest networks can not reach each other > via the > > > > > >> public IPs. > > > > > >> > > > > > >> Here is my ACS setup: > > > > > >> ACS 4.9.3.0 (both management and agents) > > > > > >> KVM Hypervisor based on Ubuntu 16.04 > > > > > >> Ceph as primary storage. NFS as secondary storage > > > > > >> Advanced Networking with vlan separation > > > > > >> 2 x Public IP ranges with /26 netmask. > > > > > >> > > > > > >> > > > > > >> > > > > > >> Here is an example when routing DOES NOT work: > > > > > >> > > > > > >> Case 1 - Advanced Networking, vlan separation, VRs route all traffic > and provide > > > > > >> all networking services (dhcp, fw, port forwarding, load balancing, etc) > > > > > >> > > > > > >> Guest Network 1: > > > > > >> > > > > > >> Public IP: XXX.XXX.XXX.10/26 > > > > > >> Private IP range: 10.1.1.0/24 > > > > > >> guest vm1 IP: 10.1.1.100/24 > > > > > >> > > > > > >> Guest Network 2: > > > > > >> Public IP: XXX.XXX.XXX.20/26 > > > > > >> Private IP range: 10.1.1.0/24 > > > > > >> guest vm2 IP: 10.1.1.200/24 > > > > > >> > > > > > >> > > > > > >> I've created ACLs on both guest networks to allow traffic from > 0.0.0.0/0 on port > > > > > >> 80. I've created the port forwarding rules to forward port 80 from > public > > > > > >> XXX.XXX.XXX.10 and XXX.XXX.XXX.XXX.20 onto 10.1.1.100 and 10.1.1.200 > > > > > >> respectively. > > > > > >> > > > > > >> This setup works perfectly well when I am initiating the connections > from > > > > > >> outside of our CloudStack. However, vm2 can't reach vm1 on port 80 > using the > > > > > >> public IP XXX.XXX.XXX.10 and vice versa, vm1 can't reach vm2 on public > IP > > > > > >> XXX.XXX.XXX.20. > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> Here is an example when the routing DOES work: > > > > > >> > > > > > >> Case 2 - Advanced Networking, vlan separation, VRs are not used. Public > IPs are > > > > > >> given directly to a guest vm > > > > > >> > > > > > >> Guest Network 1: > > > > > >> > > > > > >> guest vm1 Public IP: XXX.XXX.XXX.100/26 > > > > > >> > > > > > >> Guest Network 2: > > > > > >> > > > > > >> guest vm2 Public IP: XXX.XXX.XXX.110/26 > > > > > >> > > > > > >> In the Case 2, the guest vm has a public IP address directly assigned > to its > > > > > >> network interface. VRs are not used for this networking. Each guest has > a fw > > > > > >> rule to allow incoming traffic on port 80 from 0.0.0.0/0. Both vm1 and > vm2 can > > > > > >> access each other on port 80. Also, vms from Case 1 above can access > port 80 on > > > > > >> vms from Case 2, similarly, vms from Case 2 can access port 80 on vms > from Case > > > > > >> 1. > > > > > >> > > > > > >> > > > > > >> > > > > > >> So, it seems that the rules on the VR in Case 1 do not allow traffic > that > > > > > >> originates from other VRs within the same public network range. The > trace route > > > > > >> shows the last hop being the VR's private IP address. How do I change > that > > > > > >> behaviour and fix the networking issue? > > > > > >> > > > > > >> Thanks > > > > > >> > > > > > >> Andrei > > > > > >> > > > > > >> > > > > > >> > > > > > >> dag.sonst...@shapeblue.com > > > > > >> www.shapeblue.com > > > > > >> 53 Chandos Place, Covent Garden, London WC2N 4HSUK > > > > > >> @shapeblue > > > > > > > > > > > > > > > > > > dag.sonst...@shapeblue.com > > > > > > www.shapeblue.com > > > > > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > > > > > > @shapeblue
