Hi Andrei, Next step is to do some tcpdumping on the two VRs. Set some ping’s going and check:
Private NIC: tcpdump -i eth0 icmp Public NIC: tcpdump -i eth2 icmp That way you should be able to see how far your traffic reaches. Regards, Dag Sonstebo Cloud Architect ShapeBlue On 23/02/2018, 16:17, "Andrei Mikhailovsky" <and...@arhont.com.INVALID> wrote: Bump. Any ideas anyone? This issue is really annoying. Thanks dag.sonst...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue ----- Original Message ----- > From: "Andrei Mikhailovsky" <and...@arhont.com.INVALID> > To: "users" <users@cloudstack.apache.org>, "users" <users@cloudstack.apache.org> > Sent: Wednesday, 21 February, 2018 22:10:25 > Subject: Re: VR routing issues in Advanced Mode > Dag, > > > > > Yeah, we have egress traffic enabled. We also use VPCs on some of the networks > and they are also effected by this issue along with None VPC networks. > > > > > Any thoughts? > > > > > Andrei Mikhailovsky > > > > > > > > From: Dag Sonstebo > > > Sent: Wednesday 21 February, 18:30 > > > Subject: Re: VR routing issues in Advanced Mode > > > To: users@cloudstack.apache.org > > > > > > > Hi Andrei, > > > > > Understand. To get all the obvious things out the way – have you allowed egress > traffic on the two networks (you mention ACLs which we only use on VPCs and > basic networks)? > > > > > Regards, > > > Dag Sonstebo > > > Cloud Architect > > > ShapeBlue > > > > > On 21/02/2018, 14:51, "Andrei Mikhailovsky" <and...@arhont.com.INVALID> wrote: > > > > > Hi Dag, > > > > > Please see my comments below: > > > > >> Hi Andrei, > > >> > > >> You’re confusing the matter with your masking of public IP ranges. You said you > > >> have “2 x Public IP ranges with /26 netmask” – but since you are masking them > > >> out with X’s your email doesn’t make sense. If all the X’s are the same then a > > >> .10 and a .20 IP address would be on the same /26 network. > > >> > > >> I will assume that you do in fact have 2 x 26-bit networks, e.g.: > > >> > > >> 192.168.0.0/26 – with default gateway 192.168.0.1 > > >> 192.168.0.64/26 – with default gateway 192.168.0.65 > > >> > > > > > > > That is correct. I do have two separate /26 networks similar to what you've > described above. However, one /26 is used for direct public IP service > offering, where VRs are not involved in networking at all and the second /26 is > used for the service offering where VRs are used to provide the networking > function. > > > > > > >> If your two guest networks have VRs on separate public IP ranges you will have > > >> e.g. > > >> > > >> VR1: public IP 192.168.0.10 > > >> VR2: public IP 192.168.0.70 > > >> > > > > > > > Nope, the guest vms with VRs that can't talk to each other are on the same /26 > network. (in your example that would be on the same 192.168.0.0/26) > > > > > > >> For a VM hosted behind VR1 to reach a service NAT’ed on VR2 you need to set up > > >> routing and possibly firewalling on the data centre device which handles the > > >> default gateway for the two networks – i.e. the top of rack switch or router > > >> which hosts default gateways 192.168.0.1 and 192.168.0.65. The fact that you > > >> can reach services on both networks from outside this range makes sense. > > >> > > > > > These has been set up and vms between separate /26 networks CAN talk to each > other. The VMs on the same /26 network that doesn't use the VR service can also > talk to each other. The problem with VMs on the same /26 that use VRs can't > talk to each other using their public IP addresses. > > > > > > >> So once you have fixed this you will have VM1 > VR1 > DC_SWITCH_OR_ROUTER > VR2 > > >> > VM2. > > >> > > >> > > >> Regards, > > >> Dag Sonstebo > > >> Cloud Architect > > >> ShapeBlue > > >> > > >> On 21/02/2018, 12:27, "Andrei Mikhailovsky" <and...@arhont.com.INVALID> wrote: > > >> > > >> Hello > > >> > > >> Could someone help me to identify the routing issues that we have. The problem > > >> is the traffic from different guest networks can not reach each other via the > > >> public IPs. > > >> > > >> Here is my ACS setup: > > >> ACS 4.9.3.0 (both management and agents) > > >> KVM Hypervisor based on Ubuntu 16.04 > > >> Ceph as primary storage. NFS as secondary storage > > >> Advanced Networking with vlan separation > > >> 2 x Public IP ranges with /26 netmask. > > >> > > >> > > >> > > >> Here is an example when routing DOES NOT work: > > >> > > >> Case 1 - Advanced Networking, vlan separation, VRs route all traffic and provide > > >> all networking services (dhcp, fw, port forwarding, load balancing, etc) > > >> > > >> Guest Network 1: > > >> > > >> Public IP: XXX.XXX.XXX.10/26 > > >> Private IP range: 10.1.1.0/24 > > >> guest vm1 IP: 10.1.1.100/24 > > >> > > >> Guest Network 2: > > >> Public IP: XXX.XXX.XXX.20/26 > > >> Private IP range: 10.1.1.0/24 > > >> guest vm2 IP: 10.1.1.200/24 > > >> > > >> > > >> I've created ACLs on both guest networks to allow traffic from 0.0.0.0/0 on port > > >> 80. I've created the port forwarding rules to forward port 80 from public > > >> XXX.XXX.XXX.10 and XXX.XXX.XXX.XXX.20 onto 10.1.1.100 and 10.1.1.200 > > >> respectively. > > >> > > >> This setup works perfectly well when I am initiating the connections from > > >> outside of our CloudStack. However, vm2 can't reach vm1 on port 80 using the > > >> public IP XXX.XXX.XXX.10 and vice versa, vm1 can't reach vm2 on public IP > > >> XXX.XXX.XXX.20. > > >> > > >> > > >> > > >> > > >> Here is an example when the routing DOES work: > > >> > > >> Case 2 - Advanced Networking, vlan separation, VRs are not used. Public IPs are > > >> given directly to a guest vm > > >> > > >> Guest Network 1: > > >> > > >> guest vm1 Public IP: XXX.XXX.XXX.100/26 > > >> > > >> Guest Network 2: > > >> > > >> guest vm2 Public IP: XXX.XXX.XXX.110/26 > > >> > > >> In the Case 2, the guest vm has a public IP address directly assigned to its > > >> network interface. VRs are not used for this networking. Each guest has a fw > > >> rule to allow incoming traffic on port 80 from 0.0.0.0/0. Both vm1 and vm2 can > > >> access each other on port 80. Also, vms from Case 1 above can access port 80 on > > >> vms from Case 2, similarly, vms from Case 2 can access port 80 on vms from Case > > >> 1. > > >> > > >> > > >> > > >> So, it seems that the rules on the VR in Case 1 do not allow traffic that > > >> originates from other VRs within the same public network range. The trace route > > >> shows the last hop being the VR's private IP address. How do I change that > > >> behaviour and fix the networking issue? > > >> > > >> Thanks > > >> > > >> Andrei > > >> > > >> > > >> > > >> dag.sonst...@shapeblue.com > > >> www.shapeblue.com > > >> 53 Chandos Place, Covent Garden, London WC2N 4HSUK > > >> @shapeblue > > > > > > > > > dag.sonst...@shapeblue.com > > > www.shapeblue.com > > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > > > @shapeblue