Hi Andrei, Sorry lost you - are you saying it's now all working when you allow icmp in your ACLs?
If not can you look at the tcpdumps on source and destination VRs as per my previous post? You may obviously have to run these on different interfaces depending on which VPC tier you are pinging from. Regards, Dag Sonstebo Cloud Architect ShapeBlue S: +44 20 3603 0540 | dag.sonst...@shapeblue.com | http://www.shapeblue.com <http://www.shapeblue.com/> | Twitter:@ShapeBlue <https://twitter.com/#!/shapeblue> On 27/02/2018, 10:27, "Andrei Mikhailovsky" <and...@arhont.com.INVALID> wrote: Hi Dag, Thanks, for your reply which I've missed earlier. I have done some more digging around and would like to make some corrections to the problem at hand. 1. It seems that the problem only effects VPC networks within cloudstack. 2. Networks which use non-VPC networking can talk to each other. 3. VPC to VPC traffic is not working. 4. VPC traffic can reach non-VPC network 5. non-VPC traffic can't reach VPC network In all cases, if the icmp is allowed in the ACLs, all networks can ping each other. So, the traffic is being routed and reaches the virtual router. Any advice? Thanks dag.sonst...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue ----- Original Message ----- > From: "Dag Sonstebo" <dag.sonst...@shapeblue.com> > To: "users" <users@cloudstack.apache.org> > Sent: Friday, 23 February, 2018 16:45:04 > Subject: Re: VR routing issues in Advanced Mode > Hi Andrei, > > Next step is to do some tcpdumping on the two VRs. Set some ping’s going and > check: > > Private NIC: tcpdump -i eth0 icmp > Public NIC: tcpdump -i eth2 icmp > > That way you should be able to see how far your traffic reaches. > > > Regards, > Dag Sonstebo > Cloud Architect > ShapeBlue > > On 23/02/2018, 16:17, "Andrei Mikhailovsky" <and...@arhont.com.INVALID> wrote: > > Bump. > > Any ideas anyone? This issue is really annoying. > > Thanks > > > dag.sonst...@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > > > > ----- Original Message ----- > > From: "Andrei Mikhailovsky" <and...@arhont.com.INVALID> > > To: "users" <users@cloudstack.apache.org>, "users" <users@cloudstack.apache.org> > > Sent: Wednesday, 21 February, 2018 22:10:25 > > Subject: Re: VR routing issues in Advanced Mode > > > Dag, > > > > > > > > > > Yeah, we have egress traffic enabled. We also use VPCs on some of the networks > > and they are also effected by this issue along with None VPC networks. > > > > > > > > > > Any thoughts? > > > > > > > > > > Andrei Mikhailovsky > > > > > > > > > > > > > > > > From: Dag Sonstebo > > > > > > Sent: Wednesday 21 February, 18:30 > > > > > > Subject: Re: VR routing issues in Advanced Mode > > > > > > To: users@cloudstack.apache.org > > > > > > > > > > > > > > Hi Andrei, > > > > > > > > > > Understand. To get all the obvious things out the way – have you allowed egress > > traffic on the two networks (you mention ACLs which we only use on VPCs and > > basic networks)? > > > > > > > > > > Regards, > > > > > > Dag Sonstebo > > > > > > Cloud Architect > > > > > > ShapeBlue > > > > > > > > > > On 21/02/2018, 14:51, "Andrei Mikhailovsky" <and...@arhont.com.INVALID> wrote: > > > > > > > > > > Hi Dag, > > > > > > > > > > Please see my comments below: > > > > > > > > > >> Hi Andrei, > > > > > >> > > > > > >> You’re confusing the matter with your masking of public IP ranges. You said you > > > > > >> have “2 x Public IP ranges with /26 netmask” – but since you are masking them > > > > > >> out with X’s your email doesn’t make sense. If all the X’s are the same then a > > > > > >> .10 and a .20 IP address would be on the same /26 network. > > > > > >> > > > > > >> I will assume that you do in fact have 2 x 26-bit networks, e.g.: > > > > > >> > > > > > >> 192.168.0.0/26 – with default gateway 192.168.0.1 > > > > > >> 192.168.0.64/26 – with default gateway 192.168.0.65 > > > > > >> > > > > > > > > > > > > > > That is correct. I do have two separate /26 networks similar to what you've > > described above. However, one /26 is used for direct public IP service > > offering, where VRs are not involved in networking at all and the second /26 is > > used for the service offering where VRs are used to provide the networking > > function. > > > > > > > > > > > > > >> If your two guest networks have VRs on separate public IP ranges you will have > > > > > >> e.g. > > > > > >> > > > > > >> VR1: public IP 192.168.0.10 > > > > > >> VR2: public IP 192.168.0.70 > > > > > >> > > > > > > > > > > > > > > Nope, the guest vms with VRs that can't talk to each other are on the same /26 > > network. (in your example that would be on the same 192.168.0.0/26) > > > > > > > > > > > > > >> For a VM hosted behind VR1 to reach a service NAT’ed on VR2 you need to set up > > > > > >> routing and possibly firewalling on the data centre device which handles the > > > > > >> default gateway for the two networks – i.e. the top of rack switch or router > > > > > >> which hosts default gateways 192.168.0.1 and 192.168.0.65. The fact that you > > > > > >> can reach services on both networks from outside this range makes sense. > > > > > >> > > > > > > > > > > These has been set up and vms between separate /26 networks CAN talk to each > > other. The VMs on the same /26 network that doesn't use the VR service can also > > talk to each other. The problem with VMs on the same /26 that use VRs can't > > talk to each other using their public IP addresses. > > > > > > > > > > > > > >> So once you have fixed this you will have VM1 > VR1 > DC_SWITCH_OR_ROUTER > VR2 > > > > > >> > VM2. > > > > > >> > > > > > >> > > > > > >> Regards, > > > > > >> Dag Sonstebo > > > > > >> Cloud Architect > > > > > >> ShapeBlue > > > > > >> > > > > > >> On 21/02/2018, 12:27, "Andrei Mikhailovsky" <and...@arhont.com.INVALID> wrote: > > > > > >> > > > > > >> Hello > > > > > >> > > > > > >> Could someone help me to identify the routing issues that we have. The problem > > > > > >> is the traffic from different guest networks can not reach each other via the > > > > > >> public IPs. > > > > > >> > > > > > >> Here is my ACS setup: > > > > > >> ACS 4.9.3.0 (both management and agents) > > > > > >> KVM Hypervisor based on Ubuntu 16.04 > > > > > >> Ceph as primary storage. NFS as secondary storage > > > > > >> Advanced Networking with vlan separation > > > > > >> 2 x Public IP ranges with /26 netmask. > > > > > >> > > > > > >> > > > > > >> > > > > > >> Here is an example when routing DOES NOT work: > > > > > >> > > > > > >> Case 1 - Advanced Networking, vlan separation, VRs route all traffic and provide > > > > > >> all networking services (dhcp, fw, port forwarding, load balancing, etc) > > > > > >> > > > > > >> Guest Network 1: > > > > > >> > > > > > >> Public IP: XXX.XXX.XXX.10/26 > > > > > >> Private IP range: 10.1.1.0/24 > > > > > >> guest vm1 IP: 10.1.1.100/24 > > > > > >> > > > > > >> Guest Network 2: > > > > > >> Public IP: XXX.XXX.XXX.20/26 > > > > > >> Private IP range: 10.1.1.0/24 > > > > > >> guest vm2 IP: 10.1.1.200/24 > > > > > >> > > > > > >> > > > > > >> I've created ACLs on both guest networks to allow traffic from 0.0.0.0/0 on port > > > > > >> 80. I've created the port forwarding rules to forward port 80 from public > > > > > >> XXX.XXX.XXX.10 and XXX.XXX.XXX.XXX.20 onto 10.1.1.100 and 10.1.1.200 > > > > > >> respectively. > > > > > >> > > > > > >> This setup works perfectly well when I am initiating the connections from > > > > > >> outside of our CloudStack. However, vm2 can't reach vm1 on port 80 using the > > > > > >> public IP XXX.XXX.XXX.10 and vice versa, vm1 can't reach vm2 on public IP > > > > > >> XXX.XXX.XXX.20. > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> Here is an example when the routing DOES work: > > > > > >> > > > > > >> Case 2 - Advanced Networking, vlan separation, VRs are not used. Public IPs are > > > > > >> given directly to a guest vm > > > > > >> > > > > > >> Guest Network 1: > > > > > >> > > > > > >> guest vm1 Public IP: XXX.XXX.XXX.100/26 > > > > > >> > > > > > >> Guest Network 2: > > > > > >> > > > > > >> guest vm2 Public IP: XXX.XXX.XXX.110/26 > > > > > >> > > > > > >> In the Case 2, the guest vm has a public IP address directly assigned to its > > > > > >> network interface. VRs are not used for this networking. Each guest has a fw > > > > > >> rule to allow incoming traffic on port 80 from 0.0.0.0/0. Both vm1 and vm2 can > > > > > >> access each other on port 80. Also, vms from Case 1 above can access port 80 on > > > > > >> vms from Case 2, similarly, vms from Case 2 can access port 80 on vms from Case > > > > > >> 1. > > > > > >> > > > > > >> > > > > > >> > > > > > >> So, it seems that the rules on the VR in Case 1 do not allow traffic that > > > > > >> originates from other VRs within the same public network range. The trace route > > > > > >> shows the last hop being the VR's private IP address. How do I change that > > > > > >> behaviour and fix the networking issue? > > > > > >> > > > > > >> Thanks > > > > > >> > > > > > >> Andrei > > > > > >> > > > > > >> > > > > > >> > > > > > >> dag.sonst...@shapeblue.com > > > > > >> www.shapeblue.com > > > > > >> 53 Chandos Place, Covent Garden, London WC2N 4HSUK > > > > > >> @shapeblue > > > > > > > > > > > > > > > > > > dag.sonst...@shapeblue.com > > > > > > www.shapeblue.com > > > > > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > > > > > > @shapeblue
