Hi Andrei,

Next step is to do some tcpdumping on the two VRs. Set some ping’s going and 
check:

Private NIC: tcpdump -i eth0 icmp
Public NIC: tcpdump -i eth2 icmp

That way you should be able to see how far your traffic reaches.


Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue

On 23/02/2018, 16:17, "Andrei Mikhailovsky" <and...@arhont.com.INVALID> wrote:

    Bump.
    
    Any ideas anyone? This issue is really annoying.
    
    Thanks
    
    
dag.sonst...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

----- Original Message -----
    > From: "Andrei Mikhailovsky" <and...@arhont.com.INVALID>
    > To: "users" <users@cloudstack.apache.org>, "users" 
<users@cloudstack.apache.org>
    > Sent: Wednesday, 21 February, 2018 22:10:25
    > Subject: Re: VR routing issues in Advanced Mode
    
    > Dag,
    > 
    > 
    > 
    > 
    > Yeah, we have egress traffic enabled. We also use VPCs on some of the 
networks
    > and they are also effected by this issue along with None VPC networks.
    > 
    > 
    > 
    > 
    > Any thoughts?
    > 
    > 
    > 
    > 
    > Andrei Mikhailovsky
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > From: Dag Sonstebo
    > 
    > 
    > Sent: Wednesday 21 February, 18:30
    > 
    > 
    > Subject: Re: VR routing issues in Advanced Mode
    > 
    > 
    > To: users@cloudstack.apache.org
    > 
    > 
    > 
    > 
    > 
    > 
    > Hi Andrei,
    > 
    > 
    > 
    > 
    > Understand. To get all the obvious things out the way – have you allowed 
egress
    > traffic on the two networks (you mention ACLs which we only use on VPCs 
and
    > basic networks)?
    > 
    > 
    > 
    > 
    > Regards,
    > 
    > 
    > Dag Sonstebo
    > 
    > 
    > Cloud Architect
    > 
    > 
    > ShapeBlue
    > 
    > 
    > 
    > 
    > On 21/02/2018, 14:51, "Andrei Mikhailovsky" <and...@arhont.com.INVALID> 
wrote:
    > 
    > 
    > 
    > 
    > Hi Dag,
    > 
    > 
    > 
    > 
    > Please see my comments below:
    > 
    > 
    > 
    > 
    >> Hi Andrei,
    > 
    > 
    >> 
    > 
    > 
    >> You’re confusing the matter with your masking of public IP ranges. You 
said you
    > 
    > 
    >> have “2 x Public IP ranges with /26 netmask” – but since you are masking 
them
    > 
    > 
    >> out with X’s your email doesn’t make sense. If all the X’s are the same 
then a
    > 
    > 
    >> .10 and a .20 IP address would be on the same /26 network.
    > 
    > 
    >> 
    > 
    > 
    >> I will assume that you do in fact have 2 x 26-bit networks, e.g.:
    > 
    > 
    >> 
    > 
    > 
    >> 192.168.0.0/26 – with default gateway 192.168.0.1
    > 
    > 
    >> 192.168.0.64/26 – with default gateway 192.168.0.65
    > 
    > 
    >> 
    > 
    > 
    > 
    > 
    > 
    > 
    > That is correct. I do have two separate /26 networks similar to what 
you've
    > described above. However, one /26 is used for direct public IP service
    > offering, where VRs are not involved in networking at all and the second 
/26 is
    > used for the service offering where VRs are used to provide the networking
    > function.
    > 
    > 
    > 
    > 
    > 
    > 
    >> If your two guest networks have VRs on separate public IP ranges you 
will have
    > 
    > 
    >> e.g.
    > 
    > 
    >> 
    > 
    > 
    >> VR1: public IP 192.168.0.10
    > 
    > 
    >> VR2: public IP 192.168.0.70
    > 
    > 
    >> 
    > 
    > 
    > 
    > 
    > 
    > 
    > Nope, the guest vms with VRs that can't talk to each other are on the 
same /26
    > network. (in your example that would be on the same 192.168.0.0/26)
    > 
    > 
    > 
    > 
    > 
    > 
    >> For a VM hosted behind VR1 to reach a service NAT’ed on VR2 you need to 
set up
    > 
    > 
    >> routing and possibly firewalling on the data centre device which handles 
the
    > 
    > 
    >> default gateway for the two networks – i.e. the top of rack switch or 
router
    > 
    > 
    >> which hosts default gateways 192.168.0.1 and 192.168.0.65. The fact that 
you
    > 
    > 
    >> can reach services on both networks from outside this range makes sense.
    > 
    > 
    >> 
    > 
    > 
    > 
    > 
    > These has been set up and vms between separate /26 networks CAN talk to 
each
    > other. The VMs on the same /26 network that doesn't use the VR service 
can also
    > talk to each other. The problem with VMs on the same /26 that use VRs 
can't
    > talk to each other using their public IP addresses.
    > 
    > 
    > 
    > 
    > 
    > 
    >> So once you have fixed this you will have VM1 > VR1 > 
DC_SWITCH_OR_ROUTER > VR2
    > 
    > 
    >> > VM2.
    > 
    > 
    >> 
    > 
    > 
    >> 
    > 
    > 
    >> Regards,
    > 
    > 
    >> Dag Sonstebo
    > 
    > 
    >> Cloud Architect
    > 
    > 
    >> ShapeBlue
    > 
    > 
    >> 
    > 
    > 
    >> On 21/02/2018, 12:27, "Andrei Mikhailovsky" <and...@arhont.com.INVALID> 
wrote:
    > 
    > 
    >> 
    > 
    > 
    >> Hello
    > 
    > 
    >> 
    > 
    > 
    >> Could someone help me to identify the routing issues that we have. The 
problem
    > 
    > 
    >> is the traffic from different guest networks can not reach each other 
via the
    > 
    > 
    >> public IPs.
    > 
    > 
    >> 
    > 
    > 
    >> Here is my ACS setup:
    > 
    > 
    >> ACS 4.9.3.0 (both management and agents)
    > 
    > 
    >> KVM Hypervisor based on Ubuntu 16.04
    > 
    > 
    >> Ceph as primary storage. NFS as secondary storage
    > 
    > 
    >> Advanced Networking with vlan separation
    > 
    > 
    >> 2 x Public IP ranges with /26 netmask.
    > 
    > 
    >> 
    > 
    > 
    >> 
    > 
    > 
    >> 
    > 
    > 
    >> Here is an example when routing DOES NOT work:
    > 
    > 
    >> 
    > 
    > 
    >> Case 1 - Advanced Networking, vlan separation, VRs route all traffic and 
provide
    > 
    > 
    >> all networking services (dhcp, fw, port forwarding, load balancing, etc)
    > 
    > 
    >> 
    > 
    > 
    >> Guest Network 1:
    > 
    > 
    >> 
    > 
    > 
    >> Public IP: XXX.XXX.XXX.10/26
    > 
    > 
    >> Private IP range: 10.1.1.0/24
    > 
    > 
    >> guest vm1 IP: 10.1.1.100/24
    > 
    > 
    >> 
    > 
    > 
    >> Guest Network 2:
    > 
    > 
    >> Public IP: XXX.XXX.XXX.20/26
    > 
    > 
    >> Private IP range: 10.1.1.0/24
    > 
    > 
    >> guest vm2 IP: 10.1.1.200/24
    > 
    > 
    >> 
    > 
    > 
    >> 
    > 
    > 
    >> I've created ACLs on both guest networks to allow traffic from 0.0.0.0/0 
on port
    > 
    > 
    >> 80. I've created the port forwarding rules to forward port 80 from public
    > 
    > 
    >> XXX.XXX.XXX.10 and XXX.XXX.XXX.XXX.20 onto 10.1.1.100 and 10.1.1.200
    > 
    > 
    >> respectively.
    > 
    > 
    >> 
    > 
    > 
    >> This setup works perfectly well when I am initiating the connections from
    > 
    > 
    >> outside of our CloudStack. However, vm2 can't reach vm1 on port 80 using 
the
    > 
    > 
    >> public IP XXX.XXX.XXX.10 and vice versa, vm1 can't reach vm2 on public IP
    > 
    > 
    >> XXX.XXX.XXX.20.
    > 
    > 
    >> 
    > 
    > 
    >> 
    > 
    > 
    >> 
    > 
    > 
    >> 
    > 
    > 
    >> Here is an example when the routing DOES work:
    > 
    > 
    >> 
    > 
    > 
    >> Case 2 - Advanced Networking, vlan separation, VRs are not used. Public 
IPs are
    > 
    > 
    >> given directly to a guest vm
    > 
    > 
    >> 
    > 
    > 
    >> Guest Network 1:
    > 
    > 
    >> 
    > 
    > 
    >> guest vm1 Public IP: XXX.XXX.XXX.100/26
    > 
    > 
    >> 
    > 
    > 
    >> Guest Network 2:
    > 
    > 
    >> 
    > 
    > 
    >> guest vm2 Public IP: XXX.XXX.XXX.110/26
    > 
    > 
    >> 
    > 
    > 
    >> In the Case 2, the guest vm has a public IP address directly assigned to 
its
    > 
    > 
    >> network interface. VRs are not used for this networking. Each guest has 
a fw
    > 
    > 
    >> rule to allow incoming traffic on port 80 from 0.0.0.0/0. Both vm1 and 
vm2 can
    > 
    > 
    >> access each other on port 80. Also, vms from Case 1 above can access 
port 80 on
    > 
    > 
    >> vms from Case 2, similarly, vms from Case 2 can access port 80 on vms 
from Case
    > 
    > 
    >> 1.
    > 
    > 
    >> 
    > 
    > 
    >> 
    > 
    > 
    >> 
    > 
    > 
    >> So, it seems that the rules on the VR in Case 1 do not allow traffic that
    > 
    > 
    >> originates from other VRs within the same public network range. The 
trace route
    > 
    > 
    >> shows the last hop being the VR's private IP address. How do I change 
that
    > 
    > 
    >> behaviour and fix the networking issue?
    > 
    > 
    >> 
    > 
    > 
    >> Thanks
    > 
    > 
    >> 
    > 
    > 
    >> Andrei
    > 
    > 
    >> 
    > 
    > 
    >> 
    > 
    > 
    >> 
    > 
    > 
    >> dag.sonst...@shapeblue.com
    > 
    > 
    >> www.shapeblue.com
    > 
    > 
    >> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
    > 
    > 
    >> @shapeblue
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > dag.sonst...@shapeblue.com
    > 
    > 
    > www.shapeblue.com
    > 
    > 
    > 53 Chandos Place, Covent Garden, London WC2N 4HSUK
    > 
    > 
    > @shapeblue
    

Reply via email to