Repository: cloudstack-docs Updated Branches: refs/heads/master bcf67dd6f -> 5fddad01e
http://git-wip-us.apache.org/repos/asf/cloudstack-docs/blob/5fddad01/rtd/source/networking/troubleshoot_internet_traffic.rst ---------------------------------------------------------------------- diff --git a/rtd/source/networking/troubleshoot_internet_traffic.rst b/rtd/source/networking/troubleshoot_internet_traffic.rst new file mode 100644 index 0000000..b118e38 --- /dev/null +++ b/rtd/source/networking/troubleshoot_internet_traffic.rst @@ -0,0 +1,216 @@ +Troubleshooting Internet Traffic +================================ + +Below are a few troubleshooting steps to check whats going wrong with your +network... + +Trouble Shooting Steps +---------------------- + +#. The switches have to be configured correctly to pass VLAN traffic. You can + verify if VLAN traffic is working by bringing up a tagged interface on the + hosts and pinging between them as below... + + On *host1 (kvm1)* + + :: + + kvm1 ~$ vconfig add eth0 64 + kvm1 ~$ ifconfig eth0.64 1.2.3.4 netmask 255.255.255.0 up + kvm1 ~$ ping 1.2.3.5 + + On *host2 (kvm2)* + + :: + + kvm2 ~$ vconfig add eth0 64 + kvm2 ~$ ifconfig eth0.64 1.2.3.5 netmask 255.255.255.0 up + kvm2 ~$ ping 1.2.3.4 + + If the pings dont work, run *tcpdump(8)* all over the place to check + who is gobbling up the packets. Ultimately, if the switches are not + configured correctly, CloudStack networking wont work so fix the + physical networking issues before you proceed to the next steps + +#. Ensure `Traffic Labels <http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.2.0/html/Installation_Guide/about-physical-networks.html>`_ are set for the Zone. + + Traffic labels need to be set for all hypervisors including + XenServer, KVM and VMware types. You can configure traffic labels when + you creating a new zone from the *Add Zone Wizard*. + + .. image:: ../_static/images/networking-zone-traffic-labels.png + + On an existing zone, you can modify the traffic labels by going to + *Infrastructure, Zones, Physical Network* tab. + + .. image:: ../_static/images/networking-infra-traffic-labels.png + + List labels using *CloudMonkey* + + :: + + acs-manager ~$ cloudmonkey list traffictypes physicalnetworkid=41cb7ff6-8eb2-4630-b577-1da25e0e1145 + count = 4 + traffictype: + id = cd0915fe-a660-4a82-9df7-34aebf90003e + kvmnetworklabel = cloudbr0 + physicalnetworkid = 41cb7ff6-8eb2-4630-b577-1da25e0e1145 + traffictype = Guest + xennetworklabel = MGMT + ======================================================== + id = f5524b8f-6605-41e4-a982-81a356b2a196 + kvmnetworklabel = cloudbr0 + physicalnetworkid = 41cb7ff6-8eb2-4630-b577-1da25e0e1145 + traffictype = Management + xennetworklabel = MGMT + ======================================================== + id = 266bad0e-7b68-4242-b3ad-f59739346cfd + kvmnetworklabel = cloudbr0 + physicalnetworkid = 41cb7ff6-8eb2-4630-b577-1da25e0e1145 + traffictype = Public + xennetworklabel = MGMT + ======================================================== + id = a2baad4f-7ce7-45a8-9caf-a0b9240adf04 + kvmnetworklabel = cloudbr0 + physicalnetworkid = 41cb7ff6-8eb2-4630-b577-1da25e0e1145 + traffictype = Storage + xennetworklabel = MGMT + ========================================================= + +#. KVM traffic labels require to be named as *"cloudbr0"*, *"cloudbr2"*, + *"cloudbrN"* etc and the corresponding bridge must exist on the KVM + hosts. If you create labels/bridges with any other names, CloudStack + (atleast earlier versions did) seems to ignore them. CloudStack does not + create the physical bridges on the KVM hosts, you need to create them + **before** before adding the host to Cloudstack. + + :: + + kvm1 ~$ ifconfig cloudbr0 + cloudbr0 Link encap:Ethernet HWaddr 00:0C:29:EF:7D:78 + inet addr:192.168.44.22 Bcast:192.168.44.255 Mask:255.255.255.0 + inet6 addr: fe80::20c:29ff:feef:7d78/64 Scope:Link + UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 + RX packets:92435 errors:0 dropped:0 overruns:0 frame:0 + TX packets:50596 errors:0 dropped:0 overruns:0 carrier:0 + collisions:0 txqueuelen:0 + RX bytes:94985932 (90.5 MiB) TX bytes:61635793 (58.7 MiB) + +#. The Virtual Router, SSVM, CPVM *public* interface would be bridged to + a physical interface on the host. In the example below, *cloudbr0* is + the public interface and CloudStack has correctly created the virtual + interfaces bridge. This virtual interface to physical interface mapping + is done automatically by CloudStack using the traffic label settings for + the Zone. If you have provided correct settings and still dont have a + working working Internet, check the switching layer before you debug any + further. You can verify traffic using tcpdump on the virtual, physical + and bridge interfaces. + + :: + + kvm-host1 ~$ brctl show + bridge name bridge id STP enabled interfaces + breth0-64 8000.000c29ef7d78 no eth0.64 + vnet2 + cloud0 8000.fe00a9fe0219 no vnet0 + cloudbr0 8000.000c29ef7d78 no eth0 + vnet1 + vnet3 + virbr0 8000.5254008e321a yes virbr0-nic + + :: + + xenserver1 ~$ brctl show + bridge name bridge id STP enabled interfaces + xapi0 0000.e2b76d0a1149 no vif1.0 + xenbr0 0000.000c299b54dc no eth0 + xapi1 + vif1.1 + vif1.2 + +#. Pre-create labels on the XenServer Hosts. Similar to KVM bridge + setup, traffic labels must also be pre-created on the XenServer hosts + before adding them to CloudStack. + + :: + + xenserver1 ~$ xe network-list + uuid ( RO) : aaa-bbb-ccc-ddd + name-label ( RW): MGMT + name-description ( RW): + bridge ( RO): xenbr0 + + +#. The Internet would be accessible from both the SSVM and CPVM + instances by default. Their public IPs will also be directly pingable + from the Internet. Please note that these test would work only if your + switches and traffic labels are configured correctly for your + environment. If your SSVM/CPVM cant reach the Internet, its very + unlikely that the Virtual Router (VR) can also the reach the Internet + suggesting that its either a switching issue or incorrectly assigned + traffic labels. Fix the SSVM/CPVM issues before you debug VR issues. + + :: + + root@s-1-VM:~# ping -c 3 google.com + PING google.com (74.125.236.164): 56 data bytes + 64 bytes from 74.125.236.164: icmp_seq=0 ttl=55 time=26.932 ms + 64 bytes from 74.125.236.164: icmp_seq=1 ttl=55 time=29.156 ms + 64 bytes from 74.125.236.164: icmp_seq=2 ttl=55 time=25.000 ms + --- google.com ping statistics --- + 3 packets transmitted, 3 packets received, 0% packet loss + round-trip min/avg/max/stddev = 25.000/27.029/29.156/1.698 ms + + :: + + root@v-2-VM:~# ping -c 3 google.com + PING google.com (74.125.236.164): 56 data bytes + 64 bytes from 74.125.236.164: icmp_seq=0 ttl=55 time=32.125 ms + 64 bytes from 74.125.236.164: icmp_seq=1 ttl=55 time=26.324 ms + 64 bytes from 74.125.236.164: icmp_seq=2 ttl=55 time=37.001 ms + --- google.com ping statistics --- + 3 packets transmitted, 3 packets received, 0% packet loss + round-trip min/avg/max/stddev = 26.324/31.817/37.001/4.364 ms + + +#. The Virtual Router (VR) should also be able to reach the Internet + without having any Egress rules. The Egress rules only control forwarded + traffic and not traffic that originates on the VR itself. + + :: + + root@r-4-VM:~# ping -c 3 google.com + PING google.com (74.125.236.164): 56 data bytes + 64 bytes from 74.125.236.164: icmp_seq=0 ttl=55 time=28.098 ms + 64 bytes from 74.125.236.164: icmp_seq=1 ttl=55 time=34.785 ms + 64 bytes from 74.125.236.164: icmp_seq=2 ttl=55 time=69.179 ms + --- google.com ping statistics --- + 3 packets transmitted, 3 packets received, 0% packet loss + round-trip min/avg/max/stddev = 28.098/44.021/69.179/17.998 ms + +#. However, the Virtual Router's (VR) Source NAT Public IP address + **WONT** be reachable until appropriate Ingress rules are + in place. You can add *Ingress* rules under *Network, Guest Network, IP + Address, Firewall* setting page. + + .. image:: ../_static/images/networking-ingress-rule.png + +#. The VM Instances by default wont be able to access the Internet. Add + Egress rules to permit traffic. + + .. image:: ../_static/images/networking-egress-rule.png + +#. Some users have reported that flushing IPTables rules (or changing + routes) on the SSVM, CPVM or the Virtual Router makes the Internet work. + This is not expected behaviour and suggests that your networking + settings are incorrect. No IPtables/route changes are required on the + SSVM, CPVM or the VR. Go back and double check all your settings. + + +In a vast majority of the cases, the problem has turned out to be at the +switching layer where the L3 switches were configured incorrectly. + +This section was contibuted by Shanker Balan and was originally published +at [here]_ + +.. [here] http://shankerbalan.net/blog/internet-not-working-on-cloudstack-vms/ http://git-wip-us.apache.org/repos/asf/cloudstack-docs/blob/5fddad01/rtd/source/networking/vxlan.rst ---------------------------------------------------------------------- diff --git a/rtd/source/networking/vxlan.rst b/rtd/source/networking/vxlan.rst new file mode 100644 index 0000000..24520a3 --- /dev/null +++ b/rtd/source/networking/vxlan.rst @@ -0,0 +1,376 @@ +The VXLAN Plugin +================ + +System Requirements for VXLAN +----------------------------- + +In PRODUCT 4.X.0, this plugin only supports the KVM hypervisor with the +standard linux bridge. + +The following table lists the requirements for the hypervisor. + ++----------------+-----------------------------------------------+----------------------------------------------------------------------------------------------------------------+ +| Item | Requirement | Note | ++================+===============================================+================================================================================================================+ +| Hypervisor | KVM | OvsVifDriver is not supported by this plugin in PRODUCT 4.X, use BridgeVifDriver (default). | ++----------------+-----------------------------------------------+----------------------------------------------------------------------------------------------------------------+ +| Linux kernel | version >= 3.7, VXLAN kernel module enabled | It is recommended to use kernel >=3.9, since Linux kernel categorizes the VXLAN driver as experimental <3.9. | ++----------------+-----------------------------------------------+----------------------------------------------------------------------------------------------------------------+ +| iproute2 | matches kernel version | | ++----------------+-----------------------------------------------+----------------------------------------------------------------------------------------------------------------+ + +Table: Hypervisor Requirement for VXLAN + +Linux Distributions that meet the requirements +---------------------------------------------- + +The following table lists distributions which meet requirements. + ++----------------+-------------------+-------------------------------------------+----------------------------------------------------------------+ +| Distribution | Release Version | Kernel Version (Date confirmed) | Note | ++================+===================+===========================================+================================================================+ +| Ubuntu | 13.04 | 3.8.0 (2013/07/23) | | ++----------------+-------------------+-------------------------------------------+----------------------------------------------------------------+ +| Fedora | >= 17 | 3.9.10 (2013/07/23) | Latest kernel packages are available in "update" repository. | ++----------------+-------------------+-------------------------------------------+----------------------------------------------------------------+ +| CentOS | >= 6.5 | 2.6.32-431.3.1.el6.x86\_64 (2014/01/21) | | ++----------------+-------------------+-------------------------------------------+----------------------------------------------------------------+ + +Table: List of Linux distributions which meet the hypervisor +requirements + +Check the capability of your system +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To check the capability of your system, execute the following commands. + +:: + + $ sudo modprobe vxlan && echo $? + # Confirm the output is "0". + # If it's non-0 value or error message, your kernel doesn't have VXLAN kernel module. + + $ ip link add type vxlan help + # Confirm the output is usage of the command and that it's for VXLAN. + # If it's not, your iproute2 utility doesn't support VXLAN. + + +Advanced: Build kernel and iproute2 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Even if your system doesn't support VXLAN, you can compile the kernel +and iproute2 by yourself. The following procedure is an example for +CentOS 6.4. + +Build kernel +^^^^^^^^^^^^ + +:: + + $ sudo yum groupinstall "Development Tools" + $ sudo yum install ncurses-devel hmaccalc zlib-devel binutils-devel elfutils-libelf-devel bc + + $ KERNEL_VERSION=3.10.4 + # Declare the kernel version you want to build. + + $ wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-${KERNEL_VERSION}.tar.xz + $ tar xvf linux-${KERNEL_VERSION}.tar.xz + $ cd linux-${KERNEL_VERSION} + $ cp /boot/config-`uname -r` .config + $ make oldconfig + # You may keep hitting enter and choose the default. + + $ make menuconfig + # Dig into "Device Drivers" -> "Network device support", + # then select "Virtual eXtensible Local Area Network (VXLAN)" and hit space. + # Make sure it indicates "<M>" (build as module), then Save and Exit. + + # You may also want to check "IPv4 NAT" and its child nodes in "IP: Netfilter Configuration" + # and "IPv6 NAT" and its child nodes in "IPv6: Netfilter Configuration". + # In 3.10.4, you can find the options in + # "Networking support" -> "Networking options" + # -> "Network packet filtering framework (Netfilter)". + + $ make # -j N + # You may use -j N option to make the build process parallel and faster, + # generally N = 1 + (cores your machine have). + + $ sudo make modules_install + $ sudo make install + # You would get an error like "ERROR: modinfo: could not find module XXXX" here. + # This happens mainly due to config structure changes between kernel versions. + # You can ignore this error, until you find you need the kernel module. + # If you feel uneasy, you can go back to make menuconfig, + # find module XXXX by using '/' key, enable the module, build and install the kernel again. + + $ sudo vi /etc/grub.conf + # Make sure the new kernel isn't set as the default and the timeout is long enough, + # so you can select the new kernel during boot process. + # It's not a good idea to set the new kernel as the default until you confirm the kernel works fine. + + $ sudo reboot + # Select the new kernel during the boot process. + + +Build iproute2 +^^^^^^^^^^^^^^ + +:: + + $ sudo yum install db4-devel + + $ git clone git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git + $ cd iproute2 + $ git tag + # Find the version that matches the kernel. + # If you built kernel 3.10.4 as above, it would be v3.10.0. + + $ git checkout v3.10.0 + $ ./configure + $ make # -j N + $ sudo make install + + +.. note:: Please use rebuild kernel and tools at your own risk. + +Configure PRODUCT to use VXLAN Plugin +------------------------------------- + +Configure hypervisor +~~~~~~~~~~~~~~~~~~~~ + +Configure hypervisor: KVM +^^^^^^^^^^^^^^^^^^^^^^^^^ + +In addition to "KVM Hypervisor Host Installation" in "PRODUCT +Installation Guide", you have to configure the following item on the +host. + +Create bridge interface with IPv4 address +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This plugin requires an IPv4 address on the KVM host to terminate and +originate VXLAN traffic. The address should be assinged to a physical +interface or a bridge interface bound to a physical interface. Both a +private address or a public address are fine for the purpose. It is not +required to be in the same subnet for all hypervisors in a zone, but +they should be able to reach each other via IP multicast with UDP/8472 +port. A name of a physical interface or a name of a bridge interface +bound to a physical interface can be used as a traffic label. Physical +interface name fits for almost all cases, but if physical interface name +differs per host, you may use a bridge to set a same name. If you would +like to use a bridge name as a traffic label, you may create a bridge in +this way. + +Let ``cloudbr1`` be the bridge interface for the instances' private +network. + +Configure in RHEL or CentOS +''''''''''''''''''''''''''' + +When you configured the ``cloudbr1`` interface as below, + +:: + + $ sudo vi /etc/sysconfig/network-scripts/ifcfg-cloudbr1 + + +:: + + DEVICE=cloudbr1 + TYPE=Bridge + ONBOOT=yes + BOOTPROTO=none + IPV6INIT=no + IPV6_AUTOCONF=no + DELAY=5 + STP=yes + + +you would change the configuration similar to below. + +:: + + DEVICE=cloudbr1 + TYPE=Bridge + ONBOOT=yes + BOOTPROTO=static + IPADDR=192.0.2.X + NETMASK=255.255.255.0 + IPV6INIT=no + IPV6_AUTOCONF=no + DELAY=5 + STP=yes + + +Configure in Ubuntu +''''''''''''''''''' + +When you configured ``cloudbr1`` as below, + +:: + + $ sudo vi /etc/network/interfaces + +:: + + auto lo + iface lo inet loopback + + # The primary network interface + auto eth0.100 + iface eth0.100 inet static + address 192.168.42.11 + netmask 255.255.255.240 + gateway 192.168.42.1 + dns-nameservers 8.8.8.8 8.8.4.4 + dns-domain lab.example.org + + # Public network + auto cloudbr0 + iface cloudbr0 inet manual + bridge_ports eth0.200 + bridge_fd 5 + bridge_stp off + bridge_maxwait 1 + + # Private network + auto cloudbr1 + iface cloudbr1 inet manual + bridge_ports eth0.300 + bridge_fd 5 + bridge_stp off + bridge_maxwait 1 + + +you would change the configuration similar to below. + +:: + + auto lo + iface lo inet loopback + + # The primary network interface + auto eth0.100 + iface eth0.100 inet static + address 192.168.42.11 + netmask 255.255.255.240 + gateway 192.168.42.1 + dns-nameservers 8.8.8.8 8.8.4.4 + dns-domain lab.example.org + + # Public network + auto cloudbr0 + iface cloudbr0 inet manual + bridge_ports eth0.200 + bridge_fd 5 + bridge_stp off + bridge_maxwait 1 + + # Private network + auto cloudbr1 + iface cloudbr1 inet static + addres 192.0.2.X + netmask 255.255.255.0 + bridge_ports eth0.300 + bridge_fd 5 + bridge_stp off + bridge_maxwait 1 + + +Configure iptables to pass XVLAN packets +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Since VXLAN uses UDP packet to forward encapsulated the L2 frames, +UDP/8472 port must be opened. + +Configure in RHEL or CentOS +''''''''''''''''''''''''''' + +RHEL and CentOS use iptables for firewalling the system, you can open +extra ports by executing the following iptable commands: + +:: + + $ sudo iptables -I INPUT -p udp -m udp --dport 8472 -j ACCEPT + + +These iptable settings are not persistent accross reboots, we have to +save them first. + +:: + + $ sudo iptables-save > /etc/sysconfig/iptables + + +With this configuration you should be able to restart the network, +although a reboot is recommended to see if everything works properly. + +:: + + $ sudo service network restart + $ sudo reboot + + +.. warning:: Make sure you have an alternative way like IPMI or ILO to reach the machine in case you made a configuration error and the network stops functioning! + +Configure in Ubuntu +''''''''''''''''''' + +The default firewall under Ubuntu is UFW (Uncomplicated FireWall), which +is a Python wrapper around iptables. + +To open the required ports, execute the following commands: + +:: + + $ sudo ufw allow proto udp from any to any port 8472 + + +.. note:: By default UFW is not enabled on Ubuntu. Executing these commands with the firewall disabled does not enable the firewall. + +With this configuration you should be able to restart the network, +although a reboot is recommended to see if everything works properly. + +:: + + $ sudo service networking restart + $ sudo reboot + + +.. warning:: Make sure you have an alternative way like IPMI or ILO to reach the machine in case you made a configuration error and the network stops functioning! + +Setup zone using VXLAN +~~~~~~~~~~~~~~~~~~~~~~ + +In almost all parts of zone setup, you can just follow the advanced zone +setup istruction in "PRODUCT Installation Guide" to use this plugin. It +is not required to add a network element nor to reconfigure the network +offering. The only thing you have to do is configure the physical +network to use VXLAN as the isolation method for Guest Network. + +Configure the physical network +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. figure:: /_static/images/vxlan-physicalnetwork.png + +CloudStack needs to have one physical network for Guest Traffic with the +isolation method set to "VXLAN". + +.. figure:: /_static/images/vxlan-trafficlabel.png + +Guest Network traffic label should be the name of the physical interface +or the name of the bridge interface and the bridge interface and they +should have an IPv4 address. See ? for details. + +Configure the guest traffic +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. figure:: /_static/images/vxlan-vniconfig.png + +Specify a range of VNIs you would like to use for carrying guest network +traffic. + +.. warning:: VNI must be unique per zone and no duplicate VNIs can exist in the zone. Exercise care when designing your VNI allocation policy. + +
