Hi Bjoern, Thanks for the feedback. Let me try to get my head around what you explained and I will get back to you later...
Thanks, -----Original Message----- From: Bjoern Teipel [mailto:bjoern.tei...@internetbrands.com] Sent: 16 March 2013 06:08 To: Valery Fongang Cc: 'Ahmad Emneina'; 'Marcus Sorensen'; 'Edison Su'; 'Cloudstack users' Subject: Re: NIC / VLAN ISSUE ON KVM HOST Hi Valery, here is the network layout I did (Advanced, shared network) : KVM : bond0 (also eth0 or any other Ethernet device works) + +----- bond0.x (x = Network VLANID ) -----> cloud-mgmt (Linux bridge) + +----- bond0.y (another VLAN) ------- > cloud-storage (Linux bridge) + +----- cloudbr0 (Linux Bridge, just a helper) CSTK : Management using traffic/KVM label : cloud-mgmt (the Linux interface name, untagged (no tag support anyway in CSTK)) Storage using traffic label : cloud-storage (untagged) Guest Network using traffic label : cloudbr0 with VLANID A Public Network using traffic label : cloudbr0 with VLANID B The magic begins with cloudbr0, because CSTK finds out the physical interface (bond0 here) and creates new VLANs A and B to new bridges cloudVirBrA and cloudVirBrB. Everything happens automatically so you only have to create the management ,storage and cloud0 bridges. Here a real "brctl show" sample from my network: cloud-mgmt 8000.00219b937e8c no bond0.2610 cloud-storage 8000.00219b937e8c no bond0.2578 cloudVirBr2562 8000.00219b937e8c no bond0.2562 cloudVirBr2615 8000.00219b937e8c no bond0.2615 cloudbr0 8000.00219b937e8c no bond0 If you cloud0 is a bridge which connects the local link interface of the VM with the hypervisor. Also automatically created by CSTK You can also check out https://cwiki.apache.org/confluence/display/CLOUDSTACK/Cloud+Operators there are some nice examples shown for different network designs. If you are not familiar with all that routing and VLAN and bridging stuff I can send you some examples. Have fun, Bjoern On 3/14/2013 5:59 AM, Valery Fongang wrote: > Hi Bjoern, > > Thanks for your reply (and sorry for my late response) but I would like more > indication on how to do this: "Bridge on VLAN and don't use VLAN ID. That's > the combination I choose because I had trouble with the first solution.". > > My main goal is to implement Cloudstack with advanced networking with VLANs > and I have a single NIC on my host with no intention of separating the > Storage LAN. As you may have mentioned yes I have a physical network > declarartion in CloudStack Management interface a range of VLAN specified. > I'm actually not too sure of what I'm doing apart from trying to get all this > to work with VLANs. I will be very happy if you can give me a detailed config > example for my Host NIC and Network in general. > > > > > -----Original Message----- > From: Bjoern Teipel [mailto:bjoern.tei...@internetbrands.com] > Sent: 11 March 2013 19:36 > To: Ahmad Emneina > Cc: Valery Fongang; Marcus Sorensen; Edison Su; Cloudstack users > Subject: Re: NIC / VLAN ISSUE ON KVM HOST > > Looks like you specified a network in CS, probably the storage, using > a VLAN ID (VID) but you have the vlan ID already bound to a cloudbr > bridge. You can only do either one, > > Bridge on native ETH/bond interface and use Vlan ID in CS. CS will do > the rest. I had problems using a VID on the storage network > > Or > > Bridge on VLAN and don't use VLAN ID. That's the combination I choose because > I had trouble with the first solution. > > Guest and public networks worked fine for me using VID in CS. > > Bjoern > > Ahmad Emneina <aemne...@gmail.com> wrote: > >> adding the KVM gurus. Edison and Marcus. They are heavily active on >> the dev list relating to KVM. >> >> >> On Mon, Mar 11, 2013 at 3:16 AM, Valery Fongang < >> fongang.val...@pertechtual.net> wrote: >> >>> Hi, >>> >>> I have some issues sorting out how I need to set my NIC on my KVM >>> Host for CloudStack to work correctly. Here is my setup: >>> >>> - 1 KVM (Centos 6.3) host with a single NIC and I am using >>> SharedMointPoint option for my Primary Storage to leverage Local >>> Storage on this server. >>> - 1 VM acting as CMS on a different Physical Server. NFS Secondary >>> Storage is shared from this VM. >>> >>> My Network settings on the KVM host are as follow and at the bottom >>> my CMS log. How am I supposed to set my network to prevent having >>> the error in this log? >>> >>> #######################Network Config ########################### >>> >>> cloud0 Link encap:Ethernet HWaddr 76:D4:8A:DB:9C:B3 >>> inet addr:169.254.0.1 Bcast:169.254.255.255 Mask:255.255.0.0 >>> inet6 addr: fe80::74d4:8aff:fedb:9cb3/64 Scope:Link >>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >>> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >>> TX packets:6 errors:0 dropped:0 overruns:0 carrier:0 >>> collisions:0 txqueuelen:0 >>> RX bytes:0 (0.0 b) TX bytes:468 (468.0 b) >>> >>> cloudbr0 Link encap:Ethernet HWaddr C8:60:00:58:9F:13 >>> inet addr:10.26.0.143 Bcast:10.26.255.255 Mask:255.255.0.0 >>> inet6 addr: fe80::ca60:ff:fe58:9f13/64 Scope:Link >>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >>> RX packets:3215 errors:0 dropped:0 overruns:0 frame:0 >>> TX packets:2148 errors:0 dropped:0 overruns:0 carrier:0 >>> collisions:0 txqueuelen:0 >>> RX bytes:318720 (311.2 KiB) TX bytes:1490094 (1.4 MiB) >>> >>> cloudbr1 Link encap:Ethernet HWaddr C8:60:00:58:9F:13 >>> inet6 addr: fe80::ca60:ff:fe58:9f13/64 Scope:Link >>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >>> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >>> TX packets:7 errors:0 dropped:0 overruns:0 carrier:0 >>> collisions:0 txqueuelen:0 >>> RX bytes:0 (0.0 b) TX bytes:578 (578.0 b) >>> >>> eth0 Link encap:Ethernet HWaddr C8:60:00:58:9F:13 >>> inet6 addr: fe80::ca60:ff:fe58:9f13/64 Scope:Link >>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >>> RX packets:5783 errors:0 dropped:0 overruns:0 frame:0 >>> TX packets:7097 errors:0 dropped:0 overruns:0 carrier:0 >>> collisions:0 txqueuelen:1000 >>> RX bytes:640522 (625.5 KiB) TX bytes:1838861 (1.7 MiB) >>> Interrupt:34 >>> >>> eth0.1 Link encap:Ethernet HWaddr C8:60:00:58:9F:13 >>> inet6 addr: fe80::ca60:ff:fe58:9f13/64 Scope:Link >>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >>> RX packets:4798 errors:0 dropped:0 overruns:0 frame:0 >>> TX packets:3310 errors:0 dropped:6 overruns:0 carrier:0 >>> collisions:0 txqueuelen:0 >>> RX bytes:514130 (502.0 KiB) TX bytes:1551030 (1.4 MiB) >>> >>> eth0.200 Link encap:Ethernet HWaddr C8:60:00:58:9F:13 >>> inet6 addr: fe80::ca60:ff:fe58:9f13/64 Scope:Link >>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >>> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >>> TX packets:2601 errors:0 dropped:7 overruns:0 carrier:0 >>> collisions:0 txqueuelen:0 >>> RX bytes:0 (0.0 b) TX bytes:225281 (220.0 KiB) >>> >>> eth0.300 Link encap:Ethernet HWaddr C8:60:00:58:9F:13 >>> inet6 addr: fe80::ca60:ff:fe58:9f13/64 Scope:Link >>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >>> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >>> TX packets:1179 errors:0 dropped:3 overruns:0 carrier:0 >>> collisions:0 txqueuelen:0 >>> RX bytes:0 (0.0 b) TX bytes:61972 (60.5 KiB) >>> >>> lo Link encap:Local Loopback >>> inet addr:127.0.0.1 Mask:255.0.0.0 >>> inet6 addr: ::1/128 Scope:Host >>> UP LOOPBACK RUNNING MTU:16436 Metric:1 >>> RX packets:2 errors:0 dropped:0 overruns:0 frame:0 >>> TX packets:2 errors:0 dropped:0 overruns:0 carrier:0 >>> collisions:0 txqueuelen:0 >>> RX bytes:190 (190.0 b) TX bytes:190 (190.0 b) >>> >>> virbr0 Link encap:Ethernet HWaddr 52:54:00:93:DC:18 >>> inet addr:192.168.122.1 Bcast:192.168.122.255 >>> Mask:255.255.255.0 >>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >>> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >>> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 >>> collisions:0 txqueuelen:0 >>> RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) >>> ###################### brctl show ###################### [root@kvm >>> home]# brctl show >>> bridge name bridge id STP enabled interfaces >>> cloud0 8000.000000000000 no >>> cloudVirBr1 8000.000000000000 no >>> cloudbr0 8000.c86000589f13 yes eth0.1 >>> eth0.200 >>> cloudbr1 8000.c86000589f13 yes eth0.300 >>> virbr0 8000.52540093dc18 yes virbr0-nic >>> >>> >>> ##################### Recent Error Log################ >>> >>> 2013-03-11 09:56:17,421 DEBUG [cloud.alert.AlertManagerImpl] >>> (catalina-exec-11:null) Done executing storage capacity update >>> 2013-03-11 09:56:17,421 DEBUG [cloud.alert.AlertManagerImpl] >>> (catalina-exec-11:null) Executing capacity updates for public ip and >>> Vlans >>> 2013-03-11 09:56:17,429 DEBUG [cloud.alert.AlertManagerImpl] >>> (catalina-exec-11:null) Done capacity updates for public ip and >>> Vlans >>> 2013-03-11 09:56:17,429 DEBUG [cloud.alert.AlertManagerImpl] >>> (catalina-exec-11:null) Executing capacity updates for private ip >>> 2013-03-11 09:56:17,432 DEBUG [cloud.alert.AlertManagerImpl] >>> (catalina-exec-11:null) Done executing capacity updates for private >>> ip >>> 2013-03-11 09:56:17,432 DEBUG [cloud.alert.AlertManagerImpl] >>> (catalina-exec-11:null) Done recalculating system capacity >>> 2013-03-11 09:56:21,888 DEBUG [cloud.server.StatsCollector] >>> (StatsCollector-2:null) VmStatsCollector is running... >>> 2013-03-11 09:56:23,743 DEBUG [agent.transport.Request] >>> (AgentManager-Handler-5:null) Seq 5-1100559375: Processing: { Ans: >>> , >>> MgmtId: 16279293006507, via: 5, Ver: v1, Flags: 110, >>> [{"StartAnswer":{"vm":{"id":16209,"name":"s-16209-VM","type":"Second >>> a >>> ryStorageVm","cpus":1,"speed":500,"minRam":268435456,"maxRam":268435 >>> 4 56,"arch":"x86_64","os":"Debian GNU/Linux 5.0 >>> (32-bit)","bootArgs":" >>> template=domP type=secstorage >>> host=10.26.0.90 port=8250 name=s-16209-VM zone=4 pod=4 >>> guid=s-16209-VM >>> resource=com.cloud.storage.resource.PremiumSecondaryStorageResource >>> instance=SecStorage sslcopy=true role=templateProcessor mtu=1500 >>> eth2ip=172.18.1.15 eth2mask=255.255.0.0 eth2mask=255.255.0.0 >>> gateway=172.18.1.1 public.network.device=eth2 eth0ip=169.254.1.144 >>> eth0mask=255.255.0.0 eth0mask=255.255.0.0 eth1ip=10.26.0.136 >>> eth1mask=255.255.0.0 eth1mask=255.255.0.0 >>> mgmtcidr=10.26.0.0/16localgw=10.26.0.1 private.network.device=eth1 >>> eth3ip=10.26.0.134 >>> eth3mask=255.255.0.0 eth3mask=255.255.0.0 storageip=10.26.0.134 >>> storagenetmask=255.255.0.0 storagegateway=10.26.0.1 >>> internaldns1=10.26.0.160 internaldns2=8.8.8.8 dns1=10.26.0.160 >>> dns2=8.8.8.8","rebootOnCrash":false,"enableHA":false,"limitCpuUse":f >>> a >>> lse,"vncPassword":"be60d6697a342397","params":{},"disks":[{"id":1620 >>> 9 >>> ,"name":"/home","mountPoint":"efd20cac-c832-4a57-88bf-ff939e25943b"," >>> path":"efd20cac-c832-4a57-88bf-ff939e25943b","size":139264,"type":"R >>> O >>> OT","storagePoolType":"SharedMountPoint","storagePoolUuid":"390801ff >>> - >>> 45b9-4177-9620-b863a3358b2d","deviceId":0}],"nics":[{"deviceId":2,"n >>> e >>> tworkRateMbps":-1,"defaultNic":true,"ip":"172.18.1.15","netmask":"25 >>> 5 >>> .255.0.0","gateway":"172.18.1.1","mac":"06:07:fc:00:00:11","dns1":"1 >>> 0 >>> .26.0.160","dns2":"8.8.8.8","broadcastType":"Vlan","type":"Public"," >>> b >>> roadcastUri":"vlan://1","isolationUri":"vlan://1","isSecurityGroupEn >>> a >>> bled":false},{"deviceId":0,"networkRateMbps":-1,"defaultNic":false," >>> i >>> p":"169.254.1.144","netmask":"255.255.0.0","gateway":"169.254.0.1"," >>> m >>> ac":"0e:00:a9:fe:01:90","broadcastType":"LinkLocal","type":"Control" >>> , >>> "isSecurityGroupEnabled":false},{"deviceId":1,"networkRateMbps":-1," >>> d >>> efaultNic":false,"ip":"10.26.0.136","netmask":"255.255.0.0","gateway" >>> :"10.26.0.1","mac":"06:2d:d8:00:00:07","broadcastType":"Native","typ >>> e >>> ":"Management","isSecurityGroupEnabled":false},{"deviceId":3,"networ >>> k >>> RateMbps":-1,"defaultNic":false,"ip":"10.26.0.134","netmask":"255.25 >>> 5 >>> .0.0","gateway":"10.26.0.1","mac":"06:a5:3e:00:00:05","broadcastType" >>> :"Native","type":"Storage","isSecurityGroupEnabled":false}]},"result" >>> :false,"details":"Failed to create vnet 1: Set name-type for VLAN >>> subsystem. Should be visible in /proc/net/vlan/configdevice eth0.1 >>> is already a member of a bridge; can't enslave it to bridge >>> cloudVirBr1.Failed to add vlan: eth0.1 to >>> cloudVirBr1","wait":0}},{"Answer":{"result":false,"details":"Stopped >>> by previous failure","wait":0}}] } >>> 2013-03-11 09:56:23,743 DEBUG [agent.transport.Request] >>> (secstorage-1:null) Seq 5-1100559375: Received: { Ans: , MgmtId: >>> 16279293006507, via: 5, Ver: v1, Flags: 110, { StartAnswer, Answer } >>> } >>> 2013-03-11 09:56:23,743 WARN [cloud.vm.VirtualMachineManagerImpl] >>> (secstorage-1:null) Cleanup failed due to Failed to create vnet 1: >>> Set name-type for VLAN subsystem. Should be visible in >>> /proc/net/vlan/configdevice eth0.1 is already a member of a bridge; >>> can't enslave it to bridge cloudVirBr1.Failed to add vlan: eth0.1 to >>> cloudVirBr1 >>> 2013-03-11 09:56:23,743 WARN [cloud.vm.VirtualMachineManagerImpl] >>> (secstorage-1:null) Cleanup failed due to Stopped by previous >>> failure >>> 2013-03-11 09:56:23,744 WARN [cloud.vm.VirtualMachineManagerImpl] >>> (AgentManager-Handler-5:null) Cleanup failed due to Failed to create >>> vnet >>> 1: Set name-type for VLAN subsystem. Should be visible in >>> /proc/net/vlan/configdevice eth0.1 is already a member of a bridge; >>> can't enslave it to bridge cloudVirBr1.Failed to add vlan: eth0.1 to >>> cloudVirBr1 >>> 2013-03-11 09:56:23,744 WARN [cloud.vm.VirtualMachineManagerImpl] >>> (AgentManager-Handler-5:null) Cleanup failed due to Stopped by >>> previous failure >>> 2013-03-11 09:56:23,744 DEBUG [agent.manager.AgentAttache] >>> (AgentManager-Handler-5:null) Seq 5-1100559376: Sending now. is >>> current sequence. >>> 2013-03-11 09:56:23,750 INFO [cloud.vm.VirtualMachineManagerImpl] >>> (secstorage-1:null) Unable to start VM on Host[-5-Routing] due to >>> Failed to create vnet 1: Set name-type for VLAN subsystem. Should be >>> visible in /proc/net/vlan/configdevice eth0.1 is already a member of >>> a bridge; can't enslave it to bridge cloudVirBr1.Failed to add vlan: >>> eth0.1 to cloudVirBr1 >>> 2013-03-11 09:56:23,758 DEBUG [cloud.vm.VirtualMachineManagerImpl] >>> (secstorage-1:null) Cleaning up resources for the vm >>> VM[SecondaryStorageVm|s-16209-VM] in Starting state >>> 2013-03-11 09:56:23,760 DEBUG [agent.transport.Request] >>> (secstorage-1:null) Seq 5-1100559384: Waiting for Seq 1100559376 >>> Scheduling: { Cmd , MgmtId: 16279293006507, via: 5, Ver: v1, Flags: >>> 100111, >>> [{"StopCommand":{"isProxy":false,"vmName":"s-16209-VM","wait":0}}] } >>> 2013-03-11 09:56:24,213 DEBUG >>> [network.router.VirtualNetworkApplianceManagerImpl] >>> (RouterStatusMonitor-1:null) Found 0 routers. >>> 2013-03-11 09:56:25,364 DEBUG [agent.manager.AgentManagerImpl] >>> (AgentManager-Handler-6:null) Ping from 5 >>> 2013-03-11 09:56:37,004 DEBUG [cloud.server.StatsCollector] >>> (StatsCollector-2:null) HostStatsCollector is running... >>> 2013-03-11 09:56:37,598 DEBUG [cloud.vm.VirtualMachineManagerImpl] >>> (AgentManager-Handler-7:null) Cleanup succeeded. Details null >>> 2013-03-11 09:56:37,598 DEBUG [agent.transport.Request] >>> (StatsCollector-2:null) Seq 5-1100559385: Received: { Ans: , MgmtId: >>> 16279293006507, via: 5, Ver: v1, Flags: 10, { GetHostStatsAnswer } } >>> >>> >>>
