Marcus,

I setup a small PowerDNS recursor on 192.168.56.15, configured the DNS for the 
management network to use it, and the route table in the SSVM is now correct.  
However, this behavior does not seem correct.  At a minimum, it violates the 
rule of least surprise.  CloudStack shouldn't be adding gateways that are not 
configured.  Therefore, I have entered a defect[1] to remove the behavior.

With the route table fixed, I am now experiencing a new problem.  The external 
NIC (10.0.3.0/24) on the SSVM is being connected to the internal NIC 
(192.168.56.0/24) on the host.  The host-only network (192.168.56.15) is 
configured on xenbr0 and the NAT network is configured on xenbr1.  As a 
reference, the following is the contents of the /etc/network/interfaces file 
and ifconfig from devcloud host:

root@zone1:/opt/cloudstack/apache-tomcat-6.0.32/bin# cat /etc/network/interfaces
# The loopback network interface
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual

allow-hotplug eth1
iface eth1 inet manual

# The primary network interface
auto xenbr0
iface xenbr0 inet static
  address 192.168.56.15
  netmask 255.255.255.0
  network 192.168.56.0
  broadcast 192.168.56.255
  dns_nameserver 192.168.56.15
  bridge_ports eth0

auto xenbr1
iface xenbr1 inet dhcp
  bridge_ports eth1
  dns_nameserver 8.8.8.8 8.8.4.4
  post-up route add default gw 10.0.3.2

root@zone1:/opt/cloudstack/apache-tomcat-6.0.32/bin# ifconfig
eth0      Link encap:Ethernet  HWaddr 08:00:27:7e:74:9c  
          inet6 addr: fe80::a00:27ff:fe7e:749c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:777 errors:0 dropped:0 overruns:0 frame:0
          TX packets:188 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:109977 (109.9 KB)  TX bytes:11900 (11.9 KB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:df:00:00  
          inet6 addr: fe80::a00:27ff:fedf:0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4129 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3910 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:478719 (478.7 KB)  TX bytes:2542459 (2.5 MB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:360285 errors:0 dropped:0 overruns:0 frame:0
          TX packets:360285 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:169128181 (169.1 MB)  TX bytes:169128181 (169.1 MB)

vif1.0    Link encap:Ethernet  HWaddr fe:ff:ff:ff:ff:ff  
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP PROMISC  MTU:1500  Metric:1
          RX packets:6 errors:0 dropped:0 overruns:0 frame:0
          TX packets:152 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:32 
          RX bytes:292 (292.0 B)  TX bytes:9252 (9.2 KB)

vif1.1    Link encap:Ethernet  HWaddr fe:ff:ff:ff:ff:ff  
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP PROMISC  MTU:1500  Metric:1
          RX packets:566 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1405 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:32 
          RX bytes:44227 (44.2 KB)  TX bytes:173995 (173.9 KB)

vif1.2    Link encap:Ethernet  HWaddr fe:ff:ff:ff:ff:ff  
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP PROMISC  MTU:1500  Metric:1
          RX packets:3 errors:0 dropped:0 overruns:0 frame:0
          TX packets:838 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:32 
          RX bytes:84 (84.0 B)  TX bytes:111361 (111.3 KB)

vif4.0    Link encap:Ethernet  HWaddr fe:ff:ff:ff:ff:ff  
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP PROMISC  MTU:1500  Metric:1
          RX packets:64 errors:0 dropped:0 overruns:0 frame:0
          TX packets:197 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:32 
          RX bytes:10276 (10.2 KB)  TX bytes:18453 (18.4 KB)

vif4.1    Link encap:Ethernet  HWaddr fe:ff:ff:ff:ff:ff  
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP PROMISC  MTU:1500  Metric:1
          RX packets:2051 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2446 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:32 
          RX bytes:233914 (233.9 KB)  TX bytes:364243 (364.2 KB)

vif4.2    Link encap:Ethernet  HWaddr fe:ff:ff:ff:ff:ff  
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP PROMISC  MTU:1500  Metric:1
          RX packets:3 errors:0 dropped:0 overruns:0 frame:0
          TX packets:582 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:32 
          RX bytes:84 (84.0 B)  TX bytes:74700 (74.7 KB)

vif4.3    Link encap:Ethernet  HWaddr fe:ff:ff:ff:ff:ff  
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING NOARP PROMISC  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:585 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:32 
          RX bytes:0 (0.0 B)  TX bytes:74826 (74.8 KB)

xapi0     Link encap:Ethernet  HWaddr fe:ff:ff:ff:ff:ff  
          inet addr:169.254.0.1  Bcast:169.254.255.255  Mask:255.255.0.0
          inet6 addr: fe80::c870:1aff:fec2:22b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:568 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1132 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:76284 (76.2 KB)  TX bytes:109085 (109.0 KB)

xenbr0    Link encap:Ethernet  HWaddr 08:00:27:7e:74:9c  
          inet addr:192.168.56.15  Bcast:192.168.56.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe7e:749c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4162 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3281 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:469199 (469.1 KB)  TX bytes:485688 (485.6 KB)

xenbr1    Link encap:Ethernet  HWaddr 08:00:27:df:00:00  
          inet addr:10.0.3.15  Bcast:10.0.3.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fedf:0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4129 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3114 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:404327 (404.3 KB)  TX bytes:2501443 (2.5 MB)

These physical NICs on the host translate to the following Xen PIFs:

root@zone1:/opt/cloudstack/apache-tomcat-6.0.32/bin# xe pif-list
uuid ( RO)                  : 207413c9-5058-7a40-6c96-2dab21057f30
                device ( RO): eth1
    currently-attached ( RO): true
                  VLAN ( RO): -1
          network-uuid ( RO): 1679ddb1-5a21-b827-ab07-c16275d5ce72


uuid ( RO)                  : c0274787-e768-506f-3191-f0ac17b0c72b
                device ( RO): eth0
    currently-attached ( RO): true
                  VLAN ( RO): -1
          network-uuid ( RO): 8ee927b1-a35d-ac10-4471-d7a6a475839a

The following is the ifconfig from the SSVM:

root@s-5-TEST:~# ifconfig
eth0      Link encap:Ethernet  HWaddr 0e:00:a9:fe:03:8b  
          inet addr:169.254.3.139  Bcast:169.254.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:235 errors:0 dropped:0 overruns:0 frame:0
          TX packets:92 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:21966 (21.4 KiB)  TX bytes:16404 (16.0 KiB)
          Interrupt:8 

eth1      Link encap:Ethernet  HWaddr 06:bc:62:00:00:05  
          inet addr:192.168.56.104  Bcast:192.168.56.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2532 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2127 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:341242 (333.2 KiB)  TX bytes:272183 (265.8 KiB)
          Interrupt:10 

eth2      Link encap:Ethernet  HWaddr 06:12:72:00:00:37  
          inet addr:10.0.3.204  Bcast:10.0.3.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:600 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:68648 (67.0 KiB)  TX bytes:126 (126.0 B)
          Interrupt:11 

eth3      Link encap:Ethernet  HWaddr 06:25:e2:00:00:15  
          inet addr:192.168.56.120  Bcast:192.168.56.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:603 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:68732 (67.1 KiB)  TX bytes:0 (0.0 B)
          Interrupt:12 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:61 errors:0 dropped:0 overruns:0 frame:0
          TX packets:61 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:5300 (5.1 KiB)  TX bytes:5300 (5.1 KiB)

Finally, the following are the vif params for the eth2 device on the SSVM 
depicting its connection to eth0 instead of eth1:

root@zone1:/opt/cloudstack/apache-tomcat-6.0.32/bin# !1243
xe vif-param-list uuid=be44bb30-5700-b461-760e-10fe93079210
uuid ( RO)                        : be44bb30-5700-b461-760e-10fe93079210
                     vm-uuid ( RO): 7958d91f-e52d-a25d-718c-7f831ae701d7
               vm-name-label ( RO): s-5-TEST
          allowed-operations (SRO): attach; unplug_force; unplug
          current-operations (SRO): 
                      device ( RO): 2
                         MAC ( RO): 06:12:72:00:00:37
           MAC-autogenerated ( RO): false
                         MTU ( RO): 1500
          currently-attached ( RO): true
          qos_algorithm_type ( RW): ratelimit
        qos_algorithm_params (MRW): kbps: 25600
    qos_supported_algorithms (SRO): 
                other-config (MRW): nicira-iface-id: 
3d68b9f8-98d1-4ac7-92d8-fb57cb8b0adc; nicira-vm-id: 
7958d91f-e52d-a25d-718c-7f831ae701d7
                network-uuid ( RO): 8ee927b1-a35d-ac10-4471-d7a6a475839a
          network-name-label ( RO): Pool-wide network associated with eth0
                 io_read_kbs ( RO): 0.007
                io_write_kbs ( RO): 0.000

How do I configure CloudStack such that the guest network NIC on the VM will be 
connected to correct physical NIC?

Thanks for your help,
-John


[1]: https://issues.apache.org/jira/browse/CLOUDSTACK-590

On Dec 5, 2012, at 2:47 PM, Marcus Sorensen <shadow...@gmail.com> wrote:

> Yes, see your cmdline. internaldns1=10.0.3.2, so it is forcing the use of
> management network to route to 10.0.3.2 for DNS. that's where the route is
> coming from. you will want to use something on your management net for
> internal DNS, or something other than that router.
> 
> 
> On Wed, Dec 5, 2012 at 11:59 AM, John Burwell <jburw...@basho.com> wrote:
> 
>> Anthony,
>> 
>> I apologize for forgetting to response to the part of your answer the
>> first part of the question.  I had set the management.network.cidr and host
>> global settings to 192.168.0.0/24 and 192.168.56.18 respectively.  Please
>> see the zone1.devcloud.cfg Marvin configuration attached to my original
>> email for the actual setting, as well as, the network configurations used
>> when this problem occurs.
>> 
>> Thanks,
>> -John
>> 
>> On Dec 5, 2012, at 12:46 PM, Anthony Xu <xuefei...@citrix.com> wrote:
>> 
>>> Hi join,
>>> 
>>> Try following,
>>> 
>>> Set global configuration management.network.cidr to your management
>> server CIDR, if this configuration is not available in UI, you can change
>> it in DB directly.
>>> 
>>> Restart management,
>>> Stop/Start SSVM and CPVM.
>>> 
>>> 
>>> And could you post "cat /proc/cmdline" in SSVM?
>>> 
>>> 
>>> 
>>> Anthony
>>> 
>>>> -----Original Message-----
>>>> From: John Burwell [mailto:jburw...@basho.com]
>>>> Sent: Wednesday, December 05, 2012 9:11 AM
>>>> To: cloudstack-dev@incubator.apache.org
>>>> Subject: Re: SSVM Network Configuration Issue
>>>> 
>>>> All,
>>>> 
>>>> I was wondering if anyone else is experiencing this problem when using
>>>> secondary storage on a devcloud-style VM with a host-only and NAT
>>>> adapter.  One aspect of this issue that seems interesting is that
>>>> following route table from the SSVM:
>>>> 
>>>> root@s-5-TEST:~# route
>>>> Kernel IP routing table
>>>> Destination     Gateway         Genmask         Flags Metric Ref    Use
>>>> Iface
>>>> 10.0.3.2        192.168.56.1    255.255.255.255 UGH   0      0        0
>>>> eth1
>>>> 10.0.3.0        *               255.255.255.0   U     0      0        0
>>>> eth2
>>>> 192.168.56.0    *               255.255.255.0   U     0      0        0
>>>> eth1
>>>> 192.168.56.0    *               255.255.255.0   U     0      0        0
>>>> eth3
>>>> link-local      *               255.255.0.0     U     0      0        0
>>>> eth0
>>>> default         10.0.3.2        0.0.0.0         UG    0      0        0
>>>> eth2
>>>> 
>>>> In particular, the gateways for the management and guest networks do
>>>> not match to the configuration provided to the management server (i.e.
>>>> 10.0.3.2 is the gateway for the 10.0.3.0/24 network and 192.168.56.1 is
>>>> the gateway for the 192.168.56.0/24 network).  With this configuration,
>>>> the SSVM has a socket connection to the management server, but is in
>>>> alert state.  Finally, when I remove the host-only NIC and use only a
>>>> NAT adapter the SSVM's networking works as expecting leading me to
>>>> believe that the segregated network configuration is at the root of the
>>>> problem.
>>>> 
>>>> Until I can get the networking on the SSVM configured, I am unable to
>>>> complete the testing of the S3-backed Secondary Storage enhancement.
>>>> 
>>>> Thank you for your help,
>>>> -John
>>>> 
>>>> On Dec 3, 2012, at 4:46 PM, John Burwell <jburw...@basho.com> wrote:
>>>> 
>>>>> All,
>>>>> 
>>>>> I am setting up a multi-zone devcloud configuration on VirtualBox
>>>> 4.2.4 using the Ubuntu 12.04.1 and Xen 4.1.  I have configured the base
>>>> management server VM (zone1) to serve as both the zone1, as well as,
>>>> the management server (running MySql) with eth0 as a host-only adapter
>>>> and a static IP of 192.168.56.15 and eth1 as a NAT adapter (see the
>>>> attached zone1-interfaces file for the exact network configuration on
>>>> the VM).  The management and guest networks are configured as follows:
>>>>> 
>>>>> Zone 1
>>>>> Management: 192.168.56.100-149 gw 192.168.56.1 dns 10.0.3.2 (?)
>>>>> Guest: 10.0.3.200-10.0.3.220 gw 10.0.3.2 dns 8.8.8.8
>>>>> Zone 2
>>>>> Management: 192.168.56.150-200 gw 192.68.56.1 dns 10.0.3.2 (?)
>>>>> Guest: 10.0.3.221-240 gw 10.0.3.2 dns 8.8.8.8
>>>>> 
>>>>> The management server deploys and starts without error.  I then
>>>> populate the configuration it using the attached Marvin configuration
>>>> file (zone1.devcloud.cfg) and restart the management server in order to
>>>> allow the global configuration option changes to take effect.
>>>> Following the restart, the CPVM and SSVM start without error.
>>>> Unfortunately, they drop into alert status, and the SSVM is unable to
>>>> connect outbound through the guest network (very important for my tests
>>>> because I am testing S3-backed secondary storage).
>>>>> 
>>>>> From the diagnostic checks I have performed on the management server
>>>> and the SSVM, it appears that the daemon on the SSVM is connecting back
>>>> to the management server.  I have attached a set of diagnostic
>>>> information from the management server (mgmtsvr-zone1-diagnostics.log)
>>>> and SSVM server (ssvm-zone1-diagnostics.log) that includes the results
>>>> of ifconfig, route, netstat and ping checks, as well as, other
>>>> information (e.g. the contents of /var/cache/cloud/cmdline on the SSVM).
>>>> Finally, I have attached the vmops log from the management server
>>>> (vmops-zone1.log).
>>>>> 
>>>>> What changes need to be made to management server configuration in
>>>> order to start up an SSVM that can communicate with the secondary
>>>> storage NFS volumes, management server, and connect to hosts on the
>>>> Internet?
>>>>> 
>>>>> Thanks for your help,
>>>>> -John
>>>>> 
>>>>> <ssvm-zone1-diagnostics.log>
>>>>> <vmops-zone1.tar.gz>
>>>>> <mgmtsvr-zone1-diagnostics.log>
>>>>> <zone1-interfaces>
>>>>> <zone1.devcloud.cfg>
>>> 
>> 
>> 

Reply via email to