Hi, I looked up the startup command of the old router instance which worked correctly:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.sh -l r-7-VM -t all -d /var/lib/libvirt/images/r-7-VM-patchdisk -p %template=domP%name=r-7-VM%eth2ip=10.193.17.190%eth2mask=255.255.255.0%gateway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=cs6cloud.internal%dhcprange=10.124.99.1%eth1ip=169.254.2.46%eth1mask=255.255.0.0%type=router%disable_rp_filter=true%dns1=10.193.17.1 The new command (cf. below) doesn't have the parameters eth1ip and eth1mask. Thanks Kambiz Alena Prokharchyk <alena.prokharc...@citrix.com> wrote: > > I don’t think its relevant as the piece we’ve fixed, just eliminated > static nat rule programming for non-existing vm. Missing netmask on eth1 > doesn’t seem related to the problem (although we have to figure out why > its missing), as the connection that fails, happening to link local 169.x > eth0 interface. > > Edison, can you please tell us how to debug link local connection failure, > on KVM agent? > > Thank you, > Alena. > > On 3/24/14, 1:47 PM, "Kambiz Darabi" <dar...@m-creations.com> wrote: > >>Hi, >> >>thank you, the NullPointerException doesn't occur any more, but there >>still seems to be a problem during startup of the router. >> >>When I start the virtual router, it comes up, but in agent.log, there >>are lots of 'Could not connect to 169.254.1.x' messages. >> >>Then I logged into the virtual router to find out that the netmask of >>eth1 is missing in the interfaces file: >> >>root@host:~# virsh console r-19-VM >>Connected to domain r-19-VM >>Escape character is ^] >> >>Debian GNU/Linux 6.0 r-19-VM ttyS0 >> >>r-19-VM login: root >>... >>root@r-19-VM:~# cat /etc/network/interfaces >>auto lo eth0 eth1 eth2 >>iface lo inet loopback >> >>iface eth0 inet static >> address 169.254.1.59 >> netmask 255.255.0.0 >>iface eth1 inet static >> address 10.193.17.1 >> netmask >>iface eth2 inet static >> address 10.193.17.190 >> netmask 255.255.255.0 >> >>I don't know if it is relevant, but this is the line from agent.log >>where the parameters are visible: >> >>2014-03-24 21:36:17,681 DEBUG [kvm.resource.LibvirtComputingResource] >>(agentRequest-Handler-2:null) Executing: >>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.sh -l >>r-19-VM -t all -d /var/lib/libvirt/images/r-19-VM-patchdisk -p >>%template=domP%name=r-19-VM%eth2ip=10.193.17.190%eth2mask=255.255.255.0%ga >>teway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=cs6clou >>d.internal%dhcprange=10.124.99.1%eth0ip=169.254.1.60%eth0mask=255.255.0.0% >>type=router%disable_rp_filter=true%dns1=10.193.17.1 >> >> >>Any hint is appreciated. >> >>Thanks >> >> >>Kambiz >> >> >>Alena Prokharchyk <alena.prokharc...@citrix.com> wrote: >>> >>> Yes, Kambiz, you followed up right, and vm id=15 is the culprit. If vm >>> id=15 is expunged, we have to clear out the reference to it from >>> user_ip_address table. Here is the flow: >>> >>> 1) Save the db dump. >>> 2) Run the query to cleanup the reference: >>> >>> Update user_ip_address set one_to_one_nat=0, instance_id=null where >>> id=<problematic public ip address id> >>> >>> >>> >>> Let me know how it works. >>> >>> -Alena. >>> >>> On 3/24/14, 10:55 AM, "Kambiz Darabi" <dar...@m-creations.com> wrote: >>> >>>>Hi, >>>> >>>>I hope I have understood what you wrote and created the following query >>>>correctly: >>>> >>>>select uip.vm_id, uip.network_id, uip.public_ip_address, >>>> n.state as nic_state, n.removed as nic_removed, >>>> vm.state as vm_state, vm.removed as vm_removed >>>>from user_ip_address uip >>>> join nics n on uip.vm_id = n.instance_id >>>> join vm_instance vm on uip.vm_id = vm.id >>>>where uip.id in (Select ip_address_id from firewall_rules fr where >>>>fr.network_id=205); >>>> >>>> >>>>+-------+------------+-------------------+--------------+--------------- >>>>-- >>>>----+-----------+------------+ >>>>| vm_id | network_id | public_ip_address | nic_state | nic_removed >>>> | vm_state | vm_removed | >>>>+-------+------------+-------------------+--------------+--------------- >>>>-- >>>>----+-----------+------------+ >>>>| 6 | 205 | 10.193.17.169 | Allocated | NULL >>>> | Stopped | NULL | >>>>| 10 | 205 | 10.193.17.136 | Allocated | NULL >>>> | Stopped | NULL | >>>>| 12 | 205 | 10.193.17.140 | Allocated | NULL >>>> | Stopped | NULL | >>>>| 13 | 205 | 10.193.17.141 | Allocated | NULL >>>> | Stopped | NULL | >>>>| 14 | 205 | 10.193.17.142 | Allocated | NULL >>>> | Stopped | NULL | >>>>| 15 | 205 | 10.193.17.174 | Deallocating | 2014-03-18 >>>>23:00:53 | Expunging | NULL | >>>>| 16 | 205 | 10.193.17.103 | Allocated | NULL >>>> | Stopped | NULL | >>>>+-------+------------+-------------------+--------------+--------------- >>>>-- >>>>----+-----------+------------+ >>>> >>>>Is VM id 15 what you are looking for? >>>> >>>>Thank you >>>> >>>> >>>>Kambiz >>>> >>>>Alena Prokharchyk <alena.prokharc...@citrix.com> wrote: >>>>> >>>>> Kambiz, can you please try one more thing. >>>>> >>>>> 1) Locate all the firewall rules for your guest network (205, right?) >>>>> >>>>> Select id, ip_address_id from firewall_rules where network_id=205; >>>>> >>>>> 2) Now get all static nat enabled ip addresses for those rules: >>>>> >>>>> Select vm_id, network_id from user_ip_address where id in (Select id, >>>>> ip_address_id from firewall_rules where network_id=205); >>>>> >>>>> For each vmId/networkId combo, check if there is non-removed nic and >>>>> non-expunged vm. There might be some incorrect static nat ip/vm >>>>>reference >>>>> referring to vm that is removed already. If you find any, let me know >>>>>and >>>>> I will tell you how to clean it up >>>>> >>>>> -Alena. >>>>> >>>>> On 3/22/14, 5:41 AM, "Kambiz Darabi" <dar...@m-creations.com> wrote: >>>>> >>>>>>Hi Alena, >>>>>> >>>>>>thank you for your help. >>>>>> >>>>>>The query returns no rows, i.e. nics.removed was not null, but I >>>>>>removed >>>>>>the row though to see what happens: a new virtual router was created >>>>>>which also couldn't be started due to the same NPE. I reverted the >>>>>>change by restoring from the dump. >>>>>> >>>>>>I have to mention that prior to the restart, r-7-VM was the router >>>>>>which >>>>>>was used by my instances. I deleted the router using the UI after the >>>>>>first >>>>>>occurrence of the NPE, because a post with a similar problem suggested >>>>>>that the deleted router would be recreated again (and this procedure >>>>>>solved the problem). >>>>>> >>>>>>Below I have attached the state of the two tables. >>>>>> >>>>>>Anything else I can try? >>>>>> >>>>>>Thank you >>>>>> >>>>>> >>>>>>Kambiz >>>>>> >>>>>>mysql> select n.id, n.removed, n.ip4_address, n.netmask, n.gateway, >>>>>>n.ip_type, n.reserver_name, n.network_id, i.id as instance_id, i.name, >>>>>>i.state, i.type from vm_instance i join nics n on n.instance_id = i.id >>>>>>where i.type = 'DomainRouter'; >>>>>>+----+---------------------+---------------+---------------+---------- >>>>>>-- >>>>>>-+ >>>>>>---------+--------------------------+------------+-------------+------ >>>>>>-- >>>>>>-+ >>>>>>-----------+--------------+ >>>>>>| id | removed | ip4_address | netmask | gateway >>>>>>| ip_type | reserver_name | network_id | instance_id | name >>>>>>| state | type | >>>>>>+----+---------------------+---------------+---------------+---------- >>>>>>-- >>>>>>-+ >>>>>>---------+--------------------------+------------+-------------+------ >>>>>>-- >>>>>>-+ >>>>>>-----------+--------------+ >>>>>>| 9 | 2014-03-17 11:27:58 | 10.124.99.1 | 255.255.255.0 | NULL >>>>>>| NULL | ExternalGuestNetworkGuru | 204 | 4 | >>>>>>r-4-VM >>>>>>| Expunging | DomainRouter | >>>>>>| 10 | 2014-03-17 11:27:58 | NULL | NULL | NULL >>>>>>| NULL | ControlNetworkGuru | 202 | 4 | >>>>>>r-4-VM >>>>>>| Expunging | DomainRouter | >>>>>>| 11 | 2014-03-17 11:27:58 | 10.193.17.139 | 255.255.255.0 | >>>>>>10.193.17.1 >>>>>>| NULL | PublicNetworkGuru | 200 | 4 | >>>>>>r-4-VM >>>>>>| Expunging | DomainRouter | >>>>>>| 14 | 2014-03-17 11:27:52 | 10.124.99.1 | 255.255.255.0 | NULL >>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 7 | >>>>>>r-7-VM >>>>>>| Expunging | DomainRouter | >>>>>>| 15 | 2014-03-17 11:27:52 | NULL | NULL | NULL >>>>>>| NULL | ControlNetworkGuru | 202 | 7 | >>>>>>r-7-VM >>>>>>| Expunging | DomainRouter | >>>>>>| 16 | 2014-03-17 11:27:52 | 10.193.17.190 | 255.255.255.0 | >>>>>>10.193.17.1 >>>>>>| NULL | PublicNetworkGuru | 200 | 7 | >>>>>>r-7-VM >>>>>>| Expunging | DomainRouter | >>>>>>| 26 | 2014-03-18 08:11:16 | 10.124.99.1 | 255.255.255.0 | NULL >>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 18 | >>>>>>r-18-VM >>>>>>| Expunging | DomainRouter | >>>>>>| 27 | 2014-03-18 08:11:16 | NULL | NULL | NULL >>>>>>| NULL | ControlNetworkGuru | 202 | 18 | >>>>>>r-18-VM >>>>>>| Expunging | DomainRouter | >>>>>>| 28 | 2014-03-18 08:11:16 | 10.193.17.190 | 255.255.255.0 | >>>>>>10.193.17.1 >>>>>>| NULL | PublicNetworkGuru | 200 | 18 | >>>>>>r-18-VM >>>>>>| Expunging | DomainRouter | >>>>>>| 29 | NULL | 10.124.99.1 | 255.255.255.0 | NULL >>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 19 | >>>>>>r-19-VM >>>>>>| Stopped | DomainRouter | >>>>>>| 30 | NULL | NULL | NULL | NULL >>>>>>| NULL | ControlNetworkGuru | 202 | 19 | >>>>>>r-19-VM >>>>>>| Stopped | DomainRouter | >>>>>>| 31 | NULL | 10.193.17.190 | 255.255.255.0 | >>>>>>10.193.17.1 >>>>>>| NULL | PublicNetworkGuru | 200 | 19 | >>>>>>r-19-VM >>>>>>| Stopped | DomainRouter | >>>>>>+----+---------------------+---------------+---------------+---------- >>>>>>-- >>>>>>-+ >>>>>>---------+--------------------------+------------+-------------+------ >>>>>>-- >>>>>>-+ >>>>>>-----------+--------------+ >>>>>> >>>>>>mysql> select * from router_network_ref; >>>>>>+----+-----------+------------+------------+ >>>>>>| id | router_id | network_id | guest_type | >>>>>>+----+-----------+------------+------------+ >>>>>>| 1 | 4 | 204 | Isolated | >>>>>>| 2 | 7 | 205 | Isolated | >>>>>>| 3 | 18 | 205 | Isolated | >>>>>>| 4 | 19 | 205 | Isolated | >>>>>>+----+-----------+------------+------------+ >>>>>> >>>>>> >>>>>> >>>>>>Alena Prokharchyk <alena.prokharc...@citrix.com> wrote: >>>>>>> >>>>>>> The error happens not because Ip is null, but because the nic in a >>>>>>>certain >>>>>>> network can¹t be found. Looks like there is some bug in VPC nic >>>>>>> plug/unplug for Guest networks process. >>>>>>> >>>>>>> Kambiz, please do the following to fix it: >>>>>>> >>>>>>> 1) Stop the MS >>>>>>> 2) Take the DB dump of cloud db in case you have to revert back. >>>>>>> 3) Run the query: >>>>>>> >>>>>>> select * from router_network_ref where router_id=<id of your VR) and >>>>>>> network_id not in (select network_id from nics where instance_id=<ID >>>>>>>of >>>>>>> your VR> and removed is null); >>>>>>> >>>>>>> It will give you the list of networks refs that somehow weren¹t >>>>>>>cleaned >>>>>>> during the nic detach. Remove the entry returned from >>>>>>>router_network_ref >>>>>>> table. >>>>>>> >>>>>>> Let me know how it works. >>>>>>> >>>>>>> -Alena. >>>>>>> >>>>>>> >>>>>>> On 3/21/14, 3:36 PM, "Kambiz Darabi" <dar...@m-creations.com> wrote: >>>>>>> >>>>>>>>Hello, >>>>>>>> >>>>>>>>as this is my first post to the list, I would like to thank all >>>>>>>>contributors for Cloudstack which I use since last fall without any >>>>>>>>problems. I run 4.1.1 with KVM and advanced networking. >>>>>>>> >>>>>>>>After a restart of the management server (stopping and starting the >>>>>>>>java >>>>>>>>process), the virtual domain router doesn't start and >>>>>>>>management-server.log shows a NullPointerException in >>>>>>>>NetworkModelImpl.getIpInNetwork (cf. stack trace below). >>>>>>>> >>>>>>>>By putting the server in debug mode and remote debugging, I found >>>>>>>>out >>>>>>>>that the reason is a row in the table nics which has NULL in ip (cf. >>>>>>>>row >>>>>>>>with id 30 in the result of the select statement below). >>>>>>>> >>>>>>>>What can I do to quickly solve this problem? Any pointers or >>>>>>>>suggestions >>>>>>>>are appreciated as the system is currently unusable. >>>>>>>> >>>>>>>>Thank you for your help >>>>>>>> >>>>>>>> >>>>>>>>Kambiz >>>>>>>> >>>>>>>> >>>>>>>>management-server.log: >>>>>>>> >>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for >>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1] >>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for >>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1] >>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare for >>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1] >>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for >>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1] >>>>>>>>2014-03-18 10:03:27,151 WARN >>>>>>>>[network.element.VpcVirtualRouterElement] >>>>>>>>(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not associated >>>>>>>>with >>>>>>>>any VPC >>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for >>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1] >>>>>>>>2014-03-18 10:03:27,151 DEBUG [network.element.NiciraNvpElement] >>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle >>>>>>>>service >>>>>>>>Connectivity on network net1 >>>>>>>>2014-03-18 10:03:27,153 DEBUG [cloud.network.NetworkModelImpl] >>>>>>>>(Job-Executor-1:job-176) Service SecurityGroup is not supported in >>>>>>>>the >>>>>>>>network id=205 >>>>>>>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>(Job-Executor-1:job-176) Lock is acquired for network id 202 as a >>>>>>>>part >>>>>>>>of >>>>>>>>network implement >>>>>>>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>(Job-Executor-1:job-176) Network id=202 is already implemented >>>>>>>>2014-03-18 10:03:27,157 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>(Job-Executor-1:job-176) Lock is released for network id 202 as a >>>>>>>>part >>>>>>>>of >>>>>>>>network implement >>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for >>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99] >>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for >>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99] >>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare for >>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99] >>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for >>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99] >>>>>>>>2014-03-18 10:03:27,187 WARN >>>>>>>>[network.element.VpcVirtualRouterElement] >>>>>>>>(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is not >>>>>>>>associated >>>>>>>>with any VPC >>>>>>>>2014-03-18 10:03:27,188 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for >>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99] >>>>>>>>2014-03-18 10:03:27,188 DEBUG [network.element.NiciraNvpElement] >>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle >>>>>>>>service >>>>>>>>Connectivity on network null >>>>>>>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl] >>>>>>>>(Job-Executor-1:job-176) Checking if we need to prepare 1 volumes >>>>>>>>for >>>>>>>>VM[DomainRouter|r-19-VM] >>>>>>>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl] >>>>>>>>(Job-Executor-1:job-176) No need to recreate the volume: >>>>>>>>Vol[24|vm=19|ROOT], since it already has a pool assigned: 200, >>>>>>>>adding >>>>>>>>disk to VM >>>>>>>>2014-03-18 10:03:27,224 DEBUG >>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl] >>>>>>>>(Job-Executor-1:job-176) Boot Args for VM[DomainRouter|r-19-VM]: >>>>>>>>template=domP name=r-19-VM eth2ip=10.193.17.190 >>>>>>>>eth2mask=255.255.255.0 >>>>>>>>gateway=10.193.17.1 eth0ip=10.124.99.1 eth0mask=255.255.255.0 >>>>>>>>domain=cs6cloud.internal dhcprange=10.124.99.1 eth0ip=169.254.3.99 >>>>>>>>eth0mask=255.255.0.0 type=router disable_rp_filter=true >>>>>>>>dns1=10.193.17.1 >>>>>>>>2014-03-18 10:03:27,343 DEBUG >>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl] >>>>>>>>(Job-Executor-1:job-176) Found 8 ip(s) to apply as a part of domR >>>>>>>>VM[DomainRouter|r-19-VM] start. >>>>>>>>2014-03-18 10:03:27,415 DEBUG >>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl] >>>>>>>>(Job-Executor-1:job-176) Resending ipAssoc, port forwarding, load >>>>>>>>balancing rules as a part of Virtual router start >>>>>>>>2014-03-18 10:03:27,499 DEBUG >>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl] >>>>>>>>(Job-Executor-1:job-176) Found 12 firewall Egress rule(s) to apply >>>>>>>>as >>>>>>>>a >>>>>>>>part of domR VM[DomainRouter|r-19-VM] start. >>>>>>>>2014-03-18 10:03:27,593 ERROR [cloud.vm.VirtualMachineManagerImpl] >>>>>>>>(Job-Executor-1:job-176) Failed to start instance >>>>>>>>VM[DomainRouter|r-19-VM] >>>>>>>>java.lang.NullPointerException >>>>>>>> at >>>>>>>>com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.j >>>>>>>>av >>>>>>>>a: >>>>>>>>76 >>>>>>>>3) >>>>>>>> at >>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalize >>>>>>>>Ne >>>>>>>>tw >>>>>>>>or >>>>>>>>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346) >>>>>>>> at >>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.final >>>>>>>>iz >>>>>>>>eN >>>>>>>>et >>>>>>>>workRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928) >>>>>>>> at >>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalize >>>>>>>>Co >>>>>>>>mm >>>>>>>>an >>>>>>>>dsOnStart(VirtualNetworkApplianceManagerImpl.java:2241) >>>>>>>> at >>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.final >>>>>>>>iz >>>>>>>>eC >>>>>>>>om >>>>>>>>mandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767) >>>>>>>> at >>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalize >>>>>>>>De >>>>>>>>pl >>>>>>>>oy >>>>>>>>ment(VirtualNetworkApplianceManagerImpl.java:2205) >>>>>>>> at >>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineMa >>>>>>>>na >>>>>>>>ge >>>>>>>>rI >>>>>>>>mpl.java:763) >>>>>>>> at >>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerIm >>>>>>>>pl >>>>>>>>.j >>>>>>>>av >>>>>>>>a:471) >>>>>>>> at >>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(Vi >>>>>>>>rt >>>>>>>>ua >>>>>>>>lN >>>>>>>>etworkApplianceManagerImpl.java:2616) >>>>>>>> at >>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVir >>>>>>>>tu >>>>>>>>al >>>>>>>>Ro >>>>>>>>uter(VirtualNetworkApplianceManagerImpl.java:1824) >>>>>>>> at >>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRou >>>>>>>>te >>>>>>>>rs >>>>>>>>(V >>>>>>>>irtualNetworkApplianceManagerImpl.java:1924) >>>>>>>> at >>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVi >>>>>>>>rt >>>>>>>>ua >>>>>>>>lR >>>>>>>>outerInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902) >>>>>>>> at >>>>>>>>com.cloud.network.element.VirtualRouterElement.implement(VirtualRout >>>>>>>>er >>>>>>>>El >>>>>>>>em >>>>>>>>ent.java:175) >>>>>>>> at >>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndReso >>>>>>>>ur >>>>>>>>ce >>>>>>>>s( >>>>>>>>NetworkManagerImpl.java:1518) >>>>>>>> at >>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManager >>>>>>>>Im >>>>>>>>pl >>>>>>>>.j >>>>>>>>ava:1434) >>>>>>>> at >>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Interc >>>>>>>>ep >>>>>>>>to >>>>>>>>rD >>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125) >>>>>>>> at >>>>>>>>com.cloud.network.NetworkManagerImpl.startNetwork(NetworkManagerImpl >>>>>>>>.j >>>>>>>>av >>>>>>>>a: >>>>>>>>2435) >>>>>>>> at >>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRou >>>>>>>>te >>>>>>>>r( >>>>>>>>Vi >>>>>>>>rtualNetworkApplianceManagerImpl.java:2855) >>>>>>>> at >>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRou >>>>>>>>te >>>>>>>>r( >>>>>>>>Vi >>>>>>>>rtualNetworkApplianceManagerImpl.java:2824) >>>>>>>> at >>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Interc >>>>>>>>ep >>>>>>>>to >>>>>>>>rD >>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125) >>>>>>>> at >>>>>>>>org.apache.cloudstack.api.command.admin.router.StartRouterCmd.execut >>>>>>>>e( >>>>>>>>St >>>>>>>>ar >>>>>>>>tRouterCmd.java:103) >>>>>>>> >>>>>>>> >>>>>>>>table nics: >>>>>>>> >>>>>>>>mysql> select * from nics where reserver_name = >>>>>>>>'ControlNetworkGuru'; >>>>>>>>+----+--------------------------------------+-------------+--------- >>>>>>>>-- >>>>>>>>-- >>>>>>>>-- >>>>>>>>----+---------------+-------------+-------------+---------+--------- >>>>>>>>-- >>>>>>>>-- >>>>>>>>-- >>>>>>>>+------------+--------+--------------+----------+------------------- >>>>>>>>-+ >>>>>>>>-- >>>>>>>>-- >>>>>>>>----------------------------------+-----------+--------------------- >>>>>>>>+- >>>>>>>>-- >>>>>>>>-- >>>>>>>>----------+-------------+-------------+--------------------+-------- >>>>>>>>-- >>>>>>>>-- >>>>>>>>-- >>>>>>>>-------+---------------------+-------------+----------+ >>>>>>>>| id | uuid | instance_id | >>>>>>>>mac_address >>>>>>>> | ip4_address | netmask | gateway | ip_type | >>>>>>>>broadcast_uri >>>>>>>>| network_id | mode | state | strategy | reserver_name >>>>>>>> | >>>>>>>>reservation_id | device_id | update_time >>>>>>>> | >>>>>>>>isolation_uri | ip6_address | default_nic | vm_type | >>>>>>>>created >>>>>>>> | removed | ip6_gateway | ip6_cidr | >>>>>>>>+----+--------------------------------------+-------------+--------- >>>>>>>>-- >>>>>>>>-- >>>>>>>>-- >>>>>>>>----+---------------+-------------+-------------+---------+--------- >>>>>>>>-- >>>>>>>>-- >>>>>>>>-- >>>>>>>>+------------+--------+--------------+----------+------------------- >>>>>>>>-+ >>>>>>>>-- >>>>>>>>-- >>>>>>>>----------------------------------+-----------+--------------------- >>>>>>>>+- >>>>>>>>-- >>>>>>>>-- >>>>>>>>----------+-------------+-------------+--------------------+-------- >>>>>>>>-- >>>>>>>>-- >>>>>>>>-- >>>>>>>>-------+---------------------+-------------+----------+ >>>>>>>>| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 | 1 | >>>>>>>>0e:00:a9:fe:00:55 | 169.254.0.85 | 255.255.0.0 | 169.254.0.1 | Ip4 >>>>>>>>| >>>>>>>>NULL | 202 | Static | Reserved | Start | >>>>>>>>ControlNetworkGuru | 993864b4-9dde-47d6-8fd6-cf94050442c6 | >>>>>>>>0 >>>>>>>>| >>>>>>>>2014-03-17 22:21:38 | NULL | NULL | 0 | >>>>>>>>SecondaryStorageVm | 2013-09-06 12:44:42 | NULL | >>>>>>>>NULL >>>>>>>> | NULL | >>>>>>>>| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 | 2 | >>>>>>>>0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 | 169.254.0.1 | Ip4 >>>>>>>>| >>>>>>>>NULL | 202 | Static | Reserved | Start | >>>>>>>>ControlNetworkGuru | 852e0a65-c72a-448f-ac71-2bb3549a5a41 | >>>>>>>>0 >>>>>>>>| >>>>>>>>2014-03-17 22:21:38 | NULL | NULL | 0 | >>>>>>>>ConsoleProxy | 2013-09-06 12:44:42 | NULL | >>>>>>>>NULL >>>>>>>> | NULL | >>>>>>>>| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 | 4 | NULL >>>>>>>> | NULL | NULL | NULL | NULL | NULL >>>>>>>>| 202 | Static | Deallocating | Start | >>>>>>>>ControlNetworkGuru | >>>>>>>>c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 | 2014-03-17 >>>>>>>>12:27:58 | >>>>>>>>NULL | NULL | 0 | DomainRouter | >>>>>>>>2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL | NULL | >>>>>>>>| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc | 7 | NULL >>>>>>>> | NULL | NULL | NULL | NULL | NULL >>>>>>>>| 202 | Static | Deallocating | Start | >>>>>>>>ControlNetworkGuru | >>>>>>>>ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 | 2014-03-17 >>>>>>>>12:27:52 | >>>>>>>>NULL | NULL | 0 | DomainRouter | >>>>>>>>2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL | NULL | >>>>>>>>| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 | 18 | NULL >>>>>>>> | NULL | NULL | NULL | NULL | NULL >>>>>>>>| 202 | Static | Deallocating | Start | >>>>>>>>ControlNetworkGuru | >>>>>>>>ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 | 2014-03-18 >>>>>>>>09:11:16 | >>>>>>>>NULL | NULL | 0 | DomainRouter | >>>>>>>>2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL | NULL | >>>>>>>>| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d | 19 | NULL >>>>>>>> | NULL | NULL | NULL | NULL | NULL >>>>>>>>| 202 | Static | Allocated | Start | >>>>>>>>ControlNetworkGuru | >>>>>>>>e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 | 2014-03-18 >>>>>>>>09:11:44 | >>>>>>>>NULL | NULL | 0 | DomainRouter | >>>>>>>>2014-03-18 08:11:32 | NULL | NULL | NULL | >>>>>>>>+----+--------------------------------------+-------------+--------- >>>>>>>>-- >>>>>>>>-- >>>>>>>>-- >>>>>>>>----+---------------+-------------+-------------+---------+--------- >>>>>>>>-- >>>>>>>>-- >>>>>>>>-- >>>>>>>>+------------+--------+--------------+----------+------------------- >>>>>>>>-+ >>>>>>>>-- >>>>>>>>-- >>>>>>>>----------------------------------+-----------+--------------------- >>>>>>>>+- >>>>>>>>-- >>>>>>>>-- >>>>>>>>----------+-------------+-------------+--------------------+-------- >>>>>>>>-- >>>>>>>>-- >>>>>>>>-- >>>>>>>>-------+---------------------+-------------+----------+ >>>>>>>> >>>>>>>>
