Kambiz, can you please check the following:
1) Check if the gateway is set on control network: mysql> select gateway, cidr from networks where traffic_type=‘Control’; 2) For router control nic, check if network/gateway are set. Select gateway,netmask from nics where instance_id=<routerid> and network_id=<control network id> -Alena. On 3/25/14, 5:47 AM, "Kambiz Darabi" <dar...@m-creations.com> wrote: >Hi, > >I looked up the startup command of the old router instance which worked >correctly: > >/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.sh -l >r-7-VM -t all -d /var/lib/libvirt/images/r-7-VM-patchdisk -p >%template=domP%name=r-7-VM%eth2ip=10.193.17.190%eth2mask=255.255.255.0%gat >eway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=cs6cloud >.internal%dhcprange=10.124.99.1%eth1ip=169.254.2.46%eth1mask=255.255.0.0%t >ype=router%disable_rp_filter=true%dns1=10.193.17.1 > >The new command (cf. below) doesn't have the parameters eth1ip and >eth1mask. > >Thanks > > >Kambiz > >Alena Prokharchyk <alena.prokharc...@citrix.com> wrote: >> >> I don’t think its relevant as the piece we’ve fixed, just eliminated >> static nat rule programming for non-existing vm. Missing netmask on eth1 >> doesn’t seem related to the problem (although we have to figure out why >> its missing), as the connection that fails, happening to link local >>169.x >> eth0 interface. >> >> Edison, can you please tell us how to debug link local connection >>failure, >> on KVM agent? >> >> Thank you, >> Alena. >> >> On 3/24/14, 1:47 PM, "Kambiz Darabi" <dar...@m-creations.com> wrote: >> >>>Hi, >>> >>>thank you, the NullPointerException doesn't occur any more, but there >>>still seems to be a problem during startup of the router. >>> >>>When I start the virtual router, it comes up, but in agent.log, there >>>are lots of 'Could not connect to 169.254.1.x' messages. >>> >>>Then I logged into the virtual router to find out that the netmask of >>>eth1 is missing in the interfaces file: >>> >>>root@host:~# virsh console r-19-VM >>>Connected to domain r-19-VM >>>Escape character is ^] >>> >>>Debian GNU/Linux 6.0 r-19-VM ttyS0 >>> >>>r-19-VM login: root >>>... >>>root@r-19-VM:~# cat /etc/network/interfaces >>>auto lo eth0 eth1 eth2 >>>iface lo inet loopback >>> >>>iface eth0 inet static >>> address 169.254.1.59 >>> netmask 255.255.0.0 >>>iface eth1 inet static >>> address 10.193.17.1 >>> netmask >>>iface eth2 inet static >>> address 10.193.17.190 >>> netmask 255.255.255.0 >>> >>>I don't know if it is relevant, but this is the line from agent.log >>>where the parameters are visible: >>> >>>2014-03-24 21:36:17,681 DEBUG [kvm.resource.LibvirtComputingResource] >>>(agentRequest-Handler-2:null) Executing: >>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.sh -l >>>r-19-VM -t all -d /var/lib/libvirt/images/r-19-VM-patchdisk -p >>>%template=domP%name=r-19-VM%eth2ip=10.193.17.190%eth2mask=255.255.255.0% >>>ga >>>teway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=cs6cl >>>ou >>>d.internal%dhcprange=10.124.99.1%eth0ip=169.254.1.60%eth0mask=255.255.0. >>>0% >>>type=router%disable_rp_filter=true%dns1=10.193.17.1 >>> >>> >>>Any hint is appreciated. >>> >>>Thanks >>> >>> >>>Kambiz >>> >>> >>>Alena Prokharchyk <alena.prokharc...@citrix.com> wrote: >>>> >>>> Yes, Kambiz, you followed up right, and vm id=15 is the culprit. If vm >>>> id=15 is expunged, we have to clear out the reference to it from >>>> user_ip_address table. Here is the flow: >>>> >>>> 1) Save the db dump. >>>> 2) Run the query to cleanup the reference: >>>> >>>> Update user_ip_address set one_to_one_nat=0, instance_id=null where >>>> id=<problematic public ip address id> >>>> >>>> >>>> >>>> Let me know how it works. >>>> >>>> -Alena. >>>> >>>> On 3/24/14, 10:55 AM, "Kambiz Darabi" <dar...@m-creations.com> wrote: >>>> >>>>>Hi, >>>>> >>>>>I hope I have understood what you wrote and created the following >>>>>query >>>>>correctly: >>>>> >>>>>select uip.vm_id, uip.network_id, uip.public_ip_address, >>>>> n.state as nic_state, n.removed as nic_removed, >>>>> vm.state as vm_state, vm.removed as vm_removed >>>>>from user_ip_address uip >>>>> join nics n on uip.vm_id = n.instance_id >>>>> join vm_instance vm on uip.vm_id = vm.id >>>>>where uip.id in (Select ip_address_id from firewall_rules fr where >>>>>fr.network_id=205); >>>>> >>>>> >>>>>+-------+------------+-------------------+--------------+------------- >>>>>-- >>>>>-- >>>>>----+-----------+------------+ >>>>>| vm_id | network_id | public_ip_address | nic_state | nic_removed >>>>> | vm_state | vm_removed | >>>>>+-------+------------+-------------------+--------------+------------- >>>>>-- >>>>>-- >>>>>----+-----------+------------+ >>>>>| 6 | 205 | 10.193.17.169 | Allocated | NULL >>>>> | Stopped | NULL | >>>>>| 10 | 205 | 10.193.17.136 | Allocated | NULL >>>>> | Stopped | NULL | >>>>>| 12 | 205 | 10.193.17.140 | Allocated | NULL >>>>> | Stopped | NULL | >>>>>| 13 | 205 | 10.193.17.141 | Allocated | NULL >>>>> | Stopped | NULL | >>>>>| 14 | 205 | 10.193.17.142 | Allocated | NULL >>>>> | Stopped | NULL | >>>>>| 15 | 205 | 10.193.17.174 | Deallocating | 2014-03-18 >>>>>23:00:53 | Expunging | NULL | >>>>>| 16 | 205 | 10.193.17.103 | Allocated | NULL >>>>> | Stopped | NULL | >>>>>+-------+------------+-------------------+--------------+------------- >>>>>-- >>>>>-- >>>>>----+-----------+------------+ >>>>> >>>>>Is VM id 15 what you are looking for? >>>>> >>>>>Thank you >>>>> >>>>> >>>>>Kambiz >>>>> >>>>>Alena Prokharchyk <alena.prokharc...@citrix.com> wrote: >>>>>> >>>>>> Kambiz, can you please try one more thing. >>>>>> >>>>>> 1) Locate all the firewall rules for your guest network (205, >>>>>>right?) >>>>>> >>>>>> Select id, ip_address_id from firewall_rules where network_id=205; >>>>>> >>>>>> 2) Now get all static nat enabled ip addresses for those rules: >>>>>> >>>>>> Select vm_id, network_id from user_ip_address where id in (Select >>>>>>id, >>>>>> ip_address_id from firewall_rules where network_id=205); >>>>>> >>>>>> For each vmId/networkId combo, check if there is non-removed nic and >>>>>> non-expunged vm. There might be some incorrect static nat ip/vm >>>>>>reference >>>>>> referring to vm that is removed already. If you find any, let me >>>>>>know >>>>>>and >>>>>> I will tell you how to clean it up >>>>>> >>>>>> -Alena. >>>>>> >>>>>> On 3/22/14, 5:41 AM, "Kambiz Darabi" <dar...@m-creations.com> wrote: >>>>>> >>>>>>>Hi Alena, >>>>>>> >>>>>>>thank you for your help. >>>>>>> >>>>>>>The query returns no rows, i.e. nics.removed was not null, but I >>>>>>>removed >>>>>>>the row though to see what happens: a new virtual router was created >>>>>>>which also couldn't be started due to the same NPE. I reverted the >>>>>>>change by restoring from the dump. >>>>>>> >>>>>>>I have to mention that prior to the restart, r-7-VM was the router >>>>>>>which >>>>>>>was used by my instances. I deleted the router using the UI after >>>>>>>the >>>>>>>first >>>>>>>occurrence of the NPE, because a post with a similar problem >>>>>>>suggested >>>>>>>that the deleted router would be recreated again (and this procedure >>>>>>>solved the problem). >>>>>>> >>>>>>>Below I have attached the state of the two tables. >>>>>>> >>>>>>>Anything else I can try? >>>>>>> >>>>>>>Thank you >>>>>>> >>>>>>> >>>>>>>Kambiz >>>>>>> >>>>>>>mysql> select n.id, n.removed, n.ip4_address, n.netmask, n.gateway, >>>>>>>n.ip_type, n.reserver_name, n.network_id, i.id as instance_id, >>>>>>>i.name, >>>>>>>i.state, i.type from vm_instance i join nics n on n.instance_id = >>>>>>>i.id >>>>>>>where i.type = 'DomainRouter'; >>>>>>>+----+---------------------+---------------+---------------+-------- >>>>>>>-- >>>>>>>-- >>>>>>>-+ >>>>>>>---------+--------------------------+------------+-------------+---- >>>>>>>-- >>>>>>>-- >>>>>>>-+ >>>>>>>-----------+--------------+ >>>>>>>| id | removed | ip4_address | netmask | gateway >>>>>>>| ip_type | reserver_name | network_id | instance_id | >>>>>>>name >>>>>>>| state | type | >>>>>>>+----+---------------------+---------------+---------------+-------- >>>>>>>-- >>>>>>>-- >>>>>>>-+ >>>>>>>---------+--------------------------+------------+-------------+---- >>>>>>>-- >>>>>>>-- >>>>>>>-+ >>>>>>>-----------+--------------+ >>>>>>>| 9 | 2014-03-17 11:27:58 | 10.124.99.1 | 255.255.255.0 | NULL >>>>>>>| NULL | ExternalGuestNetworkGuru | 204 | 4 | >>>>>>>r-4-VM >>>>>>>| Expunging | DomainRouter | >>>>>>>| 10 | 2014-03-17 11:27:58 | NULL | NULL | NULL >>>>>>>| NULL | ControlNetworkGuru | 202 | 4 | >>>>>>>r-4-VM >>>>>>>| Expunging | DomainRouter | >>>>>>>| 11 | 2014-03-17 11:27:58 | 10.193.17.139 | 255.255.255.0 | >>>>>>>10.193.17.1 >>>>>>>| NULL | PublicNetworkGuru | 200 | 4 | >>>>>>>r-4-VM >>>>>>>| Expunging | DomainRouter | >>>>>>>| 14 | 2014-03-17 11:27:52 | 10.124.99.1 | 255.255.255.0 | NULL >>>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 7 | >>>>>>>r-7-VM >>>>>>>| Expunging | DomainRouter | >>>>>>>| 15 | 2014-03-17 11:27:52 | NULL | NULL | NULL >>>>>>>| NULL | ControlNetworkGuru | 202 | 7 | >>>>>>>r-7-VM >>>>>>>| Expunging | DomainRouter | >>>>>>>| 16 | 2014-03-17 11:27:52 | 10.193.17.190 | 255.255.255.0 | >>>>>>>10.193.17.1 >>>>>>>| NULL | PublicNetworkGuru | 200 | 7 | >>>>>>>r-7-VM >>>>>>>| Expunging | DomainRouter | >>>>>>>| 26 | 2014-03-18 08:11:16 | 10.124.99.1 | 255.255.255.0 | NULL >>>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 18 | >>>>>>>r-18-VM >>>>>>>| Expunging | DomainRouter | >>>>>>>| 27 | 2014-03-18 08:11:16 | NULL | NULL | NULL >>>>>>>| NULL | ControlNetworkGuru | 202 | 18 | >>>>>>>r-18-VM >>>>>>>| Expunging | DomainRouter | >>>>>>>| 28 | 2014-03-18 08:11:16 | 10.193.17.190 | 255.255.255.0 | >>>>>>>10.193.17.1 >>>>>>>| NULL | PublicNetworkGuru | 200 | 18 | >>>>>>>r-18-VM >>>>>>>| Expunging | DomainRouter | >>>>>>>| 29 | NULL | 10.124.99.1 | 255.255.255.0 | NULL >>>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 19 | >>>>>>>r-19-VM >>>>>>>| Stopped | DomainRouter | >>>>>>>| 30 | NULL | NULL | NULL | NULL >>>>>>>| NULL | ControlNetworkGuru | 202 | 19 | >>>>>>>r-19-VM >>>>>>>| Stopped | DomainRouter | >>>>>>>| 31 | NULL | 10.193.17.190 | 255.255.255.0 | >>>>>>>10.193.17.1 >>>>>>>| NULL | PublicNetworkGuru | 200 | 19 | >>>>>>>r-19-VM >>>>>>>| Stopped | DomainRouter | >>>>>>>+----+---------------------+---------------+---------------+-------- >>>>>>>-- >>>>>>>-- >>>>>>>-+ >>>>>>>---------+--------------------------+------------+-------------+---- >>>>>>>-- >>>>>>>-- >>>>>>>-+ >>>>>>>-----------+--------------+ >>>>>>> >>>>>>>mysql> select * from router_network_ref; >>>>>>>+----+-----------+------------+------------+ >>>>>>>| id | router_id | network_id | guest_type | >>>>>>>+----+-----------+------------+------------+ >>>>>>>| 1 | 4 | 204 | Isolated | >>>>>>>| 2 | 7 | 205 | Isolated | >>>>>>>| 3 | 18 | 205 | Isolated | >>>>>>>| 4 | 19 | 205 | Isolated | >>>>>>>+----+-----------+------------+------------+ >>>>>>> >>>>>>> >>>>>>> >>>>>>>Alena Prokharchyk <alena.prokharc...@citrix.com> wrote: >>>>>>>> >>>>>>>> The error happens not because Ip is null, but because the nic in a >>>>>>>>certain >>>>>>>> network can¹t be found. Looks like there is some bug in VPC nic >>>>>>>> plug/unplug for Guest networks process. >>>>>>>> >>>>>>>> Kambiz, please do the following to fix it: >>>>>>>> >>>>>>>> 1) Stop the MS >>>>>>>> 2) Take the DB dump of cloud db in case you have to revert back. >>>>>>>> 3) Run the query: >>>>>>>> >>>>>>>> select * from router_network_ref where router_id=<id of your VR) >>>>>>>>and >>>>>>>> network_id not in (select network_id from nics where >>>>>>>>instance_id=<ID >>>>>>>>of >>>>>>>> your VR> and removed is null); >>>>>>>> >>>>>>>> It will give you the list of networks refs that somehow weren¹t >>>>>>>>cleaned >>>>>>>> during the nic detach. Remove the entry returned from >>>>>>>>router_network_ref >>>>>>>> table. >>>>>>>> >>>>>>>> Let me know how it works. >>>>>>>> >>>>>>>> -Alena. >>>>>>>> >>>>>>>> >>>>>>>> On 3/21/14, 3:36 PM, "Kambiz Darabi" <dar...@m-creations.com> >>>>>>>>wrote: >>>>>>>> >>>>>>>>>Hello, >>>>>>>>> >>>>>>>>>as this is my first post to the list, I would like to thank all >>>>>>>>>contributors for Cloudstack which I use since last fall without >>>>>>>>>any >>>>>>>>>problems. I run 4.1.1 with KVM and advanced networking. >>>>>>>>> >>>>>>>>>After a restart of the management server (stopping and starting >>>>>>>>>the >>>>>>>>>java >>>>>>>>>process), the virtual domain router doesn't start and >>>>>>>>>management-server.log shows a NullPointerException in >>>>>>>>>NetworkModelImpl.getIpInNetwork (cf. stack trace below). >>>>>>>>> >>>>>>>>>By putting the server in debug mode and remote debugging, I found >>>>>>>>>out >>>>>>>>>that the reason is a row in the table nics which has NULL in ip >>>>>>>>>(cf. >>>>>>>>>row >>>>>>>>>with id 30 in the result of the select statement below). >>>>>>>>> >>>>>>>>>What can I do to quickly solve this problem? Any pointers or >>>>>>>>>suggestions >>>>>>>>>are appreciated as the system is currently unusable. >>>>>>>>> >>>>>>>>>Thank you for your help >>>>>>>>> >>>>>>>>> >>>>>>>>>Kambiz >>>>>>>>> >>>>>>>>> >>>>>>>>>management-server.log: >>>>>>>>> >>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for >>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1] >>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for >>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1] >>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare >>>>>>>>>for >>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1] >>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for >>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1] >>>>>>>>>2014-03-18 10:03:27,151 WARN >>>>>>>>>[network.element.VpcVirtualRouterElement] >>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not >>>>>>>>>associated >>>>>>>>>with >>>>>>>>>any VPC >>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for >>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1] >>>>>>>>>2014-03-18 10:03:27,151 DEBUG [network.element.NiciraNvpElement] >>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle >>>>>>>>>service >>>>>>>>>Connectivity on network net1 >>>>>>>>>2014-03-18 10:03:27,153 DEBUG [cloud.network.NetworkModelImpl] >>>>>>>>>(Job-Executor-1:job-176) Service SecurityGroup is not supported in >>>>>>>>>the >>>>>>>>>network id=205 >>>>>>>>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>>(Job-Executor-1:job-176) Lock is acquired for network id 202 as a >>>>>>>>>part >>>>>>>>>of >>>>>>>>>network implement >>>>>>>>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>>(Job-Executor-1:job-176) Network id=202 is already implemented >>>>>>>>>2014-03-18 10:03:27,157 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>>(Job-Executor-1:job-176) Lock is released for network id 202 as a >>>>>>>>>part >>>>>>>>>of >>>>>>>>>network implement >>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for >>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99] >>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for >>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99] >>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare >>>>>>>>>for >>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99] >>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for >>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99] >>>>>>>>>2014-03-18 10:03:27,187 WARN >>>>>>>>>[network.element.VpcVirtualRouterElement] >>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is not >>>>>>>>>associated >>>>>>>>>with any VPC >>>>>>>>>2014-03-18 10:03:27,188 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for >>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99] >>>>>>>>>2014-03-18 10:03:27,188 DEBUG [network.element.NiciraNvpElement] >>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle >>>>>>>>>service >>>>>>>>>Connectivity on network null >>>>>>>>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl] >>>>>>>>>(Job-Executor-1:job-176) Checking if we need to prepare 1 volumes >>>>>>>>>for >>>>>>>>>VM[DomainRouter|r-19-VM] >>>>>>>>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl] >>>>>>>>>(Job-Executor-1:job-176) No need to recreate the volume: >>>>>>>>>Vol[24|vm=19|ROOT], since it already has a pool assigned: 200, >>>>>>>>>adding >>>>>>>>>disk to VM >>>>>>>>>2014-03-18 10:03:27,224 DEBUG >>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl] >>>>>>>>>(Job-Executor-1:job-176) Boot Args for VM[DomainRouter|r-19-VM]: >>>>>>>>>template=domP name=r-19-VM eth2ip=10.193.17.190 >>>>>>>>>eth2mask=255.255.255.0 >>>>>>>>>gateway=10.193.17.1 eth0ip=10.124.99.1 eth0mask=255.255.255.0 >>>>>>>>>domain=cs6cloud.internal dhcprange=10.124.99.1 eth0ip=169.254.3.99 >>>>>>>>>eth0mask=255.255.0.0 type=router disable_rp_filter=true >>>>>>>>>dns1=10.193.17.1 >>>>>>>>>2014-03-18 10:03:27,343 DEBUG >>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl] >>>>>>>>>(Job-Executor-1:job-176) Found 8 ip(s) to apply as a part of domR >>>>>>>>>VM[DomainRouter|r-19-VM] start. >>>>>>>>>2014-03-18 10:03:27,415 DEBUG >>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl] >>>>>>>>>(Job-Executor-1:job-176) Resending ipAssoc, port forwarding, load >>>>>>>>>balancing rules as a part of Virtual router start >>>>>>>>>2014-03-18 10:03:27,499 DEBUG >>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl] >>>>>>>>>(Job-Executor-1:job-176) Found 12 firewall Egress rule(s) to apply >>>>>>>>>as >>>>>>>>>a >>>>>>>>>part of domR VM[DomainRouter|r-19-VM] start. >>>>>>>>>2014-03-18 10:03:27,593 ERROR [cloud.vm.VirtualMachineManagerImpl] >>>>>>>>>(Job-Executor-1:job-176) Failed to start instance >>>>>>>>>VM[DomainRouter|r-19-VM] >>>>>>>>>java.lang.NullPointerException >>>>>>>>> at >>>>>>>>>com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl >>>>>>>>>.j >>>>>>>>>av >>>>>>>>>a: >>>>>>>>>76 >>>>>>>>>3) >>>>>>>>> at >>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finali >>>>>>>>>ze >>>>>>>>>Ne >>>>>>>>>tw >>>>>>>>>or >>>>>>>>>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346) >>>>>>>>> at >>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.fin >>>>>>>>>al >>>>>>>>>iz >>>>>>>>>eN >>>>>>>>>et >>>>>>>>>workRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928 >>>>>>>>>) >>>>>>>>> at >>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finali >>>>>>>>>ze >>>>>>>>>Co >>>>>>>>>mm >>>>>>>>>an >>>>>>>>>dsOnStart(VirtualNetworkApplianceManagerImpl.java:2241) >>>>>>>>> at >>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.fin >>>>>>>>>al >>>>>>>>>iz >>>>>>>>>eC >>>>>>>>>om >>>>>>>>>mandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767) >>>>>>>>> at >>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finali >>>>>>>>>ze >>>>>>>>>De >>>>>>>>>pl >>>>>>>>>oy >>>>>>>>>ment(VirtualNetworkApplianceManagerImpl.java:2205) >>>>>>>>> at >>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachine >>>>>>>>>Ma >>>>>>>>>na >>>>>>>>>ge >>>>>>>>>rI >>>>>>>>>mpl.java:763) >>>>>>>>> at >>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManager >>>>>>>>>Im >>>>>>>>>pl >>>>>>>>>.j >>>>>>>>>av >>>>>>>>>a:471) >>>>>>>>> at >>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start( >>>>>>>>>Vi >>>>>>>>>rt >>>>>>>>>ua >>>>>>>>>lN >>>>>>>>>etworkApplianceManagerImpl.java:2616) >>>>>>>>> at >>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startV >>>>>>>>>ir >>>>>>>>>tu >>>>>>>>>al >>>>>>>>>Ro >>>>>>>>>uter(VirtualNetworkApplianceManagerImpl.java:1824) >>>>>>>>> at >>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startR >>>>>>>>>ou >>>>>>>>>te >>>>>>>>>rs >>>>>>>>>(V >>>>>>>>>irtualNetworkApplianceManagerImpl.java:1924) >>>>>>>>> at >>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deploy >>>>>>>>>Vi >>>>>>>>>rt >>>>>>>>>ua >>>>>>>>>lR >>>>>>>>>outerInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902) >>>>>>>>> at >>>>>>>>>com.cloud.network.element.VirtualRouterElement.implement(VirtualRo >>>>>>>>>ut >>>>>>>>>er >>>>>>>>>El >>>>>>>>>em >>>>>>>>>ent.java:175) >>>>>>>>> at >>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndRe >>>>>>>>>so >>>>>>>>>ur >>>>>>>>>ce >>>>>>>>>s( >>>>>>>>>NetworkManagerImpl.java:1518) >>>>>>>>> at >>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManag >>>>>>>>>er >>>>>>>>>Im >>>>>>>>>pl >>>>>>>>>.j >>>>>>>>>ava:1434) >>>>>>>>> at >>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Inte >>>>>>>>>rc >>>>>>>>>ep >>>>>>>>>to >>>>>>>>>rD >>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125) >>>>>>>>> at >>>>>>>>>com.cloud.network.NetworkManagerImpl.startNetwork(NetworkManagerIm >>>>>>>>>pl >>>>>>>>>.j >>>>>>>>>av >>>>>>>>>a: >>>>>>>>>2435) >>>>>>>>> at >>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startR >>>>>>>>>ou >>>>>>>>>te >>>>>>>>>r( >>>>>>>>>Vi >>>>>>>>>rtualNetworkApplianceManagerImpl.java:2855) >>>>>>>>> at >>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startR >>>>>>>>>ou >>>>>>>>>te >>>>>>>>>r( >>>>>>>>>Vi >>>>>>>>>rtualNetworkApplianceManagerImpl.java:2824) >>>>>>>>> at >>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Inte >>>>>>>>>rc >>>>>>>>>ep >>>>>>>>>to >>>>>>>>>rD >>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125) >>>>>>>>> at >>>>>>>>>org.apache.cloudstack.api.command.admin.router.StartRouterCmd.exec >>>>>>>>>ut >>>>>>>>>e( >>>>>>>>>St >>>>>>>>>ar >>>>>>>>>tRouterCmd.java:103) >>>>>>>>> >>>>>>>>> >>>>>>>>>table nics: >>>>>>>>> >>>>>>>>>mysql> select * from nics where reserver_name = >>>>>>>>>'ControlNetworkGuru'; >>>>>>>>>+----+--------------------------------------+-------------+------- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>----+---------------+-------------+-------------+---------+------- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>+------------+--------+--------------+----------+----------------- >>>>>>>>>-- >>>>>>>>>-+ >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>----------------------------------+-----------+------------------- >>>>>>>>>-- >>>>>>>>>+- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>----------+-------------+-------------+--------------------+------ >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>-------+---------------------+-------------+----------+ >>>>>>>>>| id | uuid | instance_id | >>>>>>>>>mac_address >>>>>>>>> | ip4_address | netmask | gateway | ip_type | >>>>>>>>>broadcast_uri >>>>>>>>>| network_id | mode | state | strategy | reserver_name >>>>>>>>> | >>>>>>>>>reservation_id | device_id | update_time >>>>>>>>> | >>>>>>>>>isolation_uri | ip6_address | default_nic | vm_type | >>>>>>>>>created >>>>>>>>> | removed | ip6_gateway | ip6_cidr | >>>>>>>>>+----+--------------------------------------+-------------+------- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>----+---------------+-------------+-------------+---------+------- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>+------------+--------+--------------+----------+----------------- >>>>>>>>>-- >>>>>>>>>-+ >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>----------------------------------+-----------+------------------- >>>>>>>>>-- >>>>>>>>>+- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>----------+-------------+-------------+--------------------+------ >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>-------+---------------------+-------------+----------+ >>>>>>>>>| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 | 1 | >>>>>>>>>0e:00:a9:fe:00:55 | 169.254.0.85 | 255.255.0.0 | 169.254.0.1 | >>>>>>>>>Ip4 >>>>>>>>>| >>>>>>>>>NULL | 202 | Static | Reserved | Start | >>>>>>>>>ControlNetworkGuru | 993864b4-9dde-47d6-8fd6-cf94050442c6 | >>>>>>>>>0 >>>>>>>>>| >>>>>>>>>2014-03-17 22:21:38 | NULL | NULL | 0 | >>>>>>>>>SecondaryStorageVm | 2013-09-06 12:44:42 | NULL | >>>>>>>>>NULL >>>>>>>>> | NULL | >>>>>>>>>| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 | 2 | >>>>>>>>>0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 | 169.254.0.1 | >>>>>>>>>Ip4 >>>>>>>>>| >>>>>>>>>NULL | 202 | Static | Reserved | Start | >>>>>>>>>ControlNetworkGuru | 852e0a65-c72a-448f-ac71-2bb3549a5a41 | >>>>>>>>>0 >>>>>>>>>| >>>>>>>>>2014-03-17 22:21:38 | NULL | NULL | 0 | >>>>>>>>>ConsoleProxy | 2013-09-06 12:44:42 | NULL | >>>>>>>>>NULL >>>>>>>>> | NULL | >>>>>>>>>| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 | 4 | NULL >>>>>>>>> | NULL | NULL | NULL | NULL | NULL >>>>>>>>>| 202 | Static | Deallocating | Start | >>>>>>>>>ControlNetworkGuru | >>>>>>>>>c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 | 2014-03-17 >>>>>>>>>12:27:58 | >>>>>>>>>NULL | NULL | 0 | DomainRouter | >>>>>>>>>2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL | NULL >>>>>>>>> | >>>>>>>>>| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc | 7 | NULL >>>>>>>>> | NULL | NULL | NULL | NULL | NULL >>>>>>>>>| 202 | Static | Deallocating | Start | >>>>>>>>>ControlNetworkGuru | >>>>>>>>>ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 | 2014-03-17 >>>>>>>>>12:27:52 | >>>>>>>>>NULL | NULL | 0 | DomainRouter | >>>>>>>>>2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL | NULL >>>>>>>>> | >>>>>>>>>| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 | 18 | NULL >>>>>>>>> | NULL | NULL | NULL | NULL | NULL >>>>>>>>>| 202 | Static | Deallocating | Start | >>>>>>>>>ControlNetworkGuru | >>>>>>>>>ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 | 2014-03-18 >>>>>>>>>09:11:16 | >>>>>>>>>NULL | NULL | 0 | DomainRouter | >>>>>>>>>2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL | NULL >>>>>>>>> | >>>>>>>>>| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d | 19 | NULL >>>>>>>>> | NULL | NULL | NULL | NULL | NULL >>>>>>>>>| 202 | Static | Allocated | Start | >>>>>>>>>ControlNetworkGuru | >>>>>>>>>e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 | 2014-03-18 >>>>>>>>>09:11:44 | >>>>>>>>>NULL | NULL | 0 | DomainRouter | >>>>>>>>>2014-03-18 08:11:32 | NULL | NULL | NULL >>>>>>>>> | >>>>>>>>>+----+--------------------------------------+-------------+------- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>----+---------------+-------------+-------------+---------+------- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>+------------+--------+--------------+----------+----------------- >>>>>>>>>-- >>>>>>>>>-+ >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>----------------------------------+-----------+------------------- >>>>>>>>>-- >>>>>>>>>+- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>----------+-------------+-------------+--------------------+------ >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>-- >>>>>>>>>-------+---------------------+-------------+----------+ >>>>>>>>> >>>>>>>>>