The error happens not because Ip is null, but because the nic in a certain network can¹t be found. Looks like there is some bug in VPC nic plug/unplug for Guest networks process.
Kambiz, please do the following to fix it: 1) Stop the MS 2) Take the DB dump of cloud db in case you have to revert back. 3) Run the query: select * from router_network_ref where router_id=<id of your VR) and network_id not in (select network_id from nics where instance_id=<ID of your VR> and removed is null); It will give you the list of networks refs that somehow weren¹t cleaned during the nic detach. Remove the entry returned from router_network_ref table. Let me know how it works. -Alena. On 3/21/14, 3:36 PM, "Kambiz Darabi" <dar...@m-creations.com> wrote: >Hello, > >as this is my first post to the list, I would like to thank all >contributors for Cloudstack which I use since last fall without any >problems. I run 4.1.1 with KVM and advanced networking. > >After a restart of the management server (stopping and starting the java >process), the virtual domain router doesn't start and >management-server.log shows a NullPointerException in >NetworkModelImpl.getIpInNetwork (cf. stack trace below). > >By putting the server in debug mode and remote debugging, I found out >that the reason is a row in the table nics which has NULL in ip (cf. row >with id 30 in the result of the select statement below). > >What can I do to quickly solve this problem? Any pointers or suggestions >are appreciated as the system is currently unusable. > >Thank you for your help > > >Kambiz > > >management-server.log: > >2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl] >(Job-Executor-1:job-176) Asking VirtualRouter to prepare for >Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1] >2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl] >(Job-Executor-1:job-176) Asking Ovs to prepare for >Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1] >2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl] >(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare for >Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1] >2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl] >(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for >Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1] >2014-03-18 10:03:27,151 WARN [network.element.VpcVirtualRouterElement] >(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not associated with >any VPC >2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl] >(Job-Executor-1:job-176) Asking NiciraNvp to prepare for >Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1] >2014-03-18 10:03:27,151 DEBUG [network.element.NiciraNvpElement] >(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle service >Connectivity on network net1 >2014-03-18 10:03:27,153 DEBUG [cloud.network.NetworkModelImpl] >(Job-Executor-1:job-176) Service SecurityGroup is not supported in the >network id=205 >2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl] >(Job-Executor-1:job-176) Lock is acquired for network id 202 as a part of >network implement >2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl] >(Job-Executor-1:job-176) Network id=202 is already implemented >2014-03-18 10:03:27,157 DEBUG [cloud.network.NetworkManagerImpl] >(Job-Executor-1:job-176) Lock is released for network id 202 as a part of >network implement >2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl] >(Job-Executor-1:job-176) Asking VirtualRouter to prepare for >Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99] >2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl] >(Job-Executor-1:job-176) Asking Ovs to prepare for >Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99] >2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl] >(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare for >Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99] >2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl] >(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for >Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99] >2014-03-18 10:03:27,187 WARN [network.element.VpcVirtualRouterElement] >(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is not associated >with any VPC >2014-03-18 10:03:27,188 DEBUG [cloud.network.NetworkManagerImpl] >(Job-Executor-1:job-176) Asking NiciraNvp to prepare for >Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99] >2014-03-18 10:03:27,188 DEBUG [network.element.NiciraNvpElement] >(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle service >Connectivity on network null >2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl] >(Job-Executor-1:job-176) Checking if we need to prepare 1 volumes for >VM[DomainRouter|r-19-VM] >2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl] >(Job-Executor-1:job-176) No need to recreate the volume: >Vol[24|vm=19|ROOT], since it already has a pool assigned: 200, adding >disk to VM >2014-03-18 10:03:27,224 DEBUG >[network.router.VirtualNetworkApplianceManagerImpl] >(Job-Executor-1:job-176) Boot Args for VM[DomainRouter|r-19-VM]: >template=domP name=r-19-VM eth2ip=10.193.17.190 eth2mask=255.255.255.0 >gateway=10.193.17.1 eth0ip=10.124.99.1 eth0mask=255.255.255.0 >domain=cs6cloud.internal dhcprange=10.124.99.1 eth0ip=169.254.3.99 >eth0mask=255.255.0.0 type=router disable_rp_filter=true dns1=10.193.17.1 >2014-03-18 10:03:27,343 DEBUG >[network.router.VirtualNetworkApplianceManagerImpl] >(Job-Executor-1:job-176) Found 8 ip(s) to apply as a part of domR >VM[DomainRouter|r-19-VM] start. >2014-03-18 10:03:27,415 DEBUG >[network.router.VirtualNetworkApplianceManagerImpl] >(Job-Executor-1:job-176) Resending ipAssoc, port forwarding, load >balancing rules as a part of Virtual router start >2014-03-18 10:03:27,499 DEBUG >[network.router.VirtualNetworkApplianceManagerImpl] >(Job-Executor-1:job-176) Found 12 firewall Egress rule(s) to apply as a >part of domR VM[DomainRouter|r-19-VM] start. >2014-03-18 10:03:27,593 ERROR [cloud.vm.VirtualMachineManagerImpl] >(Job-Executor-1:job-176) Failed to start instance VM[DomainRouter|r-19-VM] >java.lang.NullPointerException > at >com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:76 >3) > at >com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetwor >kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346) > at >com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNet >workRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928) > at >com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeComman >dsOnStart(VirtualNetworkApplianceManagerImpl.java:2241) > at >com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCom >mandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767) > at >com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeploy >ment(VirtualNetworkApplianceManagerImpl.java:2205) > at >com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerI >mpl.java:763) > at >com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.jav >a:471) > at >com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualN >etworkApplianceManagerImpl.java:2616) > at >com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRo >uter(VirtualNetworkApplianceManagerImpl.java:1824) > at >com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(V >irtualNetworkApplianceManagerImpl.java:1924) > at >com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualR >outerInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902) > at >com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElem >ent.java:175) > at >com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources( >NetworkManagerImpl.java:1518) > at >com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl.j >ava:1434) > at >com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorD >ispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > at >com.cloud.network.NetworkManagerImpl.startNetwork(NetworkManagerImpl.java: >2435) > at >com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(Vi >rtualNetworkApplianceManagerImpl.java:2855) > at >com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(Vi >rtualNetworkApplianceManagerImpl.java:2824) > at >com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorD >ispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > at >org.apache.cloudstack.api.command.admin.router.StartRouterCmd.execute(Star >tRouterCmd.java:103) > > >table nics: > >mysql> select * from nics where reserver_name = 'ControlNetworkGuru'; >+----+--------------------------------------+-------------+--------------- >----+---------------+-------------+-------------+---------+--------------- >+------------+--------+--------------+----------+--------------------+---- >----------------------------------+-----------+---------------------+----- >----------+-------------+-------------+--------------------+-------------- >-------+---------------------+-------------+----------+ >| id | uuid | instance_id | mac_address > | ip4_address | netmask | gateway | ip_type | broadcast_uri >| network_id | mode | state | strategy | reserver_name | >reservation_id | device_id | update_time | >isolation_uri | ip6_address | default_nic | vm_type | created > | removed | ip6_gateway | ip6_cidr | >+----+--------------------------------------+-------------+--------------- >----+---------------+-------------+-------------+---------+--------------- >+------------+--------+--------------+----------+--------------------+---- >----------------------------------+-----------+---------------------+----- >----------+-------------+-------------+--------------------+-------------- >-------+---------------------+-------------+----------+ >| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 | 1 | >0e:00:a9:fe:00:55 | 169.254.0.85 | 255.255.0.0 | 169.254.0.1 | Ip4 | >NULL | 202 | Static | Reserved | Start | >ControlNetworkGuru | 993864b4-9dde-47d6-8fd6-cf94050442c6 | 0 | >2014-03-17 22:21:38 | NULL | NULL | 0 | >SecondaryStorageVm | 2013-09-06 12:44:42 | NULL | NULL > | NULL | >| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 | 2 | >0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 | 169.254.0.1 | Ip4 | >NULL | 202 | Static | Reserved | Start | >ControlNetworkGuru | 852e0a65-c72a-448f-ac71-2bb3549a5a41 | 0 | >2014-03-17 22:21:38 | NULL | NULL | 0 | >ConsoleProxy | 2013-09-06 12:44:42 | NULL | NULL > | NULL | >| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 | 4 | NULL > | NULL | NULL | NULL | NULL | NULL >| 202 | Static | Deallocating | Start | ControlNetworkGuru | >c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 | 2014-03-17 12:27:58 | >NULL | NULL | 0 | DomainRouter | >2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL | NULL | >| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc | 7 | NULL > | NULL | NULL | NULL | NULL | NULL >| 202 | Static | Deallocating | Start | ControlNetworkGuru | >ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 | 2014-03-17 12:27:52 | >NULL | NULL | 0 | DomainRouter | >2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL | NULL | >| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 | 18 | NULL > | NULL | NULL | NULL | NULL | NULL >| 202 | Static | Deallocating | Start | ControlNetworkGuru | >ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 | 2014-03-18 09:11:16 | >NULL | NULL | 0 | DomainRouter | >2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL | NULL | >| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d | 19 | NULL > | NULL | NULL | NULL | NULL | NULL >| 202 | Static | Allocated | Start | ControlNetworkGuru | >e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 | 2014-03-18 09:11:44 | >NULL | NULL | 0 | DomainRouter | >2014-03-18 08:11:32 | NULL | NULL | NULL | >+----+--------------------------------------+-------------+--------------- >----+---------------+-------------+-------------+---------+--------------- >+------------+--------+--------------+----------+--------------------+---- >----------------------------------+-----------+---------------------+----- >----------+-------------+-------------+--------------------+-------------- >-------+---------------------+-------------+----------+ > >
