Hello,
as this is my first post to the list, I would like to thank all
contributors for Cloudstack which I use since last fall without any
problems. I run 4.1.1 with KVM and advanced networking.
After a restart of the management server (stopping and starting the java
process), the virtual domain router doesn't start and
management-server.log shows a NullPointerException in
NetworkModelImpl.getIpInNetwork (cf. stack trace below).
By putting the server in debug mode and remote debugging, I found out
that the reason is a row in the table nics which has NULL in ip (cf. row
with id 30 in the result of the select statement below).
What can I do to quickly solve this problem? Any pointers or suggestions
are appreciated as the system is currently unusable.
Thank you for your help
Kambiz
management-server.log:
2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-1:job-176) Asking Ovs to prepare for
Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare for
Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
2014-03-18 10:03:27,151 WARN [network.element.VpcVirtualRouterElement]
(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not associated with any
VPC
2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
2014-03-18 10:03:27,151 DEBUG [network.element.NiciraNvpElement]
(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle service
Connectivity on network net1
2014-03-18 10:03:27,153 DEBUG [cloud.network.NetworkModelImpl]
(Job-Executor-1:job-176) Service SecurityGroup is not supported in the network
id=205
2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-1:job-176) Lock is acquired for network id 202 as a part of
network implement
2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-1:job-176) Network id=202 is already implemented
2014-03-18 10:03:27,157 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-1:job-176) Lock is released for network id 202 as a part of
network implement
2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-1:job-176) Asking Ovs to prepare for
Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare for
Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
2014-03-18 10:03:27,187 WARN [network.element.VpcVirtualRouterElement]
(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is not associated with any
VPC
2014-03-18 10:03:27,188 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
2014-03-18 10:03:27,188 DEBUG [network.element.NiciraNvpElement]
(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle service
Connectivity on network null
2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl]
(Job-Executor-1:job-176) Checking if we need to prepare 1 volumes for
VM[DomainRouter|r-19-VM]
2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl]
(Job-Executor-1:job-176) No need to recreate the volume: Vol[24|vm=19|ROOT],
since it already has a pool assigned: 200, adding disk to VM
2014-03-18 10:03:27,224 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-1:job-176)
Boot Args for VM[DomainRouter|r-19-VM]: template=domP name=r-19-VM
eth2ip=10.193.17.190 eth2mask=255.255.255.0 gateway=10.193.17.1
eth0ip=10.124.99.1 eth0mask=255.255.255.0 domain=cs6cloud.internal
dhcprange=10.124.99.1 eth0ip=169.254.3.99 eth0mask=255.255.0.0 type=router
disable_rp_filter=true dns1=10.193.17.1
2014-03-18 10:03:27,343 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-1:job-176)
Found 8 ip(s) to apply as a part of domR VM[DomainRouter|r-19-VM] start.
2014-03-18 10:03:27,415 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-1:job-176)
Resending ipAssoc, port forwarding, load balancing rules as a part of Virtual
router start
2014-03-18 10:03:27,499 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-1:job-176)
Found 12 firewall Egress rule(s) to apply as a part of domR
VM[DomainRouter|r-19-VM] start.
2014-03-18 10:03:27,593 ERROR [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-1:job-176) Failed to start instance VM[DomainRouter|r-19-VM]
java.lang.NullPointerException
at
com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:763)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346)
at
com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
at
com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeployment(VirtualNetworkApplianceManagerImpl.java:2205)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:763)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2616)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1824)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(VirtualNetworkApplianceManagerImpl.java:1924)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902)
at
com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:175)
at
com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:1518)
at
com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl.java:1434)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
com.cloud.network.NetworkManagerImpl.startNetwork(NetworkManagerImpl.java:2435)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(VirtualNetworkApplianceManagerImpl.java:2855)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(VirtualNetworkApplianceManagerImpl.java:2824)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.admin.router.StartRouterCmd.execute(StartRouterCmd.java:103)
table nics:
mysql> select * from nics where reserver_name = 'ControlNetworkGuru';
+----+--------------------------------------+-------------+-------------------+---------------+-------------+-------------+---------+---------------+------------+--------+--------------+----------+--------------------+--------------------------------------+-----------+---------------------+---------------+-------------+-------------+--------------------+---------------------+---------------------+-------------+----------+
| id | uuid | instance_id | mac_address |
ip4_address | netmask | gateway | ip_type | broadcast_uri |
network_id | mode | state | strategy | reserver_name |
reservation_id | device_id | update_time |
isolation_uri | ip6_address | default_nic | vm_type | created
| removed | ip6_gateway | ip6_cidr |
+----+--------------------------------------+-------------+-------------------+---------------+-------------+-------------+---------+---------------+------------+--------+--------------+----------+--------------------+--------------------------------------+-----------+---------------------+---------------+-------------+-------------+--------------------+---------------------+---------------------+-------------+----------+
| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 | 1 | 0e:00:a9:fe:00:55 |
169.254.0.85 | 255.255.0.0 | 169.254.0.1 | Ip4 | NULL |
202 | Static | Reserved | Start | ControlNetworkGuru |
993864b4-9dde-47d6-8fd6-cf94050442c6 | 0 | 2014-03-17 22:21:38 | NULL
| NULL | 0 | SecondaryStorageVm | 2013-09-06 12:44:42 |
NULL | NULL | NULL |
| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 | 2 | 0e:00:a9:fe:02:d3 |
169.254.2.211 | 255.255.0.0 | 169.254.0.1 | Ip4 | NULL |
202 | Static | Reserved | Start | ControlNetworkGuru |
852e0a65-c72a-448f-ac71-2bb3549a5a41 | 0 | 2014-03-17 22:21:38 | NULL
| NULL | 0 | ConsoleProxy | 2013-09-06 12:44:42 |
NULL | NULL | NULL |
| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 | 4 | NULL |
NULL | NULL | NULL | NULL | NULL |
202 | Static | Deallocating | Start | ControlNetworkGuru |
c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 | 2014-03-17 12:27:58 | NULL
| NULL | 0 | DomainRouter | 2013-09-10 08:08:39 |
2014-03-17 11:27:58 | NULL | NULL |
| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc | 7 | NULL |
NULL | NULL | NULL | NULL | NULL |
202 | Static | Deallocating | Start | ControlNetworkGuru |
ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 | 2014-03-17 12:27:52 | NULL
| NULL | 0 | DomainRouter | 2013-09-12 10:58:03 |
2014-03-17 11:27:52 | NULL | NULL |
| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 | 18 | NULL |
NULL | NULL | NULL | NULL | NULL |
202 | Static | Deallocating | Start | ControlNetworkGuru |
ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 | 2014-03-18 09:11:16 | NULL
| NULL | 0 | DomainRouter | 2014-03-17 11:28:50 |
2014-03-18 08:11:16 | NULL | NULL |
| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d | 19 | NULL |
NULL | NULL | NULL | NULL | NULL |
202 | Static | Allocated | Start | ControlNetworkGuru |
e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 | 2014-03-18 09:11:44 | NULL
| NULL | 0 | DomainRouter | 2014-03-18 08:11:32 |
NULL | NULL | NULL |
+----+--------------------------------------+-------------+-------------------+---------------+-------------+-------------+---------+---------------+------------+--------+--------------+----------+--------------------+--------------------------------------+-----------+---------------------+---------------+-------------+-------------+--------------------+---------------------+---------------------+-------------+----------+
