Kambiz, the debug statements below are for the case when eth1 is a control interface as it was in your old command. I’ve looked at the new command, eth1 is not control, its either public or guest
eth0: - control iface eth0 inet static address 169.254.1.59 netmask 255.255.0.0 eth1: iface eth1 inet static address 10.193.17.1 Netmask So you need to execute the mysql statements for the traffic type of VR nic eth1 -Alena. On 3/25/14, 9:57 AM, "Alena Prokharchyk" <alena.prokharc...@citrix.com> wrote: >Kambiz, can you please check the following: > > >1) Check if the gateway is set on control network: > >mysql> select gateway, cidr from networks where traffic_type=‘Control’; > >2) For router control nic, check if network/gateway are set. > >Select gateway,netmask from nics where instance_id=<routerid> and >network_id=<control network id> > >-Alena. > >On 3/25/14, 5:47 AM, "Kambiz Darabi" <dar...@m-creations.com> wrote: > >>Hi, >> >>I looked up the startup command of the old router instance which worked >>correctly: >> >>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.sh -l >>r-7-VM -t all -d /var/lib/libvirt/images/r-7-VM-patchdisk -p >>%template=domP%name=r-7-VM%eth2ip=10.193.17.190%eth2mask=255.255.255.0%ga >>t >>eway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=cs6clou >>d >>.internal%dhcprange=10.124.99.1%eth1ip=169.254.2.46%eth1mask=255.255.0.0% >>t >>ype=router%disable_rp_filter=true%dns1=10.193.17.1 >> >>The new command (cf. below) doesn't have the parameters eth1ip and >>eth1mask. >> >>Thanks >> >> >>Kambiz >> >>Alena Prokharchyk <alena.prokharc...@citrix.com> wrote: >>> >>> I don’t think its relevant as the piece we’ve fixed, just eliminated >>> static nat rule programming for non-existing vm. Missing netmask on >>>eth1 >>> doesn’t seem related to the problem (although we have to figure out why >>> its missing), as the connection that fails, happening to link local >>>169.x >>> eth0 interface. >>> >>> Edison, can you please tell us how to debug link local connection >>>failure, >>> on KVM agent? >>> >>> Thank you, >>> Alena. >>> >>> On 3/24/14, 1:47 PM, "Kambiz Darabi" <dar...@m-creations.com> wrote: >>> >>>>Hi, >>>> >>>>thank you, the NullPointerException doesn't occur any more, but there >>>>still seems to be a problem during startup of the router. >>>> >>>>When I start the virtual router, it comes up, but in agent.log, there >>>>are lots of 'Could not connect to 169.254.1.x' messages. >>>> >>>>Then I logged into the virtual router to find out that the netmask of >>>>eth1 is missing in the interfaces file: >>>> >>>>root@host:~# virsh console r-19-VM >>>>Connected to domain r-19-VM >>>>Escape character is ^] >>>> >>>>Debian GNU/Linux 6.0 r-19-VM ttyS0 >>>> >>>>r-19-VM login: root >>>>... >>>>root@r-19-VM:~# cat /etc/network/interfaces >>>>auto lo eth0 eth1 eth2 >>>>iface lo inet loopback >>>> >>>>iface eth0 inet static >>>> address 169.254.1.59 >>>> netmask 255.255.0.0 >>>>iface eth1 inet static >>>> address 10.193.17.1 >>>> netmask >>>>iface eth2 inet static >>>> address 10.193.17.190 >>>> netmask 255.255.255.0 >>>> >>>>I don't know if it is relevant, but this is the line from agent.log >>>>where the parameters are visible: >>>> >>>>2014-03-24 21:36:17,681 DEBUG [kvm.resource.LibvirtComputingResource] >>>>(agentRequest-Handler-2:null) Executing: >>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.sh -l >>>>r-19-VM -t all -d /var/lib/libvirt/images/r-19-VM-patchdisk -p >>>>%template=domP%name=r-19-VM%eth2ip=10.193.17.190%eth2mask=255.255.255.0 >>>>% >>>>ga >>>>teway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=cs6c >>>>l >>>>ou >>>>d.internal%dhcprange=10.124.99.1%eth0ip=169.254.1.60%eth0mask=255.255.0 >>>>. >>>>0% >>>>type=router%disable_rp_filter=true%dns1=10.193.17.1 >>>> >>>> >>>>Any hint is appreciated. >>>> >>>>Thanks >>>> >>>> >>>>Kambiz >>>> >>>> >>>>Alena Prokharchyk <alena.prokharc...@citrix.com> wrote: >>>>> >>>>> Yes, Kambiz, you followed up right, and vm id=15 is the culprit. If >>>>>vm >>>>> id=15 is expunged, we have to clear out the reference to it from >>>>> user_ip_address table. Here is the flow: >>>>> >>>>> 1) Save the db dump. >>>>> 2) Run the query to cleanup the reference: >>>>> >>>>> Update user_ip_address set one_to_one_nat=0, instance_id=null where >>>>> id=<problematic public ip address id> >>>>> >>>>> >>>>> >>>>> Let me know how it works. >>>>> >>>>> -Alena. >>>>> >>>>> On 3/24/14, 10:55 AM, "Kambiz Darabi" <dar...@m-creations.com> wrote: >>>>> >>>>>>Hi, >>>>>> >>>>>>I hope I have understood what you wrote and created the following >>>>>>query >>>>>>correctly: >>>>>> >>>>>>select uip.vm_id, uip.network_id, uip.public_ip_address, >>>>>> n.state as nic_state, n.removed as nic_removed, >>>>>> vm.state as vm_state, vm.removed as vm_removed >>>>>>from user_ip_address uip >>>>>> join nics n on uip.vm_id = n.instance_id >>>>>> join vm_instance vm on uip.vm_id = vm.id >>>>>>where uip.id in (Select ip_address_id from firewall_rules fr where >>>>>>fr.network_id=205); >>>>>> >>>>>> >>>>>>+-------+------------+-------------------+--------------+------------ >>>>>>- >>>>>>-- >>>>>>-- >>>>>>----+-----------+------------+ >>>>>>| vm_id | network_id | public_ip_address | nic_state | nic_removed >>>>>> | vm_state | vm_removed | >>>>>>+-------+------------+-------------------+--------------+------------ >>>>>>- >>>>>>-- >>>>>>-- >>>>>>----+-----------+------------+ >>>>>>| 6 | 205 | 10.193.17.169 | Allocated | NULL >>>>>> | Stopped | NULL | >>>>>>| 10 | 205 | 10.193.17.136 | Allocated | NULL >>>>>> | Stopped | NULL | >>>>>>| 12 | 205 | 10.193.17.140 | Allocated | NULL >>>>>> | Stopped | NULL | >>>>>>| 13 | 205 | 10.193.17.141 | Allocated | NULL >>>>>> | Stopped | NULL | >>>>>>| 14 | 205 | 10.193.17.142 | Allocated | NULL >>>>>> | Stopped | NULL | >>>>>>| 15 | 205 | 10.193.17.174 | Deallocating | 2014-03-18 >>>>>>23:00:53 | Expunging | NULL | >>>>>>| 16 | 205 | 10.193.17.103 | Allocated | NULL >>>>>> | Stopped | NULL | >>>>>>+-------+------------+-------------------+--------------+------------ >>>>>>- >>>>>>-- >>>>>>-- >>>>>>----+-----------+------------+ >>>>>> >>>>>>Is VM id 15 what you are looking for? >>>>>> >>>>>>Thank you >>>>>> >>>>>> >>>>>>Kambiz >>>>>> >>>>>>Alena Prokharchyk <alena.prokharc...@citrix.com> wrote: >>>>>>> >>>>>>> Kambiz, can you please try one more thing. >>>>>>> >>>>>>> 1) Locate all the firewall rules for your guest network (205, >>>>>>>right?) >>>>>>> >>>>>>> Select id, ip_address_id from firewall_rules where network_id=205; >>>>>>> >>>>>>> 2) Now get all static nat enabled ip addresses for those rules: >>>>>>> >>>>>>> Select vm_id, network_id from user_ip_address where id in (Select >>>>>>>id, >>>>>>> ip_address_id from firewall_rules where network_id=205); >>>>>>> >>>>>>> For each vmId/networkId combo, check if there is non-removed nic >>>>>>>and >>>>>>> non-expunged vm. There might be some incorrect static nat ip/vm >>>>>>>reference >>>>>>> referring to vm that is removed already. If you find any, let me >>>>>>>know >>>>>>>and >>>>>>> I will tell you how to clean it up >>>>>>> >>>>>>> -Alena. >>>>>>> >>>>>>> On 3/22/14, 5:41 AM, "Kambiz Darabi" <dar...@m-creations.com> >>>>>>>wrote: >>>>>>> >>>>>>>>Hi Alena, >>>>>>>> >>>>>>>>thank you for your help. >>>>>>>> >>>>>>>>The query returns no rows, i.e. nics.removed was not null, but I >>>>>>>>removed >>>>>>>>the row though to see what happens: a new virtual router was >>>>>>>>created >>>>>>>>which also couldn't be started due to the same NPE. I reverted the >>>>>>>>change by restoring from the dump. >>>>>>>> >>>>>>>>I have to mention that prior to the restart, r-7-VM was the router >>>>>>>>which >>>>>>>>was used by my instances. I deleted the router using the UI after >>>>>>>>the >>>>>>>>first >>>>>>>>occurrence of the NPE, because a post with a similar problem >>>>>>>>suggested >>>>>>>>that the deleted router would be recreated again (and this >>>>>>>>procedure >>>>>>>>solved the problem). >>>>>>>> >>>>>>>>Below I have attached the state of the two tables. >>>>>>>> >>>>>>>>Anything else I can try? >>>>>>>> >>>>>>>>Thank you >>>>>>>> >>>>>>>> >>>>>>>>Kambiz >>>>>>>> >>>>>>>>mysql> select n.id, n.removed, n.ip4_address, n.netmask, n.gateway, >>>>>>>>n.ip_type, n.reserver_name, n.network_id, i.id as instance_id, >>>>>>>>i.name, >>>>>>>>i.state, i.type from vm_instance i join nics n on n.instance_id = >>>>>>>>i.id >>>>>>>>where i.type = 'DomainRouter'; >>>>>>>>+----+---------------------+---------------+---------------+------- >>>>>>>>- >>>>>>>>-- >>>>>>>>-- >>>>>>>>-+ >>>>>>>>---------+--------------------------+------------+-------------+--- >>>>>>>>- >>>>>>>>-- >>>>>>>>-- >>>>>>>>-+ >>>>>>>>-----------+--------------+ >>>>>>>>| id | removed | ip4_address | netmask | >>>>>>>>gateway >>>>>>>>| ip_type | reserver_name | network_id | instance_id | >>>>>>>>name >>>>>>>>| state | type | >>>>>>>>+----+---------------------+---------------+---------------+------- >>>>>>>>- >>>>>>>>-- >>>>>>>>-- >>>>>>>>-+ >>>>>>>>---------+--------------------------+------------+-------------+--- >>>>>>>>- >>>>>>>>-- >>>>>>>>-- >>>>>>>>-+ >>>>>>>>-----------+--------------+ >>>>>>>>| 9 | 2014-03-17 11:27:58 | 10.124.99.1 | 255.255.255.0 | NULL >>>>>>>>| NULL | ExternalGuestNetworkGuru | 204 | 4 | >>>>>>>>r-4-VM >>>>>>>>| Expunging | DomainRouter | >>>>>>>>| 10 | 2014-03-17 11:27:58 | NULL | NULL | NULL >>>>>>>>| NULL | ControlNetworkGuru | 202 | 4 | >>>>>>>>r-4-VM >>>>>>>>| Expunging | DomainRouter | >>>>>>>>| 11 | 2014-03-17 11:27:58 | 10.193.17.139 | 255.255.255.0 | >>>>>>>>10.193.17.1 >>>>>>>>| NULL | PublicNetworkGuru | 200 | 4 | >>>>>>>>r-4-VM >>>>>>>>| Expunging | DomainRouter | >>>>>>>>| 14 | 2014-03-17 11:27:52 | 10.124.99.1 | 255.255.255.0 | NULL >>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 7 | >>>>>>>>r-7-VM >>>>>>>>| Expunging | DomainRouter | >>>>>>>>| 15 | 2014-03-17 11:27:52 | NULL | NULL | NULL >>>>>>>>| NULL | ControlNetworkGuru | 202 | 7 | >>>>>>>>r-7-VM >>>>>>>>| Expunging | DomainRouter | >>>>>>>>| 16 | 2014-03-17 11:27:52 | 10.193.17.190 | 255.255.255.0 | >>>>>>>>10.193.17.1 >>>>>>>>| NULL | PublicNetworkGuru | 200 | 7 | >>>>>>>>r-7-VM >>>>>>>>| Expunging | DomainRouter | >>>>>>>>| 26 | 2014-03-18 08:11:16 | 10.124.99.1 | 255.255.255.0 | NULL >>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 18 | >>>>>>>>r-18-VM >>>>>>>>| Expunging | DomainRouter | >>>>>>>>| 27 | 2014-03-18 08:11:16 | NULL | NULL | NULL >>>>>>>>| NULL | ControlNetworkGuru | 202 | 18 | >>>>>>>>r-18-VM >>>>>>>>| Expunging | DomainRouter | >>>>>>>>| 28 | 2014-03-18 08:11:16 | 10.193.17.190 | 255.255.255.0 | >>>>>>>>10.193.17.1 >>>>>>>>| NULL | PublicNetworkGuru | 200 | 18 | >>>>>>>>r-18-VM >>>>>>>>| Expunging | DomainRouter | >>>>>>>>| 29 | NULL | 10.124.99.1 | 255.255.255.0 | NULL >>>>>>>>| NULL | ExternalGuestNetworkGuru | 205 | 19 | >>>>>>>>r-19-VM >>>>>>>>| Stopped | DomainRouter | >>>>>>>>| 30 | NULL | NULL | NULL | NULL >>>>>>>>| NULL | ControlNetworkGuru | 202 | 19 | >>>>>>>>r-19-VM >>>>>>>>| Stopped | DomainRouter | >>>>>>>>| 31 | NULL | 10.193.17.190 | 255.255.255.0 | >>>>>>>>10.193.17.1 >>>>>>>>| NULL | PublicNetworkGuru | 200 | 19 | >>>>>>>>r-19-VM >>>>>>>>| Stopped | DomainRouter | >>>>>>>>+----+---------------------+---------------+---------------+------- >>>>>>>>- >>>>>>>>-- >>>>>>>>-- >>>>>>>>-+ >>>>>>>>---------+--------------------------+------------+-------------+--- >>>>>>>>- >>>>>>>>-- >>>>>>>>-- >>>>>>>>-+ >>>>>>>>-----------+--------------+ >>>>>>>> >>>>>>>>mysql> select * from router_network_ref; >>>>>>>>+----+-----------+------------+------------+ >>>>>>>>| id | router_id | network_id | guest_type | >>>>>>>>+----+-----------+------------+------------+ >>>>>>>>| 1 | 4 | 204 | Isolated | >>>>>>>>| 2 | 7 | 205 | Isolated | >>>>>>>>| 3 | 18 | 205 | Isolated | >>>>>>>>| 4 | 19 | 205 | Isolated | >>>>>>>>+----+-----------+------------+------------+ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>Alena Prokharchyk <alena.prokharc...@citrix.com> wrote: >>>>>>>>> >>>>>>>>> The error happens not because Ip is null, but because the nic in >>>>>>>>>a >>>>>>>>>certain >>>>>>>>> network can¹t be found. Looks like there is some bug in VPC nic >>>>>>>>> plug/unplug for Guest networks process. >>>>>>>>> >>>>>>>>> Kambiz, please do the following to fix it: >>>>>>>>> >>>>>>>>> 1) Stop the MS >>>>>>>>> 2) Take the DB dump of cloud db in case you have to revert back. >>>>>>>>> 3) Run the query: >>>>>>>>> >>>>>>>>> select * from router_network_ref where router_id=<id of your VR) >>>>>>>>>and >>>>>>>>> network_id not in (select network_id from nics where >>>>>>>>>instance_id=<ID >>>>>>>>>of >>>>>>>>> your VR> and removed is null); >>>>>>>>> >>>>>>>>> It will give you the list of networks refs that somehow weren¹t >>>>>>>>>cleaned >>>>>>>>> during the nic detach. Remove the entry returned from >>>>>>>>>router_network_ref >>>>>>>>> table. >>>>>>>>> >>>>>>>>> Let me know how it works. >>>>>>>>> >>>>>>>>> -Alena. >>>>>>>>> >>>>>>>>> >>>>>>>>> On 3/21/14, 3:36 PM, "Kambiz Darabi" <dar...@m-creations.com> >>>>>>>>>wrote: >>>>>>>>> >>>>>>>>>>Hello, >>>>>>>>>> >>>>>>>>>>as this is my first post to the list, I would like to thank all >>>>>>>>>>contributors for Cloudstack which I use since last fall without >>>>>>>>>>any >>>>>>>>>>problems. I run 4.1.1 with KVM and advanced networking. >>>>>>>>>> >>>>>>>>>>After a restart of the management server (stopping and starting >>>>>>>>>>the >>>>>>>>>>java >>>>>>>>>>process), the virtual domain router doesn't start and >>>>>>>>>>management-server.log shows a NullPointerException in >>>>>>>>>>NetworkModelImpl.getIpInNetwork (cf. stack trace below). >>>>>>>>>> >>>>>>>>>>By putting the server in debug mode and remote debugging, I found >>>>>>>>>>out >>>>>>>>>>that the reason is a row in the table nics which has NULL in ip >>>>>>>>>>(cf. >>>>>>>>>>row >>>>>>>>>>with id 30 in the result of the select statement below). >>>>>>>>>> >>>>>>>>>>What can I do to quickly solve this problem? Any pointers or >>>>>>>>>>suggestions >>>>>>>>>>are appreciated as the system is currently unusable. >>>>>>>>>> >>>>>>>>>>Thank you for your help >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>Kambiz >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>management-server.log: >>>>>>>>>> >>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for >>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1] >>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for >>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1] >>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare >>>>>>>>>>for >>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1] >>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for >>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1] >>>>>>>>>>2014-03-18 10:03:27,151 WARN >>>>>>>>>>[network.element.VpcVirtualRouterElement] >>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not >>>>>>>>>>associated >>>>>>>>>>with >>>>>>>>>>any VPC >>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for >>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1] >>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [network.element.NiciraNvpElement] >>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle >>>>>>>>>>service >>>>>>>>>>Connectivity on network net1 >>>>>>>>>>2014-03-18 10:03:27,153 DEBUG [cloud.network.NetworkModelImpl] >>>>>>>>>>(Job-Executor-1:job-176) Service SecurityGroup is not supported >>>>>>>>>>in >>>>>>>>>>the >>>>>>>>>>network id=205 >>>>>>>>>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>>>(Job-Executor-1:job-176) Lock is acquired for network id 202 as a >>>>>>>>>>part >>>>>>>>>>of >>>>>>>>>>network implement >>>>>>>>>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>>>(Job-Executor-1:job-176) Network id=202 is already implemented >>>>>>>>>>2014-03-18 10:03:27,157 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>>>(Job-Executor-1:job-176) Lock is released for network id 202 as a >>>>>>>>>>part >>>>>>>>>>of >>>>>>>>>>network implement >>>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for >>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99] >>>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for >>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99] >>>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare >>>>>>>>>>for >>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99] >>>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for >>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99] >>>>>>>>>>2014-03-18 10:03:27,187 WARN >>>>>>>>>>[network.element.VpcVirtualRouterElement] >>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is not >>>>>>>>>>associated >>>>>>>>>>with any VPC >>>>>>>>>>2014-03-18 10:03:27,188 DEBUG [cloud.network.NetworkManagerImpl] >>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for >>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99] >>>>>>>>>>2014-03-18 10:03:27,188 DEBUG [network.element.NiciraNvpElement] >>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle >>>>>>>>>>service >>>>>>>>>>Connectivity on network null >>>>>>>>>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl] >>>>>>>>>>(Job-Executor-1:job-176) Checking if we need to prepare 1 volumes >>>>>>>>>>for >>>>>>>>>>VM[DomainRouter|r-19-VM] >>>>>>>>>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl] >>>>>>>>>>(Job-Executor-1:job-176) No need to recreate the volume: >>>>>>>>>>Vol[24|vm=19|ROOT], since it already has a pool assigned: 200, >>>>>>>>>>adding >>>>>>>>>>disk to VM >>>>>>>>>>2014-03-18 10:03:27,224 DEBUG >>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl] >>>>>>>>>>(Job-Executor-1:job-176) Boot Args for VM[DomainRouter|r-19-VM]: >>>>>>>>>>template=domP name=r-19-VM eth2ip=10.193.17.190 >>>>>>>>>>eth2mask=255.255.255.0 >>>>>>>>>>gateway=10.193.17.1 eth0ip=10.124.99.1 eth0mask=255.255.255.0 >>>>>>>>>>domain=cs6cloud.internal dhcprange=10.124.99.1 >>>>>>>>>>eth0ip=169.254.3.99 >>>>>>>>>>eth0mask=255.255.0.0 type=router disable_rp_filter=true >>>>>>>>>>dns1=10.193.17.1 >>>>>>>>>>2014-03-18 10:03:27,343 DEBUG >>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl] >>>>>>>>>>(Job-Executor-1:job-176) Found 8 ip(s) to apply as a part of domR >>>>>>>>>>VM[DomainRouter|r-19-VM] start. >>>>>>>>>>2014-03-18 10:03:27,415 DEBUG >>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl] >>>>>>>>>>(Job-Executor-1:job-176) Resending ipAssoc, port forwarding, load >>>>>>>>>>balancing rules as a part of Virtual router start >>>>>>>>>>2014-03-18 10:03:27,499 DEBUG >>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl] >>>>>>>>>>(Job-Executor-1:job-176) Found 12 firewall Egress rule(s) to >>>>>>>>>>apply >>>>>>>>>>as >>>>>>>>>>a >>>>>>>>>>part of domR VM[DomainRouter|r-19-VM] start. >>>>>>>>>>2014-03-18 10:03:27,593 ERROR >>>>>>>>>>[cloud.vm.VirtualMachineManagerImpl] >>>>>>>>>>(Job-Executor-1:job-176) Failed to start instance >>>>>>>>>>VM[DomainRouter|r-19-VM] >>>>>>>>>>java.lang.NullPointerException >>>>>>>>>> at >>>>>>>>>>com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImp >>>>>>>>>>l >>>>>>>>>>.j >>>>>>>>>>av >>>>>>>>>>a: >>>>>>>>>>76 >>>>>>>>>>3) >>>>>>>>>> at >>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.final >>>>>>>>>>i >>>>>>>>>>ze >>>>>>>>>>Ne >>>>>>>>>>tw >>>>>>>>>>or >>>>>>>>>>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346) >>>>>>>>>> at >>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.fi >>>>>>>>>>n >>>>>>>>>>al >>>>>>>>>>iz >>>>>>>>>>eN >>>>>>>>>>et >>>>>>>>>>workRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:92 >>>>>>>>>>8 >>>>>>>>>>) >>>>>>>>>> at >>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.final >>>>>>>>>>i >>>>>>>>>>ze >>>>>>>>>>Co >>>>>>>>>>mm >>>>>>>>>>an >>>>>>>>>>dsOnStart(VirtualNetworkApplianceManagerImpl.java:2241) >>>>>>>>>> at >>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.fi >>>>>>>>>>n >>>>>>>>>>al >>>>>>>>>>iz >>>>>>>>>>eC >>>>>>>>>>om >>>>>>>>>>mandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767) >>>>>>>>>> at >>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.final >>>>>>>>>>i >>>>>>>>>>ze >>>>>>>>>>De >>>>>>>>>>pl >>>>>>>>>>oy >>>>>>>>>>ment(VirtualNetworkApplianceManagerImpl.java:2205) >>>>>>>>>> at >>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachin >>>>>>>>>>e >>>>>>>>>>Ma >>>>>>>>>>na >>>>>>>>>>ge >>>>>>>>>>rI >>>>>>>>>>mpl.java:763) >>>>>>>>>> at >>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManage >>>>>>>>>>r >>>>>>>>>>Im >>>>>>>>>>pl >>>>>>>>>>.j >>>>>>>>>>av >>>>>>>>>>a:471) >>>>>>>>>> at >>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start >>>>>>>>>>( >>>>>>>>>>Vi >>>>>>>>>>rt >>>>>>>>>>ua >>>>>>>>>>lN >>>>>>>>>>etworkApplianceManagerImpl.java:2616) >>>>>>>>>> at >>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start >>>>>>>>>>V >>>>>>>>>>ir >>>>>>>>>>tu >>>>>>>>>>al >>>>>>>>>>Ro >>>>>>>>>>uter(VirtualNetworkApplianceManagerImpl.java:1824) >>>>>>>>>> at >>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start >>>>>>>>>>R >>>>>>>>>>ou >>>>>>>>>>te >>>>>>>>>>rs >>>>>>>>>>(V >>>>>>>>>>irtualNetworkApplianceManagerImpl.java:1924) >>>>>>>>>> at >>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deplo >>>>>>>>>>y >>>>>>>>>>Vi >>>>>>>>>>rt >>>>>>>>>>ua >>>>>>>>>>lR >>>>>>>>>>outerInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902) >>>>>>>>>> at >>>>>>>>>>com.cloud.network.element.VirtualRouterElement.implement(VirtualR >>>>>>>>>>o >>>>>>>>>>ut >>>>>>>>>>er >>>>>>>>>>El >>>>>>>>>>em >>>>>>>>>>ent.java:175) >>>>>>>>>> at >>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndR >>>>>>>>>>e >>>>>>>>>>so >>>>>>>>>>ur >>>>>>>>>>ce >>>>>>>>>>s( >>>>>>>>>>NetworkManagerImpl.java:1518) >>>>>>>>>> at >>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkMana >>>>>>>>>>g >>>>>>>>>>er >>>>>>>>>>Im >>>>>>>>>>pl >>>>>>>>>>.j >>>>>>>>>>ava:1434) >>>>>>>>>> at >>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Int >>>>>>>>>>e >>>>>>>>>>rc >>>>>>>>>>ep >>>>>>>>>>to >>>>>>>>>>rD >>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125) >>>>>>>>>> at >>>>>>>>>>com.cloud.network.NetworkManagerImpl.startNetwork(NetworkManagerI >>>>>>>>>>m >>>>>>>>>>pl >>>>>>>>>>.j >>>>>>>>>>av >>>>>>>>>>a: >>>>>>>>>>2435) >>>>>>>>>> at >>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start >>>>>>>>>>R >>>>>>>>>>ou >>>>>>>>>>te >>>>>>>>>>r( >>>>>>>>>>Vi >>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2855) >>>>>>>>>> at >>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start >>>>>>>>>>R >>>>>>>>>>ou >>>>>>>>>>te >>>>>>>>>>r( >>>>>>>>>>Vi >>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2824) >>>>>>>>>> at >>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Int >>>>>>>>>>e >>>>>>>>>>rc >>>>>>>>>>ep >>>>>>>>>>to >>>>>>>>>>rD >>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125) >>>>>>>>>> at >>>>>>>>>>org.apache.cloudstack.api.command.admin.router.StartRouterCmd.exe >>>>>>>>>>c >>>>>>>>>>ut >>>>>>>>>>e( >>>>>>>>>>St >>>>>>>>>>ar >>>>>>>>>>tRouterCmd.java:103) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>table nics: >>>>>>>>>> >>>>>>>>>>mysql> select * from nics where reserver_name = >>>>>>>>>>'ControlNetworkGuru'; >>>>>>>>>>+----+--------------------------------------+-------------+------ >>>>>>>>>>- >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>----+---------------+-------------+-------------+---------+------ >>>>>>>>>>- >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>+------------+--------+--------------+----------+---------------- >>>>>>>>>>- >>>>>>>>>>-- >>>>>>>>>>-+ >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>----------------------------------+-----------+------------------ >>>>>>>>>>- >>>>>>>>>>-- >>>>>>>>>>+- >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>----------+-------------+-------------+--------------------+----- >>>>>>>>>>- >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>-------+---------------------+-------------+----------+ >>>>>>>>>>| id | uuid | instance_id | >>>>>>>>>>mac_address >>>>>>>>>> | ip4_address | netmask | gateway | ip_type | >>>>>>>>>>broadcast_uri >>>>>>>>>>| network_id | mode | state | strategy | reserver_name >>>>>>>>>> | >>>>>>>>>>reservation_id | device_id | update_time >>>>>>>>>> | >>>>>>>>>>isolation_uri | ip6_address | default_nic | vm_type | >>>>>>>>>>created >>>>>>>>>> | removed | ip6_gateway | ip6_cidr | >>>>>>>>>>+----+--------------------------------------+-------------+------ >>>>>>>>>>- >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>----+---------------+-------------+-------------+---------+------ >>>>>>>>>>- >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>+------------+--------+--------------+----------+---------------- >>>>>>>>>>- >>>>>>>>>>-- >>>>>>>>>>-+ >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>----------------------------------+-----------+------------------ >>>>>>>>>>- >>>>>>>>>>-- >>>>>>>>>>+- >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>----------+-------------+-------------+--------------------+----- >>>>>>>>>>- >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>-------+---------------------+-------------+----------+ >>>>>>>>>>| 2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 | 1 | >>>>>>>>>>0e:00:a9:fe:00:55 | 169.254.0.85 | 255.255.0.0 | 169.254.0.1 | >>>>>>>>>>Ip4 >>>>>>>>>>| >>>>>>>>>>NULL | 202 | Static | Reserved | Start | >>>>>>>>>>ControlNetworkGuru | 993864b4-9dde-47d6-8fd6-cf94050442c6 | >>>>>>>>>>0 >>>>>>>>>>| >>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL | 0 | >>>>>>>>>>SecondaryStorageVm | 2013-09-06 12:44:42 | NULL | >>>>>>>>>>NULL >>>>>>>>>> | NULL | >>>>>>>>>>| 6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 | 2 | >>>>>>>>>>0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 | 169.254.0.1 | >>>>>>>>>>Ip4 >>>>>>>>>>| >>>>>>>>>>NULL | 202 | Static | Reserved | Start | >>>>>>>>>>ControlNetworkGuru | 852e0a65-c72a-448f-ac71-2bb3549a5a41 | >>>>>>>>>>0 >>>>>>>>>>| >>>>>>>>>>2014-03-17 22:21:38 | NULL | NULL | 0 | >>>>>>>>>>ConsoleProxy | 2013-09-06 12:44:42 | NULL | >>>>>>>>>>NULL >>>>>>>>>> | NULL | >>>>>>>>>>| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 | 4 | NULL >>>>>>>>>> | NULL | NULL | NULL | NULL | NULL >>>>>>>>>>| 202 | Static | Deallocating | Start | >>>>>>>>>>ControlNetworkGuru | >>>>>>>>>>c28e8ddc-c106-462e-96c8-5d5216dad9b7 | 1 | 2014-03-17 >>>>>>>>>>12:27:58 | >>>>>>>>>>NULL | NULL | 0 | DomainRouter | >>>>>>>>>>2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL | NULL >>>>>>>>>> | >>>>>>>>>>| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc | 7 | NULL >>>>>>>>>> | NULL | NULL | NULL | NULL | NULL >>>>>>>>>>| 202 | Static | Deallocating | Start | >>>>>>>>>>ControlNetworkGuru | >>>>>>>>>>ca1aa99e-e630-4533-9642-523d8a8b1fea | 1 | 2014-03-17 >>>>>>>>>>12:27:52 | >>>>>>>>>>NULL | NULL | 0 | DomainRouter | >>>>>>>>>>2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL | NULL >>>>>>>>>> | >>>>>>>>>>| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 | 18 | NULL >>>>>>>>>> | NULL | NULL | NULL | NULL | NULL >>>>>>>>>>| 202 | Static | Deallocating | Start | >>>>>>>>>>ControlNetworkGuru | >>>>>>>>>>ad8e0e50-72aa-4c68-8634-8dc89f12fe01 | 1 | 2014-03-18 >>>>>>>>>>09:11:16 | >>>>>>>>>>NULL | NULL | 0 | DomainRouter | >>>>>>>>>>2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL | NULL >>>>>>>>>> | >>>>>>>>>>| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d | 19 | NULL >>>>>>>>>> | NULL | NULL | NULL | NULL | NULL >>>>>>>>>>| 202 | Static | Allocated | Start | >>>>>>>>>>ControlNetworkGuru | >>>>>>>>>>e81ba56d-a101-4c60-b44f-a0890d56aad9 | 1 | 2014-03-18 >>>>>>>>>>09:11:44 | >>>>>>>>>>NULL | NULL | 0 | DomainRouter | >>>>>>>>>>2014-03-18 08:11:32 | NULL | NULL | NULL >>>>>>>>>> | >>>>>>>>>>+----+--------------------------------------+-------------+------ >>>>>>>>>>- >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>----+---------------+-------------+-------------+---------+------ >>>>>>>>>>- >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>+------------+--------+--------------+----------+---------------- >>>>>>>>>>- >>>>>>>>>>-- >>>>>>>>>>-+ >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>----------------------------------+-----------+------------------ >>>>>>>>>>- >>>>>>>>>>-- >>>>>>>>>>+- >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>----------+-------------+-------------+--------------------+----- >>>>>>>>>>- >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>-- >>>>>>>>>>-------+---------------------+-------------+----------+ >>>>>>>>>> >>>>>>>>>> >
