Kambiz, can you please check the following:

1) Check if the gateway is set on control network:

mysql> select gateway, cidr from networks where traffic_type=‘Control’;

2) For router control nic, check if network/gateway are set.

Select gateway,netmask from nics where instance_id=<routerid> and
network_id=<control network id>

-Alena.

On 3/25/14, 5:47 AM, "Kambiz Darabi" <dar...@m-creations.com> wrote:

>Hi,
>
>I looked up the startup command of the old router instance which worked
>correctly:
>
>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.sh -l
>r-7-VM -t all -d /var/lib/libvirt/images/r-7-VM-patchdisk -p
>%template=domP%name=r-7-VM%eth2ip=10.193.17.190%eth2mask=255.255.255.0%gat
>eway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=cs6cloud
>.internal%dhcprange=10.124.99.1%eth1ip=169.254.2.46%eth1mask=255.255.0.0%t
>ype=router%disable_rp_filter=true%dns1=10.193.17.1
>
>The new command (cf. below) doesn't have the parameters eth1ip and
>eth1mask.
>
>Thanks
>
>
>Kambiz
>
>Alena Prokharchyk <alena.prokharc...@citrix.com> wrote:
>> 
>> I don’t think its relevant as the piece we’ve fixed, just eliminated
>> static nat rule programming for non-existing vm. Missing netmask on eth1
>> doesn’t seem related to the problem (although we have to figure out why
>> its missing), as the connection that fails, happening to link local
>>169.x
>> eth0 interface.
>>
>> Edison, can you please tell us how to debug link local connection
>>failure,
>> on KVM agent?
>>
>> Thank you,
>> Alena.
>>
>> On 3/24/14, 1:47 PM, "Kambiz Darabi" <dar...@m-creations.com> wrote:
>>
>>>Hi,
>>>
>>>thank you, the NullPointerException doesn't occur any more, but there
>>>still seems to be a problem during startup of the router.
>>>
>>>When I start the virtual router, it comes up, but in agent.log, there
>>>are lots of 'Could not connect to 169.254.1.x'  messages.
>>>
>>>Then I logged into the virtual router to find out that the netmask of
>>>eth1 is missing in the interfaces file:
>>>
>>>root@host:~# virsh console r-19-VM
>>>Connected to domain r-19-VM
>>>Escape character is ^]
>>>
>>>Debian GNU/Linux 6.0 r-19-VM ttyS0
>>>
>>>r-19-VM login: root
>>>...
>>>root@r-19-VM:~# cat /etc/network/interfaces
>>>auto lo eth0 eth1 eth2
>>>iface lo inet loopback
>>>
>>>iface  eth0 inet static
>>>  address 169.254.1.59
>>>  netmask 255.255.0.0
>>>iface  eth1 inet static
>>>  address 10.193.17.1
>>>  netmask 
>>>iface  eth2 inet static
>>>  address 10.193.17.190
>>>  netmask 255.255.255.0
>>>
>>>I don't know if it is relevant, but this is the line from agent.log
>>>where the parameters are visible:
>>>
>>>2014-03-24 21:36:17,681 DEBUG [kvm.resource.LibvirtComputingResource]
>>>(agentRequest-Handler-2:null) Executing:
>>>/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/rundomrpre.sh -l
>>>r-19-VM -t all -d /var/lib/libvirt/images/r-19-VM-patchdisk -p
>>>%template=domP%name=r-19-VM%eth2ip=10.193.17.190%eth2mask=255.255.255.0%
>>>ga
>>>teway=10.193.17.1%eth0ip=10.124.99.1%eth0mask=255.255.255.0%domain=cs6cl
>>>ou
>>>d.internal%dhcprange=10.124.99.1%eth0ip=169.254.1.60%eth0mask=255.255.0.
>>>0%
>>>type=router%disable_rp_filter=true%dns1=10.193.17.1
>>>
>>>
>>>Any hint is appreciated.
>>>
>>>Thanks
>>>
>>>
>>>Kambiz
>>>
>>>
>>>Alena Prokharchyk <alena.prokharc...@citrix.com> wrote:
>>>> 
>>>> Yes, Kambiz, you followed up right, and vm id=15 is the culprit. If vm
>>>> id=15 is expunged, we have to clear out the reference to it from
>>>> user_ip_address table. Here is the flow:
>>>>
>>>> 1) Save the db dump.
>>>> 2) Run the query to cleanup the reference:
>>>>
>>>> Update user_ip_address set one_to_one_nat=0, instance_id=null where
>>>> id=<problematic public ip address id>
>>>>
>>>>
>>>>
>>>> Let me know how it works.
>>>>
>>>> -Alena.
>>>>
>>>> On 3/24/14, 10:55 AM, "Kambiz Darabi" <dar...@m-creations.com> wrote:
>>>>
>>>>>Hi,
>>>>>
>>>>>I hope I have understood what you wrote and created the following
>>>>>query
>>>>>correctly:
>>>>>
>>>>>select uip.vm_id, uip.network_id, uip.public_ip_address,
>>>>>       n.state as nic_state, n.removed as nic_removed,
>>>>>       vm.state as vm_state, vm.removed as vm_removed
>>>>>from user_ip_address uip
>>>>>     join nics n on uip.vm_id = n.instance_id
>>>>>     join vm_instance vm on uip.vm_id = vm.id
>>>>>where uip.id in (Select ip_address_id from firewall_rules fr where
>>>>>fr.network_id=205);
>>>>>
>>>>>
>>>>>+-------+------------+-------------------+--------------+-------------
>>>>>--
>>>>>--
>>>>>----+-----------+------------+
>>>>>| vm_id | network_id | public_ip_address | nic_state    | nic_removed
>>>>>    | vm_state  | vm_removed |
>>>>>+-------+------------+-------------------+--------------+-------------
>>>>>--
>>>>>--
>>>>>----+-----------+------------+
>>>>>|     6 |        205 | 10.193.17.169     | Allocated    | NULL
>>>>>    | Stopped   | NULL       |
>>>>>|    10 |        205 | 10.193.17.136     | Allocated    | NULL
>>>>>    | Stopped   | NULL       |
>>>>>|    12 |        205 | 10.193.17.140     | Allocated    | NULL
>>>>>    | Stopped   | NULL       |
>>>>>|    13 |        205 | 10.193.17.141     | Allocated    | NULL
>>>>>    | Stopped   | NULL       |
>>>>>|    14 |        205 | 10.193.17.142     | Allocated    | NULL
>>>>>    | Stopped   | NULL       |
>>>>>|    15 |        205 | 10.193.17.174     | Deallocating | 2014-03-18
>>>>>23:00:53 | Expunging | NULL       |
>>>>>|    16 |        205 | 10.193.17.103     | Allocated    | NULL
>>>>>    | Stopped   | NULL       |
>>>>>+-------+------------+-------------------+--------------+-------------
>>>>>--
>>>>>--
>>>>>----+-----------+------------+
>>>>>
>>>>>Is VM id 15 what you are looking for?
>>>>>
>>>>>Thank you
>>>>>
>>>>>
>>>>>Kambiz
>>>>>
>>>>>Alena Prokharchyk <alena.prokharc...@citrix.com> wrote:
>>>>>> 
>>>>>> Kambiz, can you please try one more thing.
>>>>>>
>>>>>> 1) Locate all the firewall rules for your guest network (205,
>>>>>>right?)
>>>>>>
>>>>>> Select id, ip_address_id from firewall_rules where network_id=205;
>>>>>>
>>>>>> 2) Now get all static nat enabled ip addresses for those rules:
>>>>>>
>>>>>> Select vm_id, network_id from user_ip_address where id in (Select
>>>>>>id,
>>>>>> ip_address_id from firewall_rules where network_id=205);
>>>>>>
>>>>>> For each vmId/networkId combo, check if there is non-removed nic and
>>>>>> non-expunged vm. There might be some incorrect static nat ip/vm
>>>>>>reference
>>>>>> referring to vm that is removed already. If you find any, let me
>>>>>>know
>>>>>>and
>>>>>> I will tell you how to clean it up
>>>>>>
>>>>>> -Alena.
>>>>>>
>>>>>> On 3/22/14, 5:41 AM, "Kambiz Darabi" <dar...@m-creations.com> wrote:
>>>>>>
>>>>>>>Hi Alena,
>>>>>>>
>>>>>>>thank you for your help.
>>>>>>>
>>>>>>>The query returns no rows, i.e. nics.removed was not null, but I
>>>>>>>removed
>>>>>>>the row though to see what happens: a new virtual router was created
>>>>>>>which also couldn't be started due to the same NPE. I reverted the
>>>>>>>change by restoring from the dump.
>>>>>>>
>>>>>>>I have to mention that prior to the restart, r-7-VM was the router
>>>>>>>which
>>>>>>>was used by my instances. I deleted the router using the UI after
>>>>>>>the
>>>>>>>first
>>>>>>>occurrence of the NPE, because a post with a similar problem
>>>>>>>suggested
>>>>>>>that the deleted router would be recreated again (and this procedure
>>>>>>>solved the problem).
>>>>>>>
>>>>>>>Below I have attached the state of the two tables.
>>>>>>>
>>>>>>>Anything else I can try?
>>>>>>>
>>>>>>>Thank you
>>>>>>>
>>>>>>>
>>>>>>>Kambiz
>>>>>>>
>>>>>>>mysql> select n.id, n.removed, n.ip4_address, n.netmask, n.gateway,
>>>>>>>n.ip_type, n.reserver_name, n.network_id, i.id as instance_id,
>>>>>>>i.name,
>>>>>>>i.state, i.type from vm_instance i join nics n on n.instance_id =
>>>>>>>i.id
>>>>>>>where i.type = 'DomainRouter';
>>>>>>>+----+---------------------+---------------+---------------+--------
>>>>>>>--
>>>>>>>--
>>>>>>>-+
>>>>>>>---------+--------------------------+------------+-------------+----
>>>>>>>--
>>>>>>>--
>>>>>>>-+
>>>>>>>-----------+--------------+
>>>>>>>| id | removed             | ip4_address   | netmask       | gateway
>>>>>>>| ip_type | reserver_name            | network_id | instance_id |
>>>>>>>name
>>>>>>>| state     | type         |
>>>>>>>+----+---------------------+---------------+---------------+--------
>>>>>>>--
>>>>>>>--
>>>>>>>-+
>>>>>>>---------+--------------------------+------------+-------------+----
>>>>>>>--
>>>>>>>--
>>>>>>>-+
>>>>>>>-----------+--------------+
>>>>>>>|  9 | 2014-03-17 11:27:58 | 10.124.99.1   | 255.255.255.0 | NULL
>>>>>>>| NULL    | ExternalGuestNetworkGuru |        204 |           4 |
>>>>>>>r-4-VM
>>>>>>>| Expunging | DomainRouter |
>>>>>>>| 10 | 2014-03-17 11:27:58 | NULL          | NULL          | NULL
>>>>>>>| NULL    | ControlNetworkGuru       |        202 |           4 |
>>>>>>>r-4-VM
>>>>>>>| Expunging | DomainRouter |
>>>>>>>| 11 | 2014-03-17 11:27:58 | 10.193.17.139 | 255.255.255.0 |
>>>>>>>10.193.17.1
>>>>>>>| NULL    | PublicNetworkGuru        |        200 |           4 |
>>>>>>>r-4-VM
>>>>>>>| Expunging | DomainRouter |
>>>>>>>| 14 | 2014-03-17 11:27:52 | 10.124.99.1   | 255.255.255.0 | NULL
>>>>>>>| NULL    | ExternalGuestNetworkGuru |        205 |           7 |
>>>>>>>r-7-VM
>>>>>>>| Expunging | DomainRouter |
>>>>>>>| 15 | 2014-03-17 11:27:52 | NULL          | NULL          | NULL
>>>>>>>| NULL    | ControlNetworkGuru       |        202 |           7 |
>>>>>>>r-7-VM
>>>>>>>| Expunging | DomainRouter |
>>>>>>>| 16 | 2014-03-17 11:27:52 | 10.193.17.190 | 255.255.255.0 |
>>>>>>>10.193.17.1
>>>>>>>| NULL    | PublicNetworkGuru        |        200 |           7 |
>>>>>>>r-7-VM
>>>>>>>| Expunging | DomainRouter |
>>>>>>>| 26 | 2014-03-18 08:11:16 | 10.124.99.1   | 255.255.255.0 | NULL
>>>>>>>| NULL    | ExternalGuestNetworkGuru |        205 |          18 |
>>>>>>>r-18-VM
>>>>>>>| Expunging | DomainRouter |
>>>>>>>| 27 | 2014-03-18 08:11:16 | NULL          | NULL          | NULL
>>>>>>>| NULL    | ControlNetworkGuru       |        202 |          18 |
>>>>>>>r-18-VM
>>>>>>>| Expunging | DomainRouter |
>>>>>>>| 28 | 2014-03-18 08:11:16 | 10.193.17.190 | 255.255.255.0 |
>>>>>>>10.193.17.1
>>>>>>>| NULL    | PublicNetworkGuru        |        200 |          18 |
>>>>>>>r-18-VM
>>>>>>>| Expunging | DomainRouter |
>>>>>>>| 29 | NULL                | 10.124.99.1   | 255.255.255.0 | NULL
>>>>>>>| NULL    | ExternalGuestNetworkGuru |        205 |          19 |
>>>>>>>r-19-VM
>>>>>>>| Stopped   | DomainRouter |
>>>>>>>| 30 | NULL                | NULL          | NULL          | NULL
>>>>>>>| NULL    | ControlNetworkGuru       |        202 |          19 |
>>>>>>>r-19-VM
>>>>>>>| Stopped   | DomainRouter |
>>>>>>>| 31 | NULL                | 10.193.17.190 | 255.255.255.0 |
>>>>>>>10.193.17.1
>>>>>>>| NULL    | PublicNetworkGuru        |        200 |          19 |
>>>>>>>r-19-VM
>>>>>>>| Stopped   | DomainRouter |
>>>>>>>+----+---------------------+---------------+---------------+--------
>>>>>>>--
>>>>>>>--
>>>>>>>-+
>>>>>>>---------+--------------------------+------------+-------------+----
>>>>>>>--
>>>>>>>--
>>>>>>>-+
>>>>>>>-----------+--------------+
>>>>>>>
>>>>>>>mysql> select * from router_network_ref;
>>>>>>>+----+-----------+------------+------------+
>>>>>>>| id | router_id | network_id | guest_type |
>>>>>>>+----+-----------+------------+------------+
>>>>>>>|  1 |         4 |        204 | Isolated   |
>>>>>>>|  2 |         7 |        205 | Isolated   |
>>>>>>>|  3 |        18 |        205 | Isolated   |
>>>>>>>|  4 |        19 |        205 | Isolated   |
>>>>>>>+----+-----------+------------+------------+
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>Alena Prokharchyk <alena.prokharc...@citrix.com> wrote:
>>>>>>>> 
>>>>>>>> The error happens not because Ip is null, but because the nic in a
>>>>>>>>certain
>>>>>>>> network can¹t be found. Looks like there is some bug in VPC nic
>>>>>>>> plug/unplug for Guest networks process.
>>>>>>>>
>>>>>>>> Kambiz, please do the following to fix it:
>>>>>>>>
>>>>>>>> 1) Stop the MS
>>>>>>>> 2) Take the DB dump of cloud db in case  you have to revert back.
>>>>>>>> 3) Run the query:
>>>>>>>>
>>>>>>>> select * from router_network_ref where router_id=<id of your VR)
>>>>>>>>and
>>>>>>>> network_id not in (select network_id from nics where
>>>>>>>>instance_id=<ID
>>>>>>>>of
>>>>>>>> your VR> and removed is null);
>>>>>>>>
>>>>>>>> It will give you the list of networks refs that somehow weren¹t
>>>>>>>>cleaned
>>>>>>>> during the nic detach. Remove the entry returned from
>>>>>>>>router_network_ref
>>>>>>>> table.
>>>>>>>>
>>>>>>>> Let me know how it works.
>>>>>>>>
>>>>>>>> -Alena.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 3/21/14, 3:36 PM, "Kambiz Darabi" <dar...@m-creations.com>
>>>>>>>>wrote:
>>>>>>>>
>>>>>>>>>Hello,
>>>>>>>>>
>>>>>>>>>as this is my first post to the list, I would like to thank all
>>>>>>>>>contributors for Cloudstack which I use since last fall without
>>>>>>>>>any
>>>>>>>>>problems. I run 4.1.1 with KVM and advanced networking.
>>>>>>>>>
>>>>>>>>>After a restart of the management server (stopping and starting
>>>>>>>>>the
>>>>>>>>>java
>>>>>>>>>process), the virtual domain router doesn't start and
>>>>>>>>>management-server.log shows a NullPointerException in
>>>>>>>>>NetworkModelImpl.getIpInNetwork (cf. stack trace below).
>>>>>>>>>
>>>>>>>>>By putting the server in debug mode and remote debugging, I found
>>>>>>>>>out
>>>>>>>>>that the reason is a row in the table nics which has NULL in ip
>>>>>>>>>(cf.
>>>>>>>>>row
>>>>>>>>>with id 30 in the result of the select statement below).
>>>>>>>>>
>>>>>>>>>What can I do to quickly solve this problem? Any pointers or
>>>>>>>>>suggestions
>>>>>>>>>are appreciated as the system is currently unusable.
>>>>>>>>>
>>>>>>>>>Thank you for your help
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>Kambiz
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>management-server.log:
>>>>>>>>>
>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare
>>>>>>>>>for
>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>2014-03-18 10:03:27,151 WARN
>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[205|Guest|8] is not
>>>>>>>>>associated
>>>>>>>>>with
>>>>>>>>>any VPC
>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>Nic[29-19-30e229ba-21bd-4ab5-8570-9f495bce5019-10.124.99.1]
>>>>>>>>>2014-03-18 10:03:27,151 DEBUG [network.element.NiciraNvpElement]
>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle
>>>>>>>>>service
>>>>>>>>>Connectivity on network net1
>>>>>>>>>2014-03-18 10:03:27,153 DEBUG [cloud.network.NetworkModelImpl]
>>>>>>>>>(Job-Executor-1:job-176) Service SecurityGroup is not supported in
>>>>>>>>>the
>>>>>>>>>network id=205
>>>>>>>>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Lock is acquired for network id 202 as a
>>>>>>>>>part
>>>>>>>>>of
>>>>>>>>>network implement
>>>>>>>>>2014-03-18 10:03:27,156 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Network id=202 is already implemented
>>>>>>>>>2014-03-18 10:03:27,157 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Lock is released for network id 202 as a
>>>>>>>>>part
>>>>>>>>>of
>>>>>>>>>network implement
>>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Asking VirtualRouter to prepare for
>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Asking Ovs to prepare for
>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Asking SecurityGroupProvider to prepare
>>>>>>>>>for
>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>2014-03-18 10:03:27,187 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Asking VpcVirtualRouter to prepare for
>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>2014-03-18 10:03:27,187 WARN
>>>>>>>>>[network.element.VpcVirtualRouterElement]
>>>>>>>>>(Job-Executor-1:job-176) Network Ntwk[202|Control|3] is not
>>>>>>>>>associated
>>>>>>>>>with any VPC
>>>>>>>>>2014-03-18 10:03:27,188 DEBUG [cloud.network.NetworkManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Asking NiciraNvp to prepare for
>>>>>>>>>Nic[30-19-30e229ba-21bd-4ab5-8570-9f495bce5019-169.254.3.99]
>>>>>>>>>2014-03-18 10:03:27,188 DEBUG [network.element.NiciraNvpElement]
>>>>>>>>>(Job-Executor-1:job-176) Checking if NiciraNvpElement can handle
>>>>>>>>>service
>>>>>>>>>Connectivity on network null
>>>>>>>>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Checking if we need to prepare 1 volumes
>>>>>>>>>for
>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>2014-03-18 10:03:27,190 DEBUG [cloud.storage.StorageManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) No need to recreate the volume:
>>>>>>>>>Vol[24|vm=19|ROOT], since it already has a pool assigned: 200,
>>>>>>>>>adding
>>>>>>>>>disk to VM
>>>>>>>>>2014-03-18 10:03:27,224 DEBUG
>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Boot Args for VM[DomainRouter|r-19-VM]:
>>>>>>>>>template=domP name=r-19-VM eth2ip=10.193.17.190
>>>>>>>>>eth2mask=255.255.255.0
>>>>>>>>>gateway=10.193.17.1 eth0ip=10.124.99.1 eth0mask=255.255.255.0
>>>>>>>>>domain=cs6cloud.internal dhcprange=10.124.99.1 eth0ip=169.254.3.99
>>>>>>>>>eth0mask=255.255.0.0 type=router disable_rp_filter=true
>>>>>>>>>dns1=10.193.17.1
>>>>>>>>>2014-03-18 10:03:27,343 DEBUG
>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Found 8 ip(s) to apply as a part of domR
>>>>>>>>>VM[DomainRouter|r-19-VM] start.
>>>>>>>>>2014-03-18 10:03:27,415 DEBUG
>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Resending ipAssoc, port forwarding, load
>>>>>>>>>balancing rules as a part of Virtual router start
>>>>>>>>>2014-03-18 10:03:27,499 DEBUG
>>>>>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Found 12 firewall Egress rule(s) to apply
>>>>>>>>>as
>>>>>>>>>a
>>>>>>>>>part of domR VM[DomainRouter|r-19-VM] start.
>>>>>>>>>2014-03-18 10:03:27,593 ERROR [cloud.vm.VirtualMachineManagerImpl]
>>>>>>>>>(Job-Executor-1:job-176) Failed to start instance
>>>>>>>>>VM[DomainRouter|r-19-VM]
>>>>>>>>>java.lang.NullPointerException
>>>>>>>>>       at 
>>>>>>>>>com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl
>>>>>>>>>.j
>>>>>>>>>av
>>>>>>>>>a:
>>>>>>>>>76
>>>>>>>>>3)
>>>>>>>>>       at 
>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finali
>>>>>>>>>ze
>>>>>>>>>Ne
>>>>>>>>>tw
>>>>>>>>>or
>>>>>>>>>kRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346)
>>>>>>>>>       at 
>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.fin
>>>>>>>>>al
>>>>>>>>>iz
>>>>>>>>>eN
>>>>>>>>>et
>>>>>>>>>workRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928
>>>>>>>>>)
>>>>>>>>>       at 
>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finali
>>>>>>>>>ze
>>>>>>>>>Co
>>>>>>>>>mm
>>>>>>>>>an
>>>>>>>>>dsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
>>>>>>>>>       at 
>>>>>>>>>com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.fin
>>>>>>>>>al
>>>>>>>>>iz
>>>>>>>>>eC
>>>>>>>>>om
>>>>>>>>>mandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767)
>>>>>>>>>       at 
>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finali
>>>>>>>>>ze
>>>>>>>>>De
>>>>>>>>>pl
>>>>>>>>>oy
>>>>>>>>>ment(VirtualNetworkApplianceManagerImpl.java:2205)
>>>>>>>>>       at 
>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachine
>>>>>>>>>Ma
>>>>>>>>>na
>>>>>>>>>ge
>>>>>>>>>rI
>>>>>>>>>mpl.java:763)
>>>>>>>>>       at 
>>>>>>>>>com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManager
>>>>>>>>>Im
>>>>>>>>>pl
>>>>>>>>>.j
>>>>>>>>>av
>>>>>>>>>a:471)
>>>>>>>>>       at 
>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(
>>>>>>>>>Vi
>>>>>>>>>rt
>>>>>>>>>ua
>>>>>>>>>lN
>>>>>>>>>etworkApplianceManagerImpl.java:2616)
>>>>>>>>>       at 
>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startV
>>>>>>>>>ir
>>>>>>>>>tu
>>>>>>>>>al
>>>>>>>>>Ro
>>>>>>>>>uter(VirtualNetworkApplianceManagerImpl.java:1824)
>>>>>>>>>       at 
>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startR
>>>>>>>>>ou
>>>>>>>>>te
>>>>>>>>>rs
>>>>>>>>>(V
>>>>>>>>>irtualNetworkApplianceManagerImpl.java:1924)
>>>>>>>>>       at 
>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deploy
>>>>>>>>>Vi
>>>>>>>>>rt
>>>>>>>>>ua
>>>>>>>>>lR
>>>>>>>>>outerInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902)
>>>>>>>>>       at 
>>>>>>>>>com.cloud.network.element.VirtualRouterElement.implement(VirtualRo
>>>>>>>>>ut
>>>>>>>>>er
>>>>>>>>>El
>>>>>>>>>em
>>>>>>>>>ent.java:175)
>>>>>>>>>       at 
>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndRe
>>>>>>>>>so
>>>>>>>>>ur
>>>>>>>>>ce
>>>>>>>>>s(
>>>>>>>>>NetworkManagerImpl.java:1518)
>>>>>>>>>       at 
>>>>>>>>>com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManag
>>>>>>>>>er
>>>>>>>>>Im
>>>>>>>>>pl
>>>>>>>>>.j
>>>>>>>>>ava:1434)
>>>>>>>>>       at 
>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Inte
>>>>>>>>>rc
>>>>>>>>>ep
>>>>>>>>>to
>>>>>>>>>rD
>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>>>       at 
>>>>>>>>>com.cloud.network.NetworkManagerImpl.startNetwork(NetworkManagerIm
>>>>>>>>>pl
>>>>>>>>>.j
>>>>>>>>>av
>>>>>>>>>a:
>>>>>>>>>2435)
>>>>>>>>>       at 
>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startR
>>>>>>>>>ou
>>>>>>>>>te
>>>>>>>>>r(
>>>>>>>>>Vi
>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2855)
>>>>>>>>>       at 
>>>>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startR
>>>>>>>>>ou
>>>>>>>>>te
>>>>>>>>>r(
>>>>>>>>>Vi
>>>>>>>>>rtualNetworkApplianceManagerImpl.java:2824)
>>>>>>>>>       at 
>>>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Inte
>>>>>>>>>rc
>>>>>>>>>ep
>>>>>>>>>to
>>>>>>>>>rD
>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>>>>>>>>       at 
>>>>>>>>>org.apache.cloudstack.api.command.admin.router.StartRouterCmd.exec
>>>>>>>>>ut
>>>>>>>>>e(
>>>>>>>>>St
>>>>>>>>>ar
>>>>>>>>>tRouterCmd.java:103)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>table nics:
>>>>>>>>>
>>>>>>>>>mysql> select * from nics where reserver_name =
>>>>>>>>>'ControlNetworkGuru';
>>>>>>>>>+----+--------------------------------------+-------------+-------
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>----+---------------+-------------+-------------+---------+-------
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>+------------+--------+--------------+----------+-----------------
>>>>>>>>>--
>>>>>>>>>-+
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>----------------------------------+-----------+-------------------
>>>>>>>>>--
>>>>>>>>>+-
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>----------+-------------+-------------+--------------------+------
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>| id | uuid                                 | instance_id |
>>>>>>>>>mac_address
>>>>>>>>>    | ip4_address   | netmask     | gateway     | ip_type |
>>>>>>>>>broadcast_uri
>>>>>>>>>| network_id | mode   | state        | strategy | reserver_name
>>>>>>>>> |
>>>>>>>>>reservation_id                       | device_id | update_time
>>>>>>>>> |
>>>>>>>>>isolation_uri | ip6_address | default_nic | vm_type            |
>>>>>>>>>created
>>>>>>>>>           | removed             | ip6_gateway | ip6_cidr |
>>>>>>>>>+----+--------------------------------------+-------------+-------
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>----+---------------+-------------+-------------+---------+-------
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>+------------+--------+--------------+----------+-----------------
>>>>>>>>>--
>>>>>>>>>-+
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>----------------------------------+-----------+-------------------
>>>>>>>>>--
>>>>>>>>>+-
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>----------+-------------+-------------+--------------------+------
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>|  2 | 289aacb8-cfd7-4879-a632-6cfbda36cbf4 |           1 |
>>>>>>>>>0e:00:a9:fe:00:55 | 169.254.0.85  | 255.255.0.0 | 169.254.0.1 |
>>>>>>>>>Ip4
>>>>>>>>>|
>>>>>>>>>NULL          |        202 | Static | Reserved     | Start    |
>>>>>>>>>ControlNetworkGuru | 993864b4-9dde-47d6-8fd6-cf94050442c6 |
>>>>>>>>>0
>>>>>>>>>|
>>>>>>>>>2014-03-17 22:21:38 | NULL          | NULL        |           0 |
>>>>>>>>>SecondaryStorageVm | 2013-09-06 12:44:42 | NULL                |
>>>>>>>>>NULL
>>>>>>>>>   | NULL     |
>>>>>>>>>|  6 | 5fdf4b1a-b90c-4c79-9d42-9eaf87eaa042 |           2 |
>>>>>>>>>0e:00:a9:fe:02:d3 | 169.254.2.211 | 255.255.0.0 | 169.254.0.1 |
>>>>>>>>>Ip4
>>>>>>>>>|
>>>>>>>>>NULL          |        202 | Static | Reserved     | Start    |
>>>>>>>>>ControlNetworkGuru | 852e0a65-c72a-448f-ac71-2bb3549a5a41 |
>>>>>>>>>0
>>>>>>>>>|
>>>>>>>>>2014-03-17 22:21:38 | NULL          | NULL        |           0 |
>>>>>>>>>ConsoleProxy       | 2013-09-06 12:44:42 | NULL                |
>>>>>>>>>NULL
>>>>>>>>>   | NULL     |
>>>>>>>>>| 10 | 4c4e6368-95d7-419a-a9b3-a5bb394197f0 |           4 | NULL
>>>>>>>>>    | NULL          | NULL        | NULL        | NULL    | NULL
>>>>>>>>>|        202 | Static | Deallocating | Start    |
>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>c28e8ddc-c106-462e-96c8-5d5216dad9b7 |         1 | 2014-03-17
>>>>>>>>>12:27:58 |
>>>>>>>>>NULL          | NULL        |           0 | DomainRouter       |
>>>>>>>>>2013-09-10 08:08:39 | 2014-03-17 11:27:58 | NULL        | NULL
>>>>>>>>> |
>>>>>>>>>| 15 | 1f2e99c0-9cd9-47aa-ab10-f190efd7a2dc |           7 | NULL
>>>>>>>>>    | NULL          | NULL        | NULL        | NULL    | NULL
>>>>>>>>>|        202 | Static | Deallocating | Start    |
>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>ca1aa99e-e630-4533-9642-523d8a8b1fea |         1 | 2014-03-17
>>>>>>>>>12:27:52 |
>>>>>>>>>NULL          | NULL        |           0 | DomainRouter       |
>>>>>>>>>2013-09-12 10:58:03 | 2014-03-17 11:27:52 | NULL        | NULL
>>>>>>>>> |
>>>>>>>>>| 27 | 1c98c4f2-f604-4a38-a813-f68833b1d250 |          18 | NULL
>>>>>>>>>    | NULL          | NULL        | NULL        | NULL    | NULL
>>>>>>>>>|        202 | Static | Deallocating | Start    |
>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>ad8e0e50-72aa-4c68-8634-8dc89f12fe01 |         1 | 2014-03-18
>>>>>>>>>09:11:16 |
>>>>>>>>>NULL          | NULL        |           0 | DomainRouter       |
>>>>>>>>>2014-03-17 11:28:50 | 2014-03-18 08:11:16 | NULL        | NULL
>>>>>>>>> |
>>>>>>>>>| 30 | cabd4cd9-c39f-423f-ad6a-ee3affe0bd9d |          19 | NULL
>>>>>>>>>    | NULL          | NULL        | NULL        | NULL    | NULL
>>>>>>>>>|        202 | Static | Allocated    | Start    |
>>>>>>>>>ControlNetworkGuru |
>>>>>>>>>e81ba56d-a101-4c60-b44f-a0890d56aad9 |         1 | 2014-03-18
>>>>>>>>>09:11:44 |
>>>>>>>>>NULL          | NULL        |           0 | DomainRouter       |
>>>>>>>>>2014-03-18 08:11:32 | NULL                | NULL        | NULL
>>>>>>>>> |
>>>>>>>>>+----+--------------------------------------+-------------+-------
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>----+---------------+-------------+-------------+---------+-------
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>+------------+--------+--------------+----------+-----------------
>>>>>>>>>--
>>>>>>>>>-+
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>----------------------------------+-----------+-------------------
>>>>>>>>>--
>>>>>>>>>+-
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>----------+-------------+-------------+--------------------+------
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>--
>>>>>>>>>-------+---------------------+-------------+----------+
>>>>>>>>>
>>>>>>>>>

Reply via email to