Re: KVM, Re-create VR failed

2014-04-14 Thread Marcus 
Your agent snippet just looks like the system trying to stop the vm.
If a vm fails to start, it will also run through the stop routine to
clean up all of the prework, so the 'failed to stop' debug is all
normal. You may need to go above and look at why it failed to start.

On Fri, Apr 11, 2014 at 3:53 PM, Serg Senko kernc...@gmail.com wrote:
 Hi,

 It's can be some know bug?
 Possible it's already solved in new releases of CS but i need the
 work-around or fix before upgrade or reference to bug id.

 Environment:
 CS 4.1.1
 libvirt-1.0.1
 qemu-kvm-1.2
 NFS Storage ( as primary for VR's )
 Advanced VLAN isolation

 After hypervisor host crashing, one of VR's has failed to start in failover
 case,
 I have stopped it through UI with force, then was removed the VR for
 re-create it again by start/create VM API call.


 Try to start the Instance associated with this network, but failed because
 the VR can't be started when newly created.

 cloudstack-agent:

 2014-04-11 07:05:34,546 DEBUG [kvm.resource.LibvirtComputingResource]
 (agentRequest-Handler-2:null) Failed to get dom xml:
 org.libvirt.LibvirtException: Domain not found: no domain with matching
 uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'

 2014-04-11 07:05:34,547 DEBUG [kvm.resource.LibvirtComputingResource]
 (agentRequest-Handler-2:null) Failed to get dom xml:
 org.libvirt.LibvirtException: Domain not found: no domain with matching
 uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'

 2014-04-11 07:05:34,548 DEBUG [kvm.resource.LibvirtComputingResource]
 (agentRequest-Handler-2:null) Failed to get dom xml:
 org.libvirt.LibvirtException: Domain not found: no domain with matching
 uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'

 2014-04-11 07:05:34,548 DEBUG [kvm.resource.LibvirtComputingResource]
 (agentRequest-Handler-2:null) Executing:
 /usr/share/cloudstack-common/scripts/vm/network/security_group.py
 destroy_network_rules_for_vm --vmname r-377-VM

 2014-04-11 07:05:34,663 DEBUG [kvm.resource.LibvirtComputingResource]
 (agentRequest-Handler-2:null) Execution is successful.

 2014-04-11 07:05:34,664 DEBUG [kvm.resource.LibvirtComputingResource]
 (agentRequest-Handler-2:null) Try to stop the vm at first

 2014-04-11 07:05:34,665 DEBUG [kvm.resource.LibvirtComputingResource]
 (agentRequest-Handler-2:null) Failed to stop VM :r-377-VM :

 org.libvirt.LibvirtException: Domain not found: no domain with matching
 uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'

 at org.libvirt.ErrorHandler.processError(Unknown Source)

 at org.libvirt.Connect.processError(Unknown Source)

 at org.libvirt.Connect.domainLookupByUUIDString(Unknown Source)

 at org.libvirt.Connect.domainLookupByUUID(Unknown Source)

 at
 com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.stopVM(LibvirtComputingResource.java:4021)

 at
 com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.stopVM(LibvirtComputingResource.java:3970)

 at
 com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2894)

 at
 com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1032)

 at com.cloud.agent.Agent.processRequest(Agent.java:525)

 at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)

 at com.cloud.utils.nio.Task.run(Task.java:83)

 at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

 at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

 at java.lang.Thread.run(Thread.java:679)

 2014-04-11 07:05:34,666 DEBUG [kvm.resource.LibvirtComputingResource]
 (agentRequest-Handler-2:null) Failed to get vm status:Domain not found: no
 domain with matching uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'

 2014-04-11 07:05:34,667 DEBUG [kvm.resource.LibvirtComputingResource]
 (agentRequest-Handler-2:null) Failed to get vm status:Domain not found: no
 domain with matching uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'

 2014-04-11 07:05:34,668 DEBUG [kvm.resource.LibvirtComputingResource]
 (agentRequest-Handler-2:null) Failed to get vm status:Domain not found: no
 domain with matching uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'




 Management CS:

 2014-04-11 07:05:40,503 DEBUG
 [network.router.VirtualNetworkApplianceManagerImpl]
 (Job-Executor-114:job-3001) Found 5 ip(s) to apply as a part of domR
 VM[DomainRouter|r-377-VM] start.

 2014-04-11 07:05:40,528 DEBUG
 [network.router.VirtualNetworkApplianceManagerImpl]
 (Job-Executor-114:job-3001) Resending ipAssoc, port forwarding, load
 balancing rules as a part of Virtual router start

 2014-04-11 07:05:40,542 DEBUG
 [network.router.VirtualNetworkApplianceManagerImpl]
 (Job-Executor-114:job-3001) Found 1 firewall Egress rule(s) to apply as a
 part of domR VM[DomainRouter|r-377-VM] start.

 2014-04-11 07:05:40,581 ERROR [cloud.vm.VirtualMachineManagerImpl]
 (Job-Executor-114:job-3001) Failed to start instance
 VM[DomainRouter|r-377-VM]

 java.lang.NullPointerException

 at

KVM, Re-create VR failed

2014-04-11 Thread Serg Senko 
Hi,

It's can be some know bug?
Possible it's already solved in new releases of CS but i need the
work-around or fix before upgrade or reference to bug id.

Environment:
CS 4.1.1
libvirt-1.0.1
qemu-kvm-1.2
NFS Storage ( as primary for VR's )
Advanced VLAN isolation

After hypervisor host crashing, one of VR's has failed to start in failover
case,
I have stopped it through UI with force, then was removed the VR for
re-create it again by start/create VM API call.


Try to start the Instance associated with this network, but failed because
the VR can't be started when newly created.

cloudstack-agent:

2014-04-11 07:05:34,546 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Failed to get dom xml:
org.libvirt.LibvirtException: Domain not found: no domain with matching
uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'

2014-04-11 07:05:34,547 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Failed to get dom xml:
org.libvirt.LibvirtException: Domain not found: no domain with matching
uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'

2014-04-11 07:05:34,548 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Failed to get dom xml:
org.libvirt.LibvirtException: Domain not found: no domain with matching
uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'

2014-04-11 07:05:34,548 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Executing:
/usr/share/cloudstack-common/scripts/vm/network/security_group.py
destroy_network_rules_for_vm --vmname r-377-VM

2014-04-11 07:05:34,663 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Execution is successful.

2014-04-11 07:05:34,664 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Try to stop the vm at first

2014-04-11 07:05:34,665 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Failed to stop VM :r-377-VM :

org.libvirt.LibvirtException: Domain not found: no domain with matching
uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'

at org.libvirt.ErrorHandler.processError(Unknown Source)

at org.libvirt.Connect.processError(Unknown Source)

at org.libvirt.Connect.domainLookupByUUIDString(Unknown Source)

at org.libvirt.Connect.domainLookupByUUID(Unknown Source)

at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.stopVM(LibvirtComputingResource.java:4021)

at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.stopVM(LibvirtComputingResource.java:3970)

at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2894)

at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1032)

at com.cloud.agent.Agent.processRequest(Agent.java:525)

at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)

at com.cloud.utils.nio.Task.run(Task.java:83)

at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

at java.lang.Thread.run(Thread.java:679)

2014-04-11 07:05:34,666 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Failed to get vm status:Domain not found: no
domain with matching uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'

2014-04-11 07:05:34,667 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Failed to get vm status:Domain not found: no
domain with matching uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'

2014-04-11 07:05:34,668 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Failed to get vm status:Domain not found: no
domain with matching uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'




Management CS:

2014-04-11 07:05:40,503 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl]
(Job-Executor-114:job-3001) Found 5 ip(s) to apply as a part of domR
VM[DomainRouter|r-377-VM] start.

2014-04-11 07:05:40,528 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl]
(Job-Executor-114:job-3001) Resending ipAssoc, port forwarding, load
balancing rules as a part of Virtual router start

2014-04-11 07:05:40,542 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl]
(Job-Executor-114:job-3001) Found 1 firewall Egress rule(s) to apply as a
part of domR VM[DomainRouter|r-377-VM] start.

2014-04-11 07:05:40,581 ERROR [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-114:job-3001) Failed to start instance
VM[DomainRouter|r-377-VM]

java.lang.NullPointerException

at
com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:763)

at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346)

at
com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928)

at