Hi,
It's can be some know bug?
Possible it's already solved in new releases of CS but i need the
work-around or fix before upgrade or reference to bug id.
Environment:
CS 4.1.1
libvirt-1.0.1
qemu-kvm-1.2
NFS Storage ( as primary for VR's )
Advanced VLAN isolation
After hypervisor host crashing, one of VR's has failed to start in failover
case,
I have stopped it through UI with force, then was removed the VR for
re-create it again by start/create VM API call.
Try to start the Instance associated with this network, but failed because
the VR can't be started when newly created.
cloudstack-agent:
2014-04-11 07:05:34,546 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Failed to get dom xml:
org.libvirt.LibvirtException: Domain not found: no domain with matching
uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
2014-04-11 07:05:34,547 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Failed to get dom xml:
org.libvirt.LibvirtException: Domain not found: no domain with matching
uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
2014-04-11 07:05:34,548 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Failed to get dom xml:
org.libvirt.LibvirtException: Domain not found: no domain with matching
uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
2014-04-11 07:05:34,548 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Executing:
/usr/share/cloudstack-common/scripts/vm/network/security_group.py
destroy_network_rules_for_vm --vmname r-377-VM
2014-04-11 07:05:34,663 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Execution is successful.
2014-04-11 07:05:34,664 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Try to stop the vm at first
2014-04-11 07:05:34,665 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Failed to stop VM :r-377-VM :
org.libvirt.LibvirtException: Domain not found: no domain with matching
uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.Connect.processError(Unknown Source)
at org.libvirt.Connect.domainLookupByUUIDString(Unknown Source)
at org.libvirt.Connect.domainLookupByUUID(Unknown Source)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.stopVM(LibvirtComputingResource.java:4021)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.stopVM(LibvirtComputingResource.java:3970)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2894)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1032)
at com.cloud.agent.Agent.processRequest(Agent.java:525)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)
at com.cloud.utils.nio.Task.run(Task.java:83)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2014-04-11 07:05:34,666 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Failed to get vm status:Domain not found: no
domain with matching uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
2014-04-11 07:05:34,667 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Failed to get vm status:Domain not found: no
domain with matching uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
2014-04-11 07:05:34,668 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Failed to get vm status:Domain not found: no
domain with matching uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
Management CS:
2014-04-11 07:05:40,503 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl]
(Job-Executor-114:job-3001) Found 5 ip(s) to apply as a part of domR
VM[DomainRouter|r-377-VM] start.
2014-04-11 07:05:40,528 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl]
(Job-Executor-114:job-3001) Resending ipAssoc, port forwarding, load
balancing rules as a part of Virtual router start
2014-04-11 07:05:40,542 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl]
(Job-Executor-114:job-3001) Found 1 firewall Egress rule(s) to apply as a
part of domR VM[DomainRouter|r-377-VM] start.
2014-04-11 07:05:40,581 ERROR [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-114:job-3001) Failed to start instance
VM[DomainRouter|r-377-VM]
java.lang.NullPointerException
at
com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:763)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346)
at
com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
at
com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeployment(VirtualNetworkApplianceManagerImpl.java:2205)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:763)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2616)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1824)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(VirtualNetworkApplianceManagerImpl.java:1924)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902)
at
com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:175)
at
com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:1518)
at
com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl.java:1434)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:1596)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:746)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471)
at
org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:212)
at
org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3871)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2579)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.user.vm.StartVMCmd.execute(StartVMCmd.java:120)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162)
at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2014-04-11 07:05:40,584 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-114:job-3001) Cleaning up resources for the vm
VM[DomainRouter|r-377-VM] in Starting state
2014-04-11 07:05:40,585 DEBUG [agent.transport.Request]
(Job-Executor-114:job-3001) Seq 5-187434248: Sending { Cmd , MgmtId:
66290989385104, via: 5, Ver: v1, Flags: 100111,
[{"StopCommand":{"isProxy":false,"vmName":"r-377-VM","wait":0}}] }
2014-04-11 07:05:40,707 DEBUG [agent.transport.Request]
(AgentManager-Handler-16:null) Seq 5-187434248: Processing: { Ans: ,
MgmtId: 66290989385104, via: 5, Ver: v1, Flags: 110,
[{"StopAnswer":{"vncPort":0,"result":true,"wait":0}}] }
2014-04-11 07:05:40,707 DEBUG [agent.manager.AgentAttache]
(AgentManager-Handler-16:null) Seq 5-187434248: No more commands found
2014-04-11 07:05:40,707 DEBUG [agent.transport.Request]
(Job-Executor-114:job-3001) Seq 5-187434248: Received: { Ans: , MgmtId:
66290989385104, via: 5, Ver: v1, Flags: 110, { StopAnswer } }
2014-04-11 07:05:40,710 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl]
(Job-Executor-114:job-3001) Successfully updated user statistics as a part
of domR VM[DomainRouter|r-377-VM] reboot/stop
2014-04-11 07:05:40,718 DEBUG [cloud.network.NetworkModelImpl]
(Job-Executor-114:job-3001) Service SecurityGroup is not supported in the
network id=204
2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking NiciraNvp to release
Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
2014-04-11 07:05:40,722 DEBUG [network.element.NiciraNvpElement]
(Job-Executor-114:job-3001) Checking if NiciraNvpElement can handle service
Connectivity on network inewdate
2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking JuniperSRX to release
Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking Netscaler to release
Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking F5BigIP to release
Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking CiscoNexus1000vVSM to release
Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking BigSwitchVnsElement to release
Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
2014-04-11 07:05:40,722 DEBUG [network.element.BigSwitchVnsElement]
(Job-Executor-114:job-3001) Checking if BigSwitchVnsElement can handle
service Connectivity on network inewdate
2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking VirtualRouter to release
Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking Ovs to release
Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking SecurityGroupProvider to release
Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking VpcVirtualRouter to release
Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
2014-04-11 07:05:40,724 DEBUG [network.guru.ControlNetworkGuru]
(Job-Executor-114:job-3001) Released nic: NicProfile[491-377-null-null-null
2014-04-11 07:05:40,725 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking NiciraNvp to release
Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
2014-04-11 07:05:40,725 DEBUG [network.element.NiciraNvpElement]
(Job-Executor-114:job-3001) Checking if NiciraNvpElement can handle service
Connectivity on network null
2014-04-11 07:05:40,725 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking JuniperSRX to release
Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
2014-04-11 07:05:40,725 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking Netscaler to release
Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
2014-04-11 07:05:40,725 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking F5BigIP to release
Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
2014-04-11 07:05:40,725 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking CiscoNexus1000vVSM to release
Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
2014-04-11 07:05:40,726 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking BigSwitchVnsElement to release
Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
2014-04-11 07:05:40,726 DEBUG [network.element.BigSwitchVnsElement]
(Job-Executor-114:job-3001) Checking if BigSwitchVnsElement can handle
service Connectivity on network null
2014-04-11 07:05:40,726 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking VirtualRouter to release
Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
2014-04-11 07:05:40,726 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking Ovs to release
Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
2014-04-11 07:05:40,726 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking SecurityGroupProvider to release
Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
2014-04-11 07:05:40,726 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking VpcVirtualRouter to release
Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
2014-04-11 07:05:40,728 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-114:job-3001) Successfully released network resources for the
vm VM[DomainRouter|r-377-VM]
2014-04-11 07:05:40,728 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-114:job-3001) Successfully cleanued up resources for the vm
VM[DomainRouter|r-377-VM] in Starting state
2014-04-11 07:05:40,731 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-114:job-3001) VM state transitted from :Starting to Stopped
with event: OperationFailedvm's original host id: null new host id: null
host id before state transition: 5
Someone know such problem? Or help me to debug it please.
--
ttyv0 "/usr/libexec/gmail Pc" webcons on secure
