[
https://issues.apache.org/jira/browse/CLOUDSTACK-7396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14106563#comment-14106563
]
Jayapal Reddy commented on CLOUDSTACK-7396:
-------------------------------------------
In the kvm host libvirt is not running:
[root@Rack2Host12 ~]# service libvirtd status
libvirtd dead but pid file exists
[root@Rack2Host12 ~]# ps aux | grep libvirtd
root 14238 0.0 0.0 103212 820 pts/13 S+ 22:56 0:00 grep libvirtd
[root@Rack2Host12 ~]#
In libvirtd logs I saw error logs related to primary storage.
I started libvirtd and it is running successfully now.
Now the VR stop should success.
*agent logs:
2014-08-21 22:54:04,848 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null)
Processing command: com.cloud.agent.api.routing.SetNetworkACLCommand
2014-08-21 22:54:04,848 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) Executing:
/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh vpc_acl.sh
169.254.3.25 -d eth3 -M 02:00:0e:b0:00:02 -i 10.1.2.1 -m 24 -a
Ingress:reverted:0:0:0:,
2014-08-21 22:54:05,132 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) Execution is successful.
2014-08-21 22:54:05,132 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null)
Seq 2-3001649151642437575: { Ans: , MgmtId: 29066118877352, via: 2, Ver: v1,
Flags: 0,
[{"com.cloud.agent.api.Answer":{"result":true,"details":"","wait":0}}] }
2014-08-21 22:54:05,158 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:null)
Request:Seq 2-3001649151642437576: { Cmd , MgmtId: 29066118877352, via: 2,
Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.SetupGuestNetworkCommand":{"dhcpRange":"10.1.2.1","networkDomain":"test.domain.org","isRedundant":false,"add":false,"nic":{"deviceId":3,"networkRateMbps":200,"defaultNic":false,"pxeDisable":true,"nicUuid":"bc8b9a9f-7c76-4a73-80d5-444972c6122a","uuid":"ab491a32-d28b-4010-9da9-ab5db94a8744","ip":"10.1.2.1","netmask":"255.255.255.0","gateway":"10.1.2.1","mac":"02:00:0e:b0:00:02","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://2303","isolationUri":"vlan://2303","isSecurityGroupEnabled":false},"accessDetails":{"router.guest.ip":"10.1.2.1","guest.vlan.tag":"2303","guest.network.gateway":"10.1.2.1","guest.bridge":"10.1.2.255","router.ip":"169.254.3.25","router.name":"r-918-VM"},"wait":0}}]
}
2014-08-21 22:54:05,158 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:null)
Processing command: com.cloud.agent.api.SetupGuestNetworkCommand
2014-08-21 22:54:05,159 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) Connection with libvirtd is broken, due to Cannot
write data: Broken pipe
2014-08-21 22:54:05,159 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) can't find connection: KVM, for vm: r-918-VM,
continue
2014-08-21 22:54:05,159 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) Connection with libvirtd is broken, due to Cannot
write data: Broken pipe
2014-08-21 22:54:05,159 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) can't find connection: LXC, for vm: r-918-VM,
continue
2014-08-21 22:54:05,160 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) can't find which hypervisor the vm used , then
use the default hypervisor
2014-08-21 22:54:05,160 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) Connection with libvirtd is broken, due to Cannot
write data: Broken pipe
2014-08-21 22:54:05,160 WARN [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-4:null) Creating guest network failed due to
org.libvirt.LibvirtException: Failed to connect socket to
'/var/run/libvirt/libvirt-sock': Connection refused
org.libvirt.LibvirtException: Failed to connect socket to
'/var/run/libvirt/libvirt-sock': Connection refused
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.Connect.processError(Unknown Source)
at org.libvirt.Connect.processError(Unknown Source)
at org.libvirt.Connect.<init>(Unknown Source)
at
com.cloud.hypervisor.kvm.resource.LibvirtConnection.getConnection(LibvirtConnection.java:51)
at
com.cloud.hypervisor.kvm.resource.LibvirtConnection.getConnection(LibvirtConnection.java:37)
at
com.cloud.hypervisor.kvm.resource.LibvirtConnection.getConnectionByVmName(LibvirtConnection.java:75)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.prepareNetworkElementCommand(LibvirtComputingResource.java:2115)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.prepareCommand(LibvirtComputingResource.java:381)
at
com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeRequest(VirtualRoutingResource.java:96)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1338)
at com.cloud.agent.Agent.processRequest(Agent.java:503)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:810)
at com.cloud.utils.nio.Task.run(Task.java:84)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
2014-08-21 22:54:05,160 ERROR [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-4:null) Failed to prepare VR command due to Creating
guest network failed due to org.libvirt.LibvirtException: Failed to connect
socket to '/var/run/libvirt/libvirt-sock': Connection refused
2014-08-21 22:54:05,160 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:null)
Seq 2-3001649151642437576: { Ans: , MgmtId: 29066118877352, via: 2, Ver: v1,
Flags: 110, [{"com.cloud.agent.api.Answer":{"result":false,"details":"Creating
guest network failed due to org.libvirt.LibvirtException: Failed to connect
socket to '/var/run/libvirt/libvirt-sock': Connection refused","wait":0}}] }
~
> [Automation] Failed to stop VPC router in KVM
> ---------------------------------------------
>
> Key: CLOUDSTACK-7396
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7396
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Automation
> Affects Versions: 4.5.0
> Reporter: Rayees Namathponnan
> Assignee: Rayees Namathponnan
> Priority: Blocker
> Fix For: 4.5.0
>
> Attachments: agent.rar, kvm.sql, management-server.rar
>
>
> This issue observed with latest automation run, deploy a VPC router and stop
> Virtual router stop fails with below exception
> {noformat}
> lhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAC-XQAGVZpcn
> R1YWxNYWNoaW5lTWFuYWdlckltcGwA, cmdVersion: 0, status: IN_PROGRESS,
> processStatus: 0, resultCode: 0, result: null, initMsid:
> 29066118877352, completeMsid: null, lastUpdated: null, lastPolled: null,
> created: Thu Aug 21 22:24:10 PDT 2014}, job origin
> :8131
> com.cloud.utils.exception.CloudRuntimeException: Unable to stop
> VM[DomainRouter|r-761-VM]
> at
> com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl.java:1523)
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStop(VirtualMachineManagerImpl.java:1377)
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStop(VirtualMachineManagerImpl.java:4594)
> at sun.reflect.GeneratedMethodAccessor162.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4738)
> at
> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> 2014-08-21 22:24:11,959 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-28:ctx-8824da69 job-8131/job-8162) Comple
> te async job-8162, jobStatus: FAILED, resultCode: 0, result:
> rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHI
> AE2phdmEubGFuZy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARMAAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3d
> hYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9sYW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7TAAUc3V
> wcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAREpvYiBmYWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBVbmFibGUgdG8gc3RvcCBWTVt
> Eb21haW5Sb3V0ZXJ8ci03NjEtVk1ddXIAHltMamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50OwJGKjw8_SI5AgAAeHAAAAANc3IAG2phdmEubGFuZy5TdGFja1R
> yYWNlRWxlbWVudGEJxZomNt2FAgAESQAKbGluZU51bWJlckwADmRlY2xhcmluZ0NsYXNzcQB-AARMAAhmaWxlTm
> 2014-08-21 22:42:19,885 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-149:ctx-5eeadd75 job-8131/job-8304) Unable to c
> omplete AsyncJobVO {id:8304, userId: 1, accountId: 1, instanceType: null,
> instanceId: null, cmd: com.cloud.vm.VmWorkStop, cm
> dInfo:
> rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKA
> AlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAADBnQAGVZpc
> nR1YWxNYWNoaW5lTWFuYWdlckltcGwA, cmdVersion: 0, status: IN_PROGRESS,
> processStatus: 0, resultCode: 0, result: null, initMsid
> : 29066118877352, completeMsid: null, lastUpdated: null, lastPolled: null,
> created: Thu Aug 21 22:42:18 PDT 2014}, job origi
> n:8131
> com.cloud.utils.exception.CloudRuntimeException: Unable to stop
> VM[DomainRouter|r-774-VM]
> at
> com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl.java:1523)
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStop(VirtualMachineManagerImpl.java:1377)
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStop(VirtualMachineManagerImpl.java:4594)
> at sun.reflect.GeneratedMethodAccessor162.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4738)
> at
> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> 2014-08-21 22:42:19,886 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-149:ctx-5eeadd75 job-8131/job-8304) Complete async
> job-8304, jobStatus: FAILED, resultCode: 0, result:
> rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFuZy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARMAAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9sYW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqY
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
