prashant kumar mishra created CLOUDSTACK-3977:
-------------------------------------------------
Summary: vmdeployement is failing ;: Resource [Host:5] is
unreachable: Host 5: Unable to send class
org.apache.cloudstack.storage.command.CopyCommand because agent is in
maintenance mode
Key: CLOUDSTACK-3977
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3977
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Storage Controller
Affects Versions: 4.2.0
Environment: hypervisor:xen6.1
Reporter: prashant kumar mishra
Priority: Critical
Fix For: 4.2.0
Steps to reproduce
------------------------------
1-preapre a CS setup with xen6.1
2-deploy a vm
3-add another host say h2 to same cluster
4-put h2 in maintenance mode
5-try to deploy one more vm
Expected
--------------
deploy vm should be successful
Actual
---------
deploy vm is failing
My observation
----------------------
->was able to deploy vm when h2 was not in maintenance mode
-->was able deploy vm when i removed h2 from cluster
Logs
--------
2013-07-31 13:35:08,675 DEBUG [storage.image.TemplateDataFactoryImpl]
(Job-Executor-77:job-291 = [ 6af942b2-9301-4aef-afff-3278581cc917 ]) template 5
is already in store:1, type:Image
2013-07-31 13:35:08,681 DEBUG [storage.image.TemplateDataFactoryImpl]
(Job-Executor-77:job-291 = [ 6af942b2-9301-4aef-afff-3278581cc917 ]) template 5
is already in store:2, type:Primary
2013-07-31 13:35:08,710 DEBUG [storage.motion.AncientDataMotionStrategy]
(Job-Executor-77:job-291 = [ 6af942b2-9301-4aef-afff-3278581cc917 ]) copyAsync
inspecting src type TEMPLATE copyAsync inspecting dest type VOLUME
2013-07-31 13:35:08,724 DEBUG [cloudstack.storage.RemoteHostEndPoint]
(Job-Executor-77:job-291 = [ 6af942b2-9301-4aef-afff-3278581cc917 ]) Failed to
send command, due to Agent:5, com.cloud.exception.AgentUnavailableException:
Resource [Host:5] is unreachable: Host 5: Unable to send class
org.apache.cloudstack.storage.command.CopyCommand because agent is in
maintenance mode
2013-07-31 13:35:08,725 DEBUG [storage.motion.AncientDataMotionStrategy]
(Job-Executor-77:job-291 = [ 6af942b2-9301-4aef-afff-3278581cc917 ]) Failed to
send to storage pool
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to
Agent:5, com.cloud.exception.AgentUnavailableException: Resource [Host:5] is
unreachable: Host 5: Unable to send class
org.apache.cloudstack.storage.command.CopyCommand because agent is in
maintenance mode
at
org.apache.cloudstack.storage.RemoteHostEndPoint.sendMessage(RemoteHostEndPoint.java:104)
at
org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.cloneVolume(AncientDataMotionStrategy.java:270)
at
org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyAsync(AncientDataMotionStrategy.java:373)
at
org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:55)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromBaseImageAsync(VolumeServiceImpl.java:501)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:538)
at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2497)
at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2554)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:913)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:618)
at
org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227)
at
org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3329)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2889)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2875)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-07-31 13:35:08,726 DEBUG [storage.motion.AncientDataMotionStrategy]
(Job-Executor-77:job-291 = [ 6af942b2-9301-4aef-afff-3278581cc917 ]) copy failed
com.cloud.utils.exception.CloudRuntimeException: Failed to send to storage pool
at
org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.cloneVolume(AncientDataMotionStrategy.java:274)
at
org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyAsync(AncientDataMotionStrategy.java:373)
at
org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:55)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromBaseImageAsync(VolumeServiceImpl.java:501)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:538)
at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2497)
at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2554)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:913)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:618)
at
org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227)
at
org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3329)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2889)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2875)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to send
command, due to Agent:5, com.cloud.exception.AgentUnavailableException:
Resource [Host:5] is unreachable: Host 5: Unable to send class
org.apache.cloudstack.storage.command.CopyCommand because agent is in
maintenance mode
at
org.apache.cloudstack.storage.RemoteHostEndPoint.sendMessage(RemoteHostEndPoint.java:104)
at
org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.cloneVolume(AncientDataMotionStrategy.java:270)
... 27 more
2013-07-31 13:35:08,740 WARN [storage.datastore.ObjectInDataStoreManagerImpl]
(Job-Executor-77:job-291 = [ 6af942b2-9301-4aef-afff-3278581cc917 ])
Unsupported data object (VOLUME,
org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@181c6ea7)
2013-07-31 13:35:08,741 DEBUG [cloud.storage.VolumeManagerImpl]
(Job-Executor-77:job-291 = [ 6af942b2-9301-4aef-afff-3278581cc917 ]) Unable to
create Vol[69|vm=72|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
Failed to send to storage pool
2013-07-31 13:35:08,741 INFO [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-77:job-291 = [ 6af942b2-9301-4aef-afff-3278581cc917 ]) Unable to
contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:2] is
unreachable: Unable to create
Vol[69|vm=72|ROOT]:com.cloud.utils.exception.CloudRuntimeException: Failed to
send to storage pool
at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2505)
at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2554)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:913)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:618)
at
org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227)
at
org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3329)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2889)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2875)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-07-31 13:35:08,751 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-77:job-291 = [ 6af942b2-9301-4aef-afff-3278581cc917 ]) Cleaning
up resources for the vm VM[User|test342] in Starting state
2013-07-31 13:35:08,754 DEBUG [agent.transport.Request]
(Job-Executor-77:job-291 = [ 6af942b2-9301-4aef-afff-3278581cc917 ]) Seq
1-1484588932: Sending { Cmd , MgmtId: 7635042566263, via: 1, Ver: v1, Flags:
100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"i-2-72-VM","wait":0}}]
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
