Hi Prashant,
1 - Yeha actually I noticed that, I am not able to create any new VMs from any
other template.
I just tried creating one Ubuntu instance , got this error in logs,
2015-04-06 13:12:25,300 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-27:ctx-149fc40a job-2588/job-2589) Unable to complete
AsyncJobVO {id:2589, userId: 2, accountId: 2, instanceType: null, instanceId:
null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAAQ3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAXBzcgAOamF2YS5sYW5nLkxvbmc7i-SQzI8j3wIAAUoABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAAAAAAAAAFxAH4ACnBwcHEAfgAKcHNyABFqYXZhLnV0aWwuSGFzaE1hcAUH2sHDFmDRAwACRgAKbG9hZEZhY3RvckkACXRocmVzaG9sZHhwP0AAAAAAAAx3CAAAABAAAAABdAAKVm1QYXNzd29yZHQAHHJPMEFCWFFBRG5OaGRtVmtYM0JoYzNOM2IzSmt4cA,
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 59778234354585, completeMsid: null, lastUpdated: null,
lastPolled: null, created: Mon Apr 06 13:12:13 CEST 2015}, job origin:2588
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM[User|i-2-67-VM]Scope=interface com.cloud.dc.DataCenter; id=1
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:947)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5195)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5340)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2015-04-06 13:12:25,302 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-27:ctx-149fc40a job-2588/job-2589) Complete async job-2589,
jobStatus: FAILED, resultCode: 0,
2015-04-06 13:12:25,396 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-27:ctx-149fc40a job-2588/job-2589) Done executing
com.cloud.vm.VmWorkStart for job-2589
2015-04-06 13:12:25,482 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-9:ctx-5e6ac400 job-2588 ctx-0c8863ad) Sync job-2590 execution
on object VmWorkJobQueue.67
2015-04-06 13:12:25,486 WARN [c.c.u.d.Merovingian2]
(API-Job-Executor-9:ctx-5e6ac400 job-2588 ctx-0c8863ad) Was unable to find lock
for the key vm_instance67 and thread id 1697914126
After some time this was 2nd exception about same instance,
2015-04-06 13:12:32,661 INFO [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-29:ctx-467f37f1 job-2588/job-2590 ctx-4b4246ef) Unable to
contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1] is
unreachable: Unable to create Vol[84|vm=67|ROOT]:Can't find template VDI under:
192.168.48.10:/tank/secstore/template/tmpl/1/5
at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1205)
at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1257)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:988)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5195)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5340)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
then this,
015-04-06 13:12:44,501 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-363:ctx-fa031563) Failed to destroy pbd
SR_BACKEND_FAILURE_40The SR scan failed [opterr=['INTERNAL_ERROR',
'Db_exn.Uniqueness_constraint_violation("VDI", "uuid",
"73b81b8a-ec8f-3423-b2d3-7a0ca1b0ac13")']]
at com.xensource.xenapi.Types.checkResponse(Types.java:2006)
at com.xensource.xenapi.Connection.dispatch(Connection.java:350)
at
com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:456)
at com.xensource.xenapi.SR.scan(SR.java:1051)
at
com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.createFileSR(Xenserver625StorageProcessor.java:107)
at
com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.createFileSr(Xenserver625StorageProcessor.java:133)
at
com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.copyTemplateToPrimaryStorage(Xenserver625StorageProcessor.java:167)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:77)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:547)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61)
at
com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102)
at
com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:307)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2015-04-06 13:12:44,509 ERROR [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-363:ctx-fa031563) Failed to forget sr
SR_BACKEND_FAILURE_40The SR scan failed [opterr=['INTERNAL_ERROR',
'Db_exn.Uniqueness_constraint_violation("VDI", "uuid",
"73b81b8a-ec8f-3423-b2d3-7a0ca1b0ac13")']]
at com.xensource.xenapi.Types.checkResponse(Types.java:2006)
at com.xensource.xenapi.Connection.dispatch(Connection.java:350)
at
com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:456)
at com.xensource.xenapi.SR.scan(SR.java:1051)
at
com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.createFileSR(Xenserver625StorageProcessor.java:107)
at
com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.createFileSr(Xenserver625StorageProcessor.java:133)
at
com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.copyTemplateToPrimaryStorage(Xenserver625StorageProcessor.java:167)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:77)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:547)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61)
at
com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102)
at
com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:307)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2015-04-06 13:12:44,510 WARN [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-363:ctx-fa031563) createFileSR failed! due to
SR_BACKEND_FAILURE_40The SR scan failed [opterr=['INTERNAL_ERROR',
'Db_exn.Uniqueness_constraint_violation("VDI", "uuid",
"73b81b8a-ec8f-3423-b2d3-7a0ca1b0ac13")']]
SR_BACKEND_FAILURE_40The SR scan failed [opterr=['INTERNAL_ERROR',
'Db_exn.Uniqueness_constraint_violation("VDI", "uuid",
"73b81b8a-ec8f-3423-b2d3-7a0ca1b0ac13")']]
at com.xensource.xenapi.Types.checkResponse(Types.java:2006)
at com.xensource.xenapi.Connection.dispatch(Connection.java:350)
at
com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:456)
at com.xensource.xenapi.SR.scan(SR.java:1051)
at
com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.createFileSR(Xenserver625StorageProcessor.java:107)
at
com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.createFileSr(Xenserver625StorageProcessor.java:133)
at
com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.copyTemplateToPrimaryStorage(Xenserver625StorageProcessor.java:167)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:77)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:547)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61)
at
com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102)
at
com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:307)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2015-04-06 13:12:44,512 WARN [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-363:ctx-fa031563) Catch Exception
com.cloud.utils.exception.CloudRuntimeException for template due to
com.cloud.utils.exception.CloudRuntimeException: createFileSR failed! due to
SR_BACKEND_FAILURE_40The SR scan failed [opterr=['INTERNAL_ERROR',
'Db_exn.Uniqueness_constraint_violation("VDI", "uuid",
"73b81b8a-ec8f-3423-b2d3-7a0ca1b0ac13")']]
com.cloud.utils.exception.CloudRuntimeException: createFileSR failed! due to
SR_BACKEND_FAILURE_40The SR scan failed [opterr=['INTERNAL_ERROR',
'Db_exn.Uniqueness_constraint_violation("VDI", "uuid",
"73b81b8a-ec8f-3423-b2d3-7a0ca1b0ac13")']]
at
com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.createFileSR(Xenserver625StorageProcessor.java:126)
at
com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.createFileSr(Xenserver625StorageProcessor.java:133)
at
com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.copyTemplateToPrimaryStorage(Xenserver625StorageProcessor.java:167)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:77)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:547)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61)
at
com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102)
at
com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:307)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: SR_BACKEND_FAILURE_40The SR scan failed [opterr=['INTERNAL_ERROR',
'Db_exn.Uniqueness_constraint_violation("VDI", "uuid",
"73b81b8a-ec8f-3423-b2d3-7a0ca1b0ac13")']]
at com.xensource.xenapi.Types.checkResponse(Types.java:2006)
at com.xensource.xenapi.Connection.dispatch(Connection.java:350)
at
com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:456)
at com.xensource.xenapi.SR.scan(SR.java:1051)
at
com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.createFileSR(Xenserver625StorageProcessor.java:107)
... 21 more
2015-04-06 13:12:44,513 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-363:ctx-fa031563) Seq 1-4183562578850489418: Response Received:
2015-04-06 13:12:44,513 DEBUG [c.c.a.t.Request] (DirectAgent-363:ctx-fa031563)
Seq 1-4183562578850489418: Processing: { Ans: , MgmtId: 59778234354585, via:
1, Ver: v1, Flags: 10,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Catch
Exception com.cloud.utils.exception.CloudRuntimeException for template due to
com.cloud.utils.exception.CloudRuntimeException: createFileSR failed! due to
SR_BACKEND_FAILURE_40The SR scan failed [opterr=['INTERNAL_ERROR',
'Db_exn.Uniqueness_constraint_violation(\"VDI\", \"uuid\",
\"73b81b8a-ec8f-3423-b2d3-7a0ca1b0ac13\")']]","wait":0}}] }
2015-04-06 13:12:44,513 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-28:ctx-42c09fe7 job-2588/job-2590 ctx-0601a974) Seq
1-4183562578850489418: Received: { Ans: , MgmtId: 59778234354585, via: 1, Ver:
v1, Flags: 10, { CopyCmdAnswer } }
2015-04-06 13:12:44,616 INFO [o.a.c.s.v.VolumeServiceImpl]
(Work-Job-Executor-28:ctx-42c09fe7 job-2588/job-2590 ctx-0601a974) releasing
lock for VMTemplateStoragePool 40
2015-04-06 13:12:44,617 WARN [c.c.u.d.Merovingian2]
(Work-Job-Executor-28:ctx-42c09fe7 job-2588/job-2590 ctx-0601a974) Was unable
to find lock for the key template_spool_ref40 and thread id 1105922582
2015-04-06 13:12:44,617 DEBUG [o.a.c.e.o.VolumeOrchestrator]
(Work-Job-Executor-28:ctx-42c09fe7 job-2588/job-2590 ctx-0601a974) Unable to
create Vol[84|vm=67|ROOT]:Catch Exception
com.cloud.utils.exception.CloudRuntimeException for template due to
com.cloud.utils.exception.CloudRuntimeException: createFileSR failed! due to
SR_BACKEND_FAILURE_40The SR scan failed [opterr=['INTERNAL_ERROR',
'Db_exn.Uniqueness_constraint_violation("VDI", "uuid",
"73b81b8a-ec8f-3423-b2d3-7a0ca1b0ac13")']]
2015-04-06 13:12:44,617 INFO [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-28:ctx-42c09fe7 job-2588/job-2590 ctx-0601a974) Unable to
contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1] is
unreachable: Unable to create Vol[84|vm=67|ROOT]:Catch Exception
com.cloud.utils.exception.CloudRuntimeException for template due to
com.cloud.utils.exception.CloudRuntimeException: createFileSR failed! due to
SR_BACKEND_FAILURE_40The SR scan failed [opterr=['INTERNAL_ERROR',
'Db_exn.Uniqueness_constraint_violation("VDI", "uuid",
"73b81b8a-ec8f-3423-b2d3-7a0ca1b0ac13")']]
at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1205)
at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1257)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:988)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5195)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5340)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2015-04-06 13:12:44,667 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-28:ctx-42c09fe7 job-2588/job-2590 ctx-0601a974) Cleaning up
resources for the vm VM[User|i-2-67-VM] in Starting state
2015-04-06 13:12:41,676 DEBUG [o.a.c.e.o.VolumeOrchestrator]
(Work-Job-Executor-28:ctx-42c09fe7 job-2588/job-2590 ctx-0601a974) Checking if
we need to prepare 1 volumes for VM[User|i-2-67-VM]
2015-04-06 13:12:33,079 DEBUG [c.c.v.d.VMInstanceDaoImpl]
(Work-Job-Executor-29:ctx-467f37f1 job-2588/job-2590 ctx-4b4246ef) Unable to
update VM[User|i-2-67-VM]: DB Data={Host=6; State=Starting; updated=5; time=Mon
Apr 06 13:12:26 CEST 2015} New Data: {Host=5; State=Starting; updated=6;
time=Mon Apr 06 13:12:26 CEST 2015} Stale Data: {Host=4; State=Starting;
updated=5; time=Mon Apr 06 13:12:26 CEST 2015}
2015-04-06 13:12:33,079 DEBUG [c.c.v.d.VMInstanceDaoImpl]
(Work-Job-Executor-29:ctx-467f37f1 job-2588/job-2590 ctx-4b4246ef) VM i-2-67-VM
state has been already been updated to Starting
2015-04-06 13:12:33,079 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-29:ctx-467f37f1 job-2588/job-2590 ctx-4b4246ef) VM state
transitted from :Starting to Starting with event: OperationRetryvm's original
host id: null new host id: 5 host id before state transition: 4
2015-04-06 13:12:44,913 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-28:ctx-42c09fe7 job-2588/job-2590 ctx-0601a974) Deployment
found - P0=VM[User|i-2-67-VM],
P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
: Dest[Zone(1)-Pod(1)-Cluster(1)-Host(5)-Storage(Volume(84|ROOT-->Pool(1))]
2015-04-06 13:12:45,013 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-28:ctx-42c09fe7 job-2588/job-2590 ctx-0601a974) VM state
transitted from :Starting to Starting with event: OperationRetryvm's original
host id: null new host id: 5 host id before state transition: 6
Template its saying not able to find, does exist at that location. I don’t know
why its throwing that exception.
2 - My ssvm, routers and storage all are fine. I have few production VMs
running , with no issues.
Yes my template was in ready , download completed state.
3 - For /usr/local/cloudstack/systemvm/ssvm-check.sh , all check are fine.
4 - info I selected foe template is correct .
5 - of course..... :/
Its weird understating what's causing problem.
/Sonali
-----Original Message-----
From: Prashant s [mailto:[email protected]]
Sent: Monday, April 6, 2015 7:30 AM
To: [email protected]
Subject: Re: problem creating instance from template
couple of questions for you Sonali
1. Are you able to build any new vms using any OTHER template that you
already have. try to create a test vm
2. cloudstack pit stop checks from web interface : from your ACS interface
check if there are any errors ?
a ) can check if any of your hosts ,
b) primary or secondary storage
c) systems vms
d) router vms
are in errored out state ?
e) is your template in READY state , download completed .. ?
3. login to system vm and run the ssvm check
/usr/local/cloudstack/systemvm/ssvm-check.sh ,
also while on the ssvm , check the *SSVM Logs* - /var/log/cloudstack/cloud.log
4. when you registered your template did you chose the right OS version etc ...
5.* does your server have the Intel Virtualization Technology (vt-x) or amd-v
in BIOS turned on , for windows you really need this on :-0*
i can only think of these issues & checks ....
On Sat, Apr 4, 2015 at 7:45 AM, Sonali Jadhav <[email protected]>
wrote:
> [root@XS04 ~]# xe vdi-list uuid=32719160-52d8-4e45-9682-7ab9ada4dad1
> uuid ( RO) : 32719160-52d8-4e45-9682-7ab9ada4dad1
> name-label ( RW):
> name-description ( RW):
> sr-uuid ( RO): 39311fd4-822d-4ccc-8c5c-bbb7fda4def5
> virtual-size ( RO): 107374182400
> sharable ( RO): false
> read-only ( RO): false
>
> Thanks,
> /Sonali
>
> -----Original Message-----
> From: giljae o [mailto:[email protected]]
> Sent: Saturday, April 4, 2015 3:11 PM
> To: [email protected]
> Subject: Re: problem creating instance from template
>
> Then,
>
> Can you do the command in XenServer where you are trying to deploy VMs
>
> xe vdi-list uuid=32719160-52d8-4e45-9682-7ab9ada4dad1
>
> James,
>
> On Sat, Apr 4, 2015 at 6:08 PM, Sonali Jadhav
> <[email protected]>
> wrote:
>
> > No, nothing that I am aware of. I have not even created manually.
> >
> > /Sonali
> >
> > -----Original Message-----
> > From: giljae o [mailto:[email protected]]
> > Sent: Saturday, April 4, 2015 2:18 PM
> > To: [email protected]
> > Subject: Re: problem creating instance from template
> >
> > Hi
> >
> > For deploying VM CS needs to download a template from 2nd Storage,
> > In the cluster where the template is copied to there the VDI uuid
> > was
> conflicted.
> > Is there any VDI on the pool CS does not know about?
> >
> >
> >
> > [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org
> > .a
> > pache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl
> > /2
> > /212/32719160-52d8-4e45-9682-7ab9ada4dad1.vhd","uuid":"34b2b442-81cd
> > -4
> > 56b-938d-6670bcfcaf01","id":212,"format":"VHD","accountId":2,"hvm":t
> > ru
> > e,"displayText":"Windows
> > Server 2012 R2
> > Std","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs:/
> > /
> > 192.168.48.10/tank/secstore
> > ","_role":"Image"}},"name":"32719160-52d8-4e45-9682-7ab9ada4dad1","h
> > yp
> > ervisorType":"XenServer"}},"destTO":{"org.apache.cloudstack.storage.
> > to
> > .TemplateObjectTO":{"uuid":"34b2b442-81cd-456b-938d-6670bcfcaf01","id"
> > :212,"format":"VHD","accountId":2,"hvm":true,"displayText":"Windows
> > Server 2012 R2
> >
> > Std","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryData
> > St
> > oreTO":{"uuid":"876d490c-a1d4-3bfe-88b7-1bdb2479541b","id":1,"poolTy
> > pe
> > ":"NetworkFilesystem","host":"192.168.6.194","path":"/tank/primstore
> > ", "port":2049,"url":"NetworkFilesystem://
> > 192.168.6.194/tank/primstore/?ROLE=Primary&STOREUUID=876d490c-
> >
> >
> > 2015-04-03 17:10:16,786 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
> > (DirectAgent-99:ctx-e6b464d2) Failed to destroy pbd
> > SR_BACKEND_FAILURE_40The SR scan failed [opterr=['INTERNAL_ERROR',
> > 'Db_exn.Uniqueness_constraint_violation("VDI", "uuid",
> > "32719160-52d8-4e45-9682-7ab9ada4dad1")']]
> > at com.xensource.xenapi.Types.checkResponse(Types.java:2006)
> > at com.xensource.xenapi.Connection.dispatch(Connection.java:350)
> > at
> >
> > com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerC
> > on
> > nection.dispatch(XenServerConnectionPool.java:456)
> > at com.xensource.xenapi.SR.scan(SR.java:1051)
> > at
> >
> > com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.creat
> > eF
> > ileSR(Xenserver625StorageProcessor.java:107)
> > at
> >
> > com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.creat
> > eF
> > ileSr(Xenserver625StorageProcessor.java:133)
> > at
> >
> > com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.copyT
> > em
> > plateToPrimaryStorage(Xenserver625StorageProcessor.java:167)
> > at
> >
> > com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execut
> > e(
> > StorageSubsystemCommandHandlerBase.java:77)
> > at
> >
> > com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handle
> > St
> > orageCommands(StorageSubsystemCommandHandlerBase.java:52)
> > at
> >
> > com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(
> > Ci
> > trixResourceBase.java:547)
> > at
> >
> > com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest
> > (X
> > enServer56Resource.java:61)
> > at
> >
> > com.cloud.hypervisor.xen.resource.XenServer610Resource.executeReques
> > t(
> > XenServer610Resource.java:102)
> > at
> >
> > com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeReq
> > ue
> > st(XenServer620SP1Resource.java:65)
> > at
> >
> > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectA
> > ge
> > ntAttache.java:307)
> > at
> >
> > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(M
> > an
> > agedContextRunnable.java:49)
> > at
> >
> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.c
> > al
> > l(DefaultManagedContext.java:56)
> > at
> >
> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.cal
> > lW
> > ithContext(DefaultManagedContext.java:103)
> > at
> >
> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.run
> > Wi
> > thContext(DefaultManagedContext.java:53)
> > at
> >
> > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Man
> > ag
> > edContextRunnable.java:46) at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:4
> > 71
> > ) at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > at
> >
> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
> > .a
> > ccess$201(ScheduledThreadPoolExecutor.java:178)
> > at
> >
> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
> > .r
> > un(ScheduledThreadPoolExecutor.java:292)
> > at
> >
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor
> > .j
> > ava:1145)
> > at
> >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
> > java:615) at java.lang.Thread.run(Thread.java:744)
> >
> >
> > On Sat, Apr 4, 2015 at 5:21 PM, Sonali Jadhav
> > <[email protected]>
> > wrote:
> >
> > > Hi,
> > >
> > > I have created windows 2012 r2 template. And I am trying to create
> > > instance from it. But its failing, I am not able to understand cause.
> > >
> > > Here is my log file,
> > > https://www.dropbox.com/s/4p4pskiy954e697/mgmtTemplateError.log?dl
> > > =0
> > >
> > > I tried lot of time, somewhere it also said that insufficient
> > > capacity on host. Which I would say not true at all. I got 4
> > > xenservers in one
> > cluster.
> > >
> > > Anyone can help me understand issue?
> > >
> > > /Sonali
> > >
> >
>
