[
https://issues.apache.org/jira/browse/CLOUDSTACK-4902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13806686#comment-13806686
]
Ivan Kozlov commented on CLOUDSTACK-4902:
-----------------------------------------
I think this bug may be related to
https://issues.apache.org/jira/browse/CLOUDSTACK-4939
My investigation shows that issue is caused by sending commands to random
hosts. so sometimes we send commands to host where vm is running and everything
is ok and sometimes command is sent not to host where vm is running so it fails.
Are you using sharedmountpoint or clvm as primary storage?
> Fail to create snapshot with KVM when run multiple Hosts in Cluster
> -------------------------------------------------------------------
>
> Key: CLOUDSTACK-4902
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4902
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM, Management Server
> Affects Versions: 4.2.0
> Environment: Managegemnet server: CentOS 6.3 64bit
> KVM Host: Ubuntu 12.04.1 64bit
> Reporter: le cong duan
> Priority: Critical
> Fix For: 4.2.0
>
>
> When only one Host in the Cluster, I always success to create snapshot for
> the volume by manual or automation.
> But when have multiple Host in Cluster. The creating is failed with status
> "CreatedOnPrimary", sometimes it is successfull.
> The following is error log on Management when occur error, there are three
> error situations.
> ------> First Situation
> 2013-10-18 22:46:27,163 DEBUG [agent.transport.Request]
> (Job-Executor-122:job-120 = [ d07688b0-
> bd86-4259-bdc2-441c36c4727d ]) Seq 11-714015661: Received: { Ans: , MgmtId:
> 113353561884, via: 11,
> Ver: v1, Flags: 110, { CopyCmdAnswer } }
> 2013-10-18 22:46:27,175 DEBUG [storage.snapshot.SnapshotManagerImpl]
> (Job-Executor-122:job-120 =
> [ d07688b0-bd86-4259-bdc2-441c36c4727d ]) Failed to create snapshot
> com.cloud.utils.exception.CloudRuntimeException:
> org.libvirt.LibvirtException: Domain snapshot not
> found: no snapshot with matching name '8a1d6db7-9ffe-43b5-a7df-df627329a168'
> at
> org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot
> (SnapshotServiceImpl.java:280)
> at
> org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot
> (XenserverSnapshotStrategy.java:138)
> at
> org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot
> (XenserverSnapshotStrategy.java:264)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot
> (SnapshotManagerImpl.java:1013)
> at com.cloud.utils.component.ComponentInstantiationPostProcessor
> $InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot
> (VolumeServiceImpl.java:1307)
> at
> com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2719)
> at
> org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute
> (CreateSnapshotCmd.java:170)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-10-18 22:46:27,181 DEBUG [storage.volume.VolumeServiceImpl]
> (Job-Executor-122:job-120 = [
> d07688b0-bd86-4259-bdc2-441c36c4727d ]) Take snapshot: 12 failed
> com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot
> (SnapshotManagerImpl.java:1040)
> at com.cloud.utils.component.ComponentInstantiationPostProcessor
> $InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot
> (VolumeServiceImpl.java:1307)
> at
> com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2719)
> at
> org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute
> (CreateSnapshotCmd.java:170)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: com.cloud.utils.exception.CloudRuntimeException:
> org.libvirt.LibvirtException: Domain
> snapshot not found: no snapshot with matching name
> '8a1d6db7-9ffe-43b5-a7df-df627329a168'
> at
> org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot
> (SnapshotServiceImpl.java:280)
> at
> org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot
> (XenserverSnapshotStrategy.java:138)
> at
> org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot
> (XenserverSnapshotStrategy.java:264)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot
> (SnapshotManagerImpl.java:1013)
> ... 16 more
> 2013-10-18 22:46:27,189 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-122:job-120 = [
> d07688b0-bd86-4259-bdc2-441c36c4727d ]) Complete async job-120 = [
> d07688b0-bd86-4259-bdc2-
> 441c36c4727d ], jobStatus: 2, resultCode: 530, result: Error Code: 530 Error
> text: Failed to create
> snapshot due to an internal error creating snapshot for volume 12
> ------> Second Situation
> 2013-10-18 23:30:23,010 DEBUG [agent.transport.Request]
> (AgentManager-Handler-10:null) Seq 8-
> 1133380250: Processing: { Ans: , MgmtId: 113353561884, via: 8, Ver: v1,
> Flags: 110,
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Failed
> to backup
> snapshot:
> /usr/share/cloudstack-common/scripts/storage/qcow2/managesnapshot.sh: line
> 121: 17900
> Segmentation fault (core dumped) $qemu_img snapshot -d \"$snapshotname\"
> $diskFailed to delete
> snapshot 0c6ec28f-eaaf-487a-a96c-cd58e6771367 for path
> /mnt/primary2/208ba7ed-ffcd-4cd7-9fd9-
> 924a3196cc08","wait":0}}] }
> 2013-10-18 23:30:23,010 DEBUG [agent.manager.AgentAttache]
> (AgentManager-Handler-10:null) Seq 8-
> 1133380250: No more commands found
> 2013-10-18 23:30:23,010 DEBUG [agent.transport.Request]
> (Job-Executor-131:job-129 = [ 1bd4b8c8-
> 0680-470b-8577-baf7fbcf0fd2 ]) Seq 8-1133380250: Received: { Ans: , MgmtId:
> 113353561884, via: 8, Ver:
> v1, Flags: 110, { CopyCmdAnswer } }
> 2013-10-18 23:30:23,022 DEBUG [storage.snapshot.SnapshotManagerImpl]
> (Job-Executor-131:job-129 =
> [ 1bd4b8c8-0680-470b-8577-baf7fbcf0fd2 ]) Failed to create snapshot
> com.cloud.utils.exception.CloudRuntimeException: Failed to backup snapshot:
> /usr/share/cloudstack-
> common/scripts/storage/qcow2/managesnapshot.sh: line 121: 17900 Segmentation
> fault (core
> dumped) $qemu_img snapshot -d "$snapshotname" $diskFailed to delete snapshot
> 0c6ec28f-eaaf-487a-
> a96c-cd58e6771367 for path /mnt/primary2/208ba7ed-ffcd-4cd7-9fd9-924a3196cc08
> at
> org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot
> (SnapshotServiceImpl.java:280)
> at
> org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot
> (XenserverSnapshotStrategy.java:138)
> at
> org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot
> (XenserverSnapshotStrategy.java:264)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot
> (SnapshotManagerImpl.java:1013)
> at com.cloud.utils.component.ComponentInstantiationPostProcessor
> $InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot
> (VolumeServiceImpl.java:1307)
> at
> com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2719)
> at
> org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute
> (CreateSnapshotCmd.java:170)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-10-18 23:30:23,028 DEBUG [storage.volume.VolumeServiceImpl]
> (Job-Executor-131:job-129 = [
> 1bd4b8c8-0680-470b-8577-baf7fbcf0fd2 ]) Take snapshot: 11 failed
> com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot
> (SnapshotManagerImpl.java:1040)
> at com.cloud.utils.component.ComponentInstantiationPostProcessor
> $InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot
> (VolumeServiceImpl.java:1307)
> at
> com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2719)
> at
> org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute
> (CreateSnapshotCmd.java:170)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to backup
> snapshot:
> /usr/share/cloudstack-common/scripts/storage/qcow2/managesnapshot.sh: line
> 121: 17900 Segmentation
> fault (core dumped) $qemu_img snapshot -d "$snapshotname" $diskFailed to
> delete snapshot
> 0c6ec28f-eaaf-487a-a96c-cd58e6771367 for path
> /mnt/primary2/208ba7ed-ffcd-4cd7-9fd9-924a3196cc08
> at
> org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot
> (SnapshotServiceImpl.java:280)
> at
> org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot
> (XenserverSnapshotStrategy.java:138)
> at
> org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot
> (XenserverSnapshotStrategy.java:264)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot
> (SnapshotManagerImpl.java:1013)
> ... 16 more
> 2013-10-18 23:30:23,036 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-131:job-129 = [
> 1bd4b8c8-0680-470b-8577-baf7fbcf0fd2 ]) Complete async job-129 = [
> 1bd4b8c8-0680-470b-8577-
> baf7fbcf0fd2 ], jobStatus: 2, resultCode: 530, result: Error Code: 530 Error
> text: Failed to create snapshot
> due to an internal error creating snapshot for volume 11
> -----> Third Situation.
> 2013-10-19 10:39:03,220 DEBUG [storage.snapshot.SnapshotManagerImpl]
> (Job-Executor-43:job-206 = [
> 661fcd14-8569-4e13-9e89-496adea6d054 ]) Failed to create snapshot
> com.cloud.utils.exception.CloudRuntimeException: Failed to backup
> 8ef1d6d8-50af-4204-8f79-
> 8b8cfd907d50 for disk /mnt/primary2/208ba7ed-ffcd-4cd7-9fd9-924a3196cc08 to
> /mnt/144a992e-7532-
> 316b-a15d-8e0ae22724c0/snapshots/2/11
> at
> org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot
> (SnapshotServiceImpl.java:280)
> at
> org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot
> (XenserverSnapshotStrategy.java:138)
> at
> org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot
> (XenserverSnapshotStrategy.java:264)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot
> (SnapshotManagerImpl.java:1013)
> at com.cloud.utils.component.ComponentInstantiationPostProcessor
> $InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot
> (VolumeServiceImpl.java:1307)
> at
> com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2719)
> at
> org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute
> (CreateSnapshotCmd.java:170)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-10-19 10:39:03,227 DEBUG [storage.volume.VolumeServiceImpl]
> (Job-Executor-43:job-206 = [
> 661fcd14-8569-4e13-9e89-496adea6d054 ]) Take snapshot: 11 failed
> com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot
> (SnapshotManagerImpl.java:1040)
> at com.cloud.utils.component.ComponentInstantiationPostProcessor
> $InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot
> (VolumeServiceImpl.java:1307)
> at
> com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2719)
> at
> org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute
> (CreateSnapshotCmd.java:170)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to backup
> 8ef1d6d8-50af-4204-
> 8f79-8b8cfd907d50 for disk /mnt/primary2/208ba7ed-ffcd-4cd7-9fd9-924a3196cc08
> to /mnt/144a992e-7532
> -316b-a15d-8e0ae22724c0/snapshots/2/11
> at
> org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot
> (SnapshotServiceImpl.java:280)
> at
> org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot
> (XenserverSnapshotStrategy.java:138)
> at
> org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot
> (XenserverSnapshotStrategy.java:264)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot
> (SnapshotManagerImpl.java:1013)
> ... 16 more
> 2013-10-19 10:39:03,235 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-43:job-206 = [
> 661fcd14-8569-4e13-9e89-496adea6d054 ]) Complete async job-206 = [
> 661fcd14-8569-4e13-9e89-
> 496adea6d054 ], jobStatus: 2, resultCode: 530, result: Error Code: 530 Error
> text: Failed to create
> snapshot due to an internal error creating snapshot for volume 11
--
This message was sent by Atlassian JIRA
(v6.1#6144)
