Abhinav Roy created CLOUDSTACK-4146:
---------------------------------------
Summary: [upgrade][2.2.13 -> 2.2.14 -> 4.2][KVM] After upgrade
creation of volumes from snapshots (which were created before upgrade) is
failing
Key: CLOUDSTACK-4146
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4146
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: KVM, Snapshot, Upgrade
Affects Versions: 4.2.0
Environment: upgrade from 2.2.13 (rhel 6.1 build) -> 2.2.14 (rhel 6.1
build) -> 4.2 (rhel 6.2 build)
MS : CentOS 6.1
KVM : CentOS 6.1
Reporter: Abhinav Roy
Priority: Critical
Fix For: 4.2.0
teps :
====================
1. Deploy CS 2.2.13 advanced zone setup with KVM.
2. Create VMs, template,snapshots, domains, accounts etc
3. Upgrade the management server and agent to 2.2.14
4. Create VMs , templates, snapshots, domain, accounts etc.
5. Have some VMs in stopped state
6. Upgrade the management server and agent to 4.2
Expected behaviour :
===================
After upgrade we should be able to create volumes from the sanpshots which were
created before upgrade
Observed behaviour:
===================
1. After upgrade i was able to create volumes from snapshots which were create
after upgrade
2. But creation of volumes from the sanpshots which were created before upgrade
is failing.
2013-08-07 16:38:39,245 DEBUG [storage.motion.AncientDataMotionStrategy]
(Job-Executor-10:job-65 = [ db7e5dca-3771-4a82-a72d-574c7e96d5e6 ]) copyAsync
inspecting src type SNAPSHOT copyAsync inspecting dest type VOLUME
2013-08-07 16:38:39,264 DEBUG [agent.transport.Request] (Job-Executor-10:job-65
= [ db7e5dca-3771-4a82-a72d-574c7e96d5e6 ]) Seq 3-1516699752: Sending { Cmd ,
MgmtId: 226870599129537, via: 3, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/9//snapshots/1/2/9/i-2-9-VM_ROOT-9_20130805162236","volume":{"uuid":"9","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"9fb7c84b-bfff-3ca2-9fc0-f50181c6b1f0","id":201,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/abhinav/kvm-pri2","port":2049}},"name":"ROOT-9","size":8589934592,"path":"c11c7a12-e9c7-495d-955b-6e814bb5b24f","volumeId":9,"vmName":"i-2-9-VM","accountId":2,"format":"QCOW2","id":9,"hypervisorType":"KVM"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.102.192.100/cpg_vol/abhinav/kvm-sec-old","_role":"Image"}},"vmName":"i-2-9-VM","name":"i-2-9-VM_ROOT-9_20130805162236","hypervisorType":"KVM","id":2}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"e3c99b55-b322-4c67-bef0-0dff0bd50e79","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"9fb7c84b-bfff-3ca2-9fc0-f50181c6b1f0","id":201,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/abhinav/kvm-pri2","port":2049}},"name":"volfromsnap-root7-2213","size":8589934592,"volumeId":34,"accountId":2,"format":"QCOW2","id":34,"hypervisorType":"None"}},"executeInSequence":false,"wait":10800}}]
}
2013-08-07 16:38:39,467 DEBUG [agent.transport.Request]
(AgentManager-Handler-6:null) Seq 3-1516699752: Processing: { Ans: , MgmtId:
226870599129537, via: 3, Ver: v1, Flags: 10,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
org.libvirt.LibvirtException: internal error '/bin/mount
10.102.192.100:/cpg_vol/abhinav/kvm-sec-old/snapshots/2/9/snapshots/1/2/9
/mnt/790b44cf-2022-3b27-8270-390d8e309290' exited with non-zero status 32 and
signal 0: mount.nfs: access denied by server while mounting
10.102.192.100:/cpg_vol/abhinav/kvm-sec-old/snapshots/2/9/snapshots/1/2/9\n","wait":0}}]
}
2013-08-07 16:38:39,467 DEBUG [agent.transport.Request] (Job-Executor-10:job-65
= [ db7e5dca-3771-4a82-a72d-574c7e96d5e6 ]) Seq 3-1516699752: Received: { Ans:
, MgmtId: 226870599129537, via: 3, Ver: v1, Flags: 10, { CopyCmdAnswer } }
2013-08-07 16:38:39,474 WARN [storage.datastore.ObjectInDataStoreManagerImpl]
(Job-Executor-10:job-65 = [ db7e5dca-3771-4a82-a72d-574c7e96d5e6 ]) Unsupported
data object (VOLUME,
org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@4dcf4da3), no need
to delete from object in store ref table
2013-08-07 16:38:39,490 WARN [storage.datastore.ObjectInDataStoreManagerImpl]
(Job-Executor-10:job-65 = [ db7e5dca-3771-4a82-a72d-574c7e96d5e6 ]) Snapshot 2
is not found on image store 4, so no need to delete
2013-08-07 16:38:39,490 DEBUG [cloud.storage.VolumeManagerImpl]
(Job-Executor-10:job-65 = [ db7e5dca-3771-4a82-a72d-574c7e96d5e6 ]) Failed to
create volume from snapshot:com.cloud.utils.exception.CloudRuntimeException:
org.libvirt.LibvirtException: internal error '/bin/mount
10.102.192.100:/cpg_vol/abhinav/kvm-sec-old/snapshots/2/9/snapshots/1/2/9
/mnt/790b44cf-2022-3b27-8270-390d8e309290' exited with non-zero status 32 and
signal 0: mount.nfs: access denied by server while mounting
10.102.192.100:/cpg_vol/abhinav/kvm-sec-old/snapshots/2/9/snapshots/1/2/9
2013-08-07 16:38:39,491 DEBUG [cloud.storage.VolumeManagerImpl]
(Job-Executor-10:job-65 = [ db7e5dca-3771-4a82-a72d-574c7e96d5e6 ]) Failed to
create volume: 34
com.cloud.utils.exception.CloudRuntimeException: Failed to create volume from
snapshot:com.cloud.utils.exception.CloudRuntimeException:
org.libvirt.LibvirtException: internal error '/bin/mount
10.102.192.100:/cpg_vol/abhinav/kvm-sec-old/snapshots/2/9/snapshots/1/2/9
/mnt/790b44cf-2022-3b27-8270-390d8e309290' exited with non-zero status 32 and
signal 0: mount.nfs: access denied by server while mounting
10.102.192.100:/cpg_vol/abhinav/kvm-sec-old/snapshots/2/9/snapshots/1/2/9
at
com.cloud.storage.VolumeManagerImpl.createVolumeFromSnapshot(VolumeManagerImpl.java:558)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
com.cloud.storage.VolumeManagerImpl.createVolumeFromSnapshot(VolumeManagerImpl.java:603)
at
com.cloud.storage.VolumeManagerImpl.createVolume(VolumeManagerImpl.java:1047)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
com.cloud.storage.VolumeManagerImpl.createVolume(VolumeManagerImpl.java:183)
at
org.apache.cloudstack.api.command.user.volume.CreateVolumeCmd.execute(CreateVolumeCmd.java:189)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
2013-08-07 16:38:39,521 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-10:job-65 = [ db7e5dca-3771-4a82-a72d-574c7e96d5e6 ]) Complete
async job-65 = [ db7e5dca-3771-4a82-a72d-574c7e96d5e6 ], jobStatus: 2,
resultCode: 530, result: Error Code: 530 Error text: Failed to create a volume
Attaching all DB dumps, agent logs and management server logs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
