Hi,
Last night I tried to create a snapshot manually from a volume.
The mangement server was performing the operation unfortunately got power
cycled through the process :(
This caused preventing me to do any further snapshots as the system said backup
is already in progress on that volume. Please try again later.
After hours it was still stuck saying the same (while vmware was not doing
anything at all) so I went to the 'snapshots' table in the DB and marked its
state from 'BackingUP' to 'Error'.
This allowed me to create snapshots again and started to export the OVF
template just fine (seen the progress via vcenter). After about an hour the
export failed around 15%.
Since then I cannot create any snapshots of that VM while can make from others
with no problems.
The log:
2012-06-14 10:42:19,918 WARN [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-8:job-1063) Cleanup failed due to BackupSnapshotCommand
exception: java.lang.Exception: Unable to finish the whole process to package
as a OVA file
2012-06-14 10:42:19,918 DEBUG [storage.snapshot.SnapshotManagerImpl]
(Job-Executor-8:job-1063) The result for
com.cloud.agent.api.BackupSnapshotCommand is BackupSnapshotCommand exception:
java.lang.Exception: Unable to finish the whole process to package as a OVA file
2012-06-14 10:42:19,919 WARN [cloud.vm.VirtualMachineManagerImpl]
(AgentManager-Handler-3:null) Cleanup failed due to BackupSnapshotCommand
exception: java.lang.Exception: Unable to finish the whole process to package
as a OVA file
2012-06-14 10:44:39,289 DEBUG [agent.transport.Request]
(AgentManager-Handler-2:null) Seq 10-577440210: Processing: { Ans: , MgmtId:
345049202747, via: 10, Ver: v1, Flags: 10,
[{"BackupSnapshotAnswer":{"full":true,"result":false,"details":"BackupSnapshotCommand
exception: java.lang.ArrayIndexOutOfBoundsException:
0\ncom.cloud.hypervisor.vmware.util.VmwareHelper.prepareDiskDevice(VmwareHelper.java:220)\ncom.cloud.hypervisor.vmware.mo.VirtualMachineMO.cloneFromDiskChain(VirtualMachineMO.java:1474)\ncom.cloud.hypervisor.vmware.mo.VirtualMachineMO.cloneFromCurrentSnapshot(VirtualMachineMO.java:1450)\ncom.cloud.hypervisor.vmware.manager.VmwareStorageManagerImpl.exportVolumeToSecondaryStroage(VmwareStorageManagerImpl.java:747)\ncom.cloud.hypervisor.vmware.manager.VmwareStorageManagerImpl.backupSnapshotToSecondaryStorage(VmwareStorageManagerImpl.java:714)\ncom.cloud.hypervisor.vmware.manager.VmwareStorageManagerImpl.execute(VmwareStorageManagerImpl.java:217)\ncom.cloud.storage.resource.VmwareSecondaryStorageResourceHandler.execute(VmwareSecondaryStorageResourceHandler.java:109)\ncom.cloud.storage.resource.VmwareSecondaryStorageResourceHandler.executeRequest(VmwareSecondaryStorageResourceHandler.java:67)\ncom.cloud.storage.resource.PremiumSecondaryStorageResource.executeRequest(PremiumSecondaryStorageResource.java:50)\ncom.cloud.agent.Agent.processRequest(Agent.java:517)\ncom.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:827)\ncom.cloud.utils.nio.Task.run(Task.java:79)\njava.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)\njava.lang.Thread.run(Thread.java:662)\n","wait":0}}]
}
2012-06-14 10:44:39,289 DEBUG [agent.transport.Request]
(Job-Executor-8:job-1063) Seq 10-577440210: Received: { Ans: , MgmtId:
345049202747, via: 10, Ver: v1, Flags: 10, { BackupSnapshotAnswer } }
2012-06-14 10:44:39,323 WARN [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-8:job-1063) Cleanup failed due to BackupSnapshotCommand
exception: java.lang.ArrayIndexOutOfBoundsException: 0
2012-06-14 10:44:39,324 DEBUG [storage.snapshot.SnapshotManagerImpl]
(Job-Executor-8:job-1063) The result for
com.cloud.agent.api.BackupSnapshotCommand is BackupSnapshotCommand exception:
java.lang.ArrayIndexOutOfBoundsException: 0
2012-06-14 10:44:39,324 WARN [cloud.vm.VirtualMachineManagerImpl]
(AgentManager-Handler-2:null) Cleanup failed due to BackupSnapshotCommand
exception: java.lang.ArrayIndexOutOfBoundsException: 0
2012-06-14 10:46:39,325 DEBUG [storage.snapshot.SnapshotManagerImpl]
(Job-Executor-8:job-1063) Retrying...
2012-06-14 10:46:39,329 WARN [storage.snapshot.SnapshotManagerImpl]
(Job-Executor-8:job-1063) Storage unavailable
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:201] is
unreachable: Unable to send command to the pool 201 due to there is no enabled
hosts up in this cluster
2012-06-14 10:46:39,330 WARN [storage.snapshot.SnapshotManagerImpl]
(Job-Executor-8:job-1063) Failed to back up snapshot on secondary storage,
deleting the record from the DB
2012-06-14 10:46:39,377 ERROR [cloud.api.ApiDispatcher]
(Job-Executor-8:job-1063) Exception while executing CreateSnapshotCmd:
com.cloud.utils.exception.CloudRuntimeException: Created snapshot:
com.cloud.storage.SnapshotVO$$EnhancerByCGLIB$$aaba1dbd@44374434<mailto:com.cloud.storage.SnapshotVO$$EnhancerByCGLIB$$aaba1dbd@44374434>
on primary but failed to backup on secondary
At the end it says Storage unavailable... I can parallel create other snapshots
with no problem so the storage is there.
That java.lang.ArrayIndexOutOfBoundsException is quite scary though.
Any advice or magic could sort this problem would be appreciated.
Regards
Tamas Monos DDI
+44(0)2034687012
Chief Technical Office
+44(0)2034687000
Veber: The Hosting Specialists Fax +44(0)871 522 7057
http://www.veber.co.uk<http://www.veber.co.uk/>
Follow us on Twitter:
www.twitter.com/veberhost<http://www.twitter.com/veberhost>
Follow us on Facebook:
www.facebook.com/veberhost<http://www.facebook.com/veberhost>
