Chandan Purushothama created CLOUDSTACK-3741:
------------------------------------------------
Summary: Object_Store_Refactor - [KVM] Creation of Template from
Snapshot Fails
Key: CLOUDSTACK-3741
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3741
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Snapshot
Affects Versions: 4.2.0
Reporter: Chandan Purushothama
Priority: Critical
Fix For: 4.2.0
================
Steps to Reproduce:
================
1. On an Advanced Zone KVM Setup, Create a Template from a Snapshot of a ROOT
disk of a linux VM.
============
Observations:
============
-----------------------------------
Management Server Log:
-----------------------------------
Exception:
2013-07-23 10:54:51,493 DEBUG [agent.transport.Request]
(AgentManager-Handler-7:null) Seq 3-1039532081: Processing: { Ans: , MgmtId:
7471666038533, via: 3, Ver: v1, Flags: 10,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"java.io.FileNotFoundException:
/mnt/SecStorage/5aac6a6b-cfe3-35bb-9e20-4b2988244554/template/tmpl/3/203/template.properties
(No such file or directory)","wait":0}}] }
2013-07-23 10:54:51,493 DEBUG [agent.transport.Request] (Job-Executor-15:job-44
= [ 5439372e-dfea-42fc-875a-8c16c1588914 ]) Seq 3-1039532081: Received: { Ans:
, MgmtId: 7471666038533, via: 3, Ver: v1, Flags: 10, { CopyCmdAnswer } }
2013-07-23 10:54:51,508 DEBUG [cloud.template.TemplateManagerImpl]
(Job-Executor-15:job-44 = [ 5439372e-dfea-42fc-875a-8c16c1588914 ]) Failed to
create templatejava.io.FileNotFoundException:
/mnt/SecStorage/5aac6a6b-cfe3-35bb-9e20-4b2988244554/template/tmpl/3/203/template.properties
(No such file or directory)
2013-07-23 10:54:51,620 ERROR [cloud.async.AsyncJobManagerImpl]
(Job-Executor-15:job-44 = [ 5439372e-dfea-42fc-875a-8c16c1588914 ]) Unexpected
exception while executing
org.apache.cloudstack.api.command.user.template.CreateTemplateCmd
com.cloud.utils.exception.CloudRuntimeException: Failed to create
templatejava.io.FileNotFoundException:
/mnt/SecStorage/5aac6a6b-cfe3-35bb-9e20-4b2988244554/template/tmpl/3/203/template.properties
(No such file or directory)
at
com.cloud.template.TemplateManagerImpl.createPrivateTemplate(TemplateManagerImpl.java:1369)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.user.template.CreateTemplateCmd.execute(CreateTemplateCmd.java:263)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-07-23 10:54:51,640 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-15:job-44 = [ 5439372e-dfea-42fc-875a-8c16c1588914 ]) Complete
async job-44 = [ 5439372e-dfea-42fc-875a-8c16c1588914 ], jobStatus: 2,
resultCode: 530, result: Error Code: 530 Error text: Failed to create
templatejava.io.FileNotFoundException:
/mnt/SecStorage/5aac6a6b-cfe3-35bb-9e20-4b2988244554/template/tmpl/3/203/template.properties
(No such file or directory)
Create Template Job Information:
[root@asfmgmt ~]# grep -i "job-44"
/var/log/cloudstack/management/management-server.log
2013-07-23 10:53:40,592 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-4:null) submit async job-44 = [
5439372e-dfea-42fc-875a-8c16c1588914 ], details: AsyncJobVO {id:44, userId: 2,
accountId: 2, sessionKey: null, instanceType: Template, instanceId: 203, cmd:
org.apache.cloudstack.api.command.user.template.CreateTemplateCmd,
cmdOriginator: null, cmdInfo:
{"sessionkey":"280DZaDeR88+K7T7HCiizNFdVic\u003d","cmdEventType":"TEMPLATE.CREATE","ctxUserId":"2","httpmethod":"GET","osTypeId":"00905de2-f320-11e2-b49d-06cba2000705","isPublic":"true","response":"json","id":"203","displayText":"TemplateFromHourlySnapshot","snapshotid":"98b9db98-ca0b-48ee-88e5-a1c97edd76f4","passwordEnabled":"false","name":"TemplateFromHourlySnapshot","_":"1374602373320","ctxAccountId":"2","ctxStartEventId":"159"},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 7471666038533,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2013-07-23 10:53:40,596 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-15:job-44 = [ 5439372e-dfea-42fc-875a-8c16c1588914 ]) Executing
org.apache.cloudstack.api.command.user.template.CreateTemplateCmd for job-44 =
[ 5439372e-dfea-42fc-875a-8c16c1588914 ]
2013-07-23 10:53:40,648 DEBUG [storage.image.TemplateDataFactoryImpl]
(Job-Executor-15:job-44 = [ 5439372e-dfea-42fc-875a-8c16c1588914 ]) template
203 is already in store:1, type:Image
2013-07-23 10:53:40,655 DEBUG [storage.motion.AncientDataMotionStrategy]
(Job-Executor-15:job-44 = [ 5439372e-dfea-42fc-875a-8c16c1588914 ]) copyAsync
inspecting src type SNAPSHOT copyAsync inspecting dest type TEMPLATE
2013-07-23 10:53:40,678 DEBUG [agent.manager.AgentAttache]
(Job-Executor-15:job-44 = [ 5439372e-dfea-42fc-875a-8c16c1588914 ]) Request
seq: 1039532081
2013-07-23 10:53:40,678 DEBUG [agent.manager.AgentAttache]
(Job-Executor-15:job-44 = [ 5439372e-dfea-42fc-875a-8c16c1588914 ]) waiting to
send 1039532081
2013-07-23 10:53:40,678 DEBUG [agent.manager.AgentAttache]
(Job-Executor-15:job-44 = [ 5439372e-dfea-42fc-875a-8c16c1588914 ]) entering
synchronize block for sending 1039532081
2013-07-23 10:53:40,679 DEBUG [agent.transport.Request] (Job-Executor-15:job-44
= [ 5439372e-dfea-42fc-875a-8c16c1588914 ]) Seq 3-1039532081: Sending { Cmd ,
MgmtId: 7471666038533, via: 3, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/3/6/47574630-dd0a-4fca-8101-8e418f82b5b7","volume":{"uuid":"031802d8-c10a-4038-9acf-b86eba54b647","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"ef3dea99-0a1b-35b9-87ec-d8cabe7ed9c1","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.232","path":"/export/home/chandan/campocr-195-103/primarykvm","port":2049}},"name":"ROOT-6","size":147456,"path":"906ec8f4-e625-4943-aedd-aff3444150b9","volumeId":6,"vmName":"i-3-6-JULYLAST","accountId":3,"format":"QCOW2","id":6},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.223.110.232/export/home/chandan/campocr-195-103/secondary/","_role":"Image"}},"vmName":"i-3-6-JULYLAST","name":"Atoms-VM-1_ROOT-6_20130723175146","hypervisorType":"KVM","id":5}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/3/203","uuid":"56856579-ec76-4d4a-ab96-91eb611ccaeb","id":203,"format":"RAW","accountId":3,"hvm":true,"displayText":"TemplateFromHourlySnapshot","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.223.110.232/export/home/chandan/campocr-195-103/secondary/","_role":"Image"}},"name":"27329e06d-f0bf-3ce6-a2fe-839df2be1608"}},"executeInSequence":false,"wait":10800}}]
}
2013-07-23 10:54:51,493 DEBUG [agent.transport.Request] (Job-Executor-15:job-44
= [ 5439372e-dfea-42fc-875a-8c16c1588914 ]) Seq 3-1039532081: Received: { Ans:
, MgmtId: 7471666038533, via: 3, Ver: v1, Flags: 10, { CopyCmdAnswer } }
2013-07-23 10:54:51,508 DEBUG [cloud.template.TemplateManagerImpl]
(Job-Executor-15:job-44 = [ 5439372e-dfea-42fc-875a-8c16c1588914 ]) Failed to
create templatejava.io.FileNotFoundException:
/mnt/SecStorage/5aac6a6b-cfe3-35bb-9e20-4b2988244554/template/tmpl/3/203/template.properties
(No such file or directory)
2013-07-23 10:54:51,620 ERROR [cloud.async.AsyncJobManagerImpl]
(Job-Executor-15:job-44 = [ 5439372e-dfea-42fc-875a-8c16c1588914 ]) Unexpected
exception while executing
org.apache.cloudstack.api.command.user.template.CreateTemplateCmd
2013-07-23 10:54:51,640 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-15:job-44 = [ 5439372e-dfea-42fc-875a-8c16c1588914 ]) Complete
async job-44 = [ 5439372e-dfea-42fc-875a-8c16c1588914 ], jobStatus: 2,
resultCode: 530, result: Error Code: 530 Error text: Failed to create
templatejava.io.FileNotFoundException:
/mnt/SecStorage/5aac6a6b-cfe3-35bb-9e20-4b2988244554/template/tmpl/3/203/template.properties
(No such file or directory)
2013-07-23 10:54:52,866 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-6:null) Async job-44 = [ 5439372e-dfea-42fc-875a-8c16c1588914 ]
completed
[root@asfmgmt ~]#
----------------------------------
SSVM Logs Information:
----------------------------------
2013-07-23 17:59:34,594 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:)
Processing command: org.apache.cloudstack.storage.command.CopyCommand
2013-07-23 17:59:34,692 DEBUG [utils.script.Script] (agentRequest-Handler-4:)
Executing: /bin/bash -c cp
/mnt/SecStorage/5aac6a6b-cfe3-35bb-9e20-4b2988244554/snapshots/3/6/47574630-dd0a-4fca-8101-8e418f82b5b7
/mnt/SecStorage/5aac6a6b-cfe3-35bb-9e20-4b2988244554/template/tmpl/3/203
2013-07-23 17:59:42,375 DEBUG [cloud.agent.Agent] (UgentTask-5:) Sending ping:
Seq 3-1113: { Cmd , MgmtId: -1, via: 3, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.PingStorageCommand":{"changes":{},"hostType":"Storage","hostId":3,"wait":0}}]
}
2013-07-23 17:59:42,409 DEBUG [cloud.agent.Agent] (Agent-Handler-3:) Received
response: Seq 3-1113: { Ans: , MgmtId: 7471666038533, via: 3, Ver: v1, Flags:
100010,
[{"com.cloud.agent.api.PingAnswer":{"_command":{"hostType":"Storage","hostId":3,"wait":0},"result":true,"wait":0}}]
}
2013-07-23 18:00:03,511 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:)
Processing command: com.cloud.agent.api.GetStorageStatsCommand
2013-07-23 18:00:42,375 DEBUG [cloud.agent.Agent] (UgentTask-5:) Sending ping:
Seq 3-1114: { Cmd , MgmtId: -1, via: 3, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.PingStorageCommand":{"changes":{},"hostType":"Storage","hostId":3,"wait":0}}]
}
2013-07-23 18:00:42,378 DEBUG [cloud.agent.Agent] (Agent-Handler-5:) Received
response: Seq 3-1114: { Ans: , MgmtId: 7471666038533, via: 3, Ver: v1, Flags:
100010,
[{"com.cloud.agent.api.PingAnswer":{"_command":{"hostType":"Storage","hostId":3,"wait":0},"result":true,"wait":0}}]
}
2013-07-23 18:00:45,207 DEBUG [utils.script.Script] (agentRequest-Handler-4:)
Execution is successful.
2013-07-23 18:00:45,226 DEBUG [storage.template.QCOW2Processor]
(agentRequest-Handler-4:) Unable to find the qcow2 file:
/mnt/SecStorage/5aac6a6b-cfe3-35bb-9e20-4b2988244554/template/tmpl/3/203/47574630-dd0a-4fca-8101-8e418f82b5b7.qcow2
2013-07-23 18:00:45,329 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:) Seq
3-1039532081: { Ans: , MgmtId: 7471666038533, via: 3, Ver: v1, Flags: 10,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"java.io.FileNotFoundException:
/mnt/SecStorage/5aac6a6b-cfe3-35bb-9e20-4b2988244554/template/tmpl/3/203/template.properties
(No such file or directory)","wait":0}}] }
======================
On the Secondary Storage:
======================
[root@nfs2 snapshots]# cd 3/6/
[root@nfs2 6]# ls
0ded9e25-38d0-4828-884a-1629f9f761a4 47574630-dd0a-4fca-8101-8e418f82b5b7
[root@nfs2 6]#
[root@nfs2 6]# cd ../../../template/tmpl/3/203/
[root@nfs2 203]# ls
47574630-dd0a-4fca-8101-8e418f82b5b7
[root@nfs2 203]#
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
