Dear all, In addition to my earlier email, please find below the excerpt from the SSVM logs on /var/log/cloud.log. The SSVM doesn't have any issues mounting to the secondary storage, but it seems to be having issues in installing the template even after it has downloaded the template successfully.
==== 2016-05-12 03:25:49,891 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:null) Processing command: com.cloud.agent.api.GetStorageStatsCommand 2016-05-12 03:25:49,891 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-1:null) Determined host X.X.0.242 corresponds to IP X.X.0.242 2016-05-12 03:25:49,892 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-1:null) Mounting device with nfs-style path of X.X.0.242:/export/cloudstack 2016-05-12 03:25:49,892 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-1:null) mount nfs://X.X.0.242/export/cloudstack on /mnt/SecStorage/1b8d363c-1a86-325d-b1b8-3c030ea93856 2016-05-12 03:25:49,892 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-1:null) making available /mnt/SecStorage/1b8d363c-1a86-325d-b1b8-3c030ea93856 on nfs://X.X.0.242/export/cloudstack 2016-05-12 03:25:49,892 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-1:null) local folder for mount will be /mnt/SecStorage/1b8d363c-1a86-325d-b1b8-3c030ea93856 2016-05-12 03:25:49,894 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-1:null) Executing: mount 2016-05-12 03:25:49,899 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-1:null) Execution is successful. 2016-05-12 03:25:49,899 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-1:null) Some device already mounted at /mnt/SecStorage/1b8d363c-1a86-325d-b1b8-3c030ea93856, no need to mount nfs://X.X.0.242/export/cl oudstack 2016-05-12 03:25:57,846 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) Request:Seq 3-7000845620747442558: { Cmd , MgmtId: 181122461655966, via: 3, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DownloadProgress Command":{"jobId":"49e4c4dd-7e12-455f-9293-f7c50659992d","request":"GET_STATUS","hvm":true,"description":"Ubuntu 14.04 LTS (64-bit) ISO","maxDownloadSizeInBytes":53687091200,"id":203,"resourceType":"TEMPLATE","installPath":"template/tmpl/ 2/203","_store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://X.X.0.242/export/cloudstack","_role":"Image"}},"url":" http://xxx.xxxxxxx.com/iso/Ubuntu/Ubuntu%2014.04%20LTS/ubuntu-14.04-server-amd64.iso ","format":"ISO","accountId" :2,"name":"203-2-72045f68-861c-31b8-adf6-19df376a7411","secUrl":"nfs://X.X.0.242/export/cloudstack","wait":0}}] } 2016-05-12 03:25:57,846 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) Processing command: org.apache.cloudstack.storage.command.DownloadProgressCommand 2016-05-12 03:25:57,847 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) Seq 3-7000845620747442558: { Ans: , MgmtId: 181122461655966, via: 3, Ver: v1, Flags: 10, [{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"49e4c4dd-7e 12-455f-9293-f7c50659992d","downloadPct":100,"errorString":"Downloaded 591396864 bytes (download complete remote=591396864bytes)","downloadStatus":"DOWNLOAD_IN_PROGRESS","downloadPath":"/mnt/SecStorage/1b8d363c-1a86-325d-b1b8-3c030ea93856 /template/tmpl/2/203/dnld7251963221322463805tmp_","installPath":"template/tmpl/2/203","templateSize":0,"templatePhySicalSize":0,"result":true,"details":"Downloaded 591396864 bytes (download complete remote=591396864bytes)","wait":0}}] } 2016-05-12 03:25:57,848 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) Request:Seq 3-7000845620747442559: { Cmd , MgmtId: 181122461655966, via: 3, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DownloadProgress Command":{"jobId":"2a1b2549-6af5-4e75-913c-7f6ac264f2ae","request":"GET_STATUS","hvm":false,"description":"CentOS 5.5(64-bit) no GUI (KVM)","checksum":"ed0e788280ff2912ea40f7f91ca7a249","maxDownloadSizeInBytes":53687091200,"id":4,"resourc eType":"TEMPLATE","installPath":"template/tmpl/1/4","_store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://X.X.0.242/export/cloudstack","_role":"Image"}},"url":" http://download.cloud.com/releases/2.2.0/eec2209b-9875-3c8d-92be-c001bd8 a0faf.qcow2.bz2","format":"QCOW2","accountId":1,"name":"centos55-x86_64","secUrl":"nfs://X.X.0.242/export/cloudstack","wait":0}}] } 2016-05-12 03:25:57,849 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) Processing command: org.apache.cloudstack.storage.command.DownloadProgressCommand 2016-05-12 03:25:57,849 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) Seq 3-7000845620747442559: { Ans: , MgmtId: 181122461655966, via: 3, Ver: v1, Flags: 10, [{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"2a1b2549-6a f5-4e75-913c-7f6ac264f2ae","downloadPct":100,"errorString":"Downloaded 472312242 bytes (download complete remote=472312242bytes)","downloadStatus":"DOWNLOAD_IN_PROGRESS","downloadPath":"/mnt/SecStorage/1b8d363c-1a86-325d-b1b8-3c030ea93856 /template/tmpl/1/4/dnld1030975883579196189tmp_","installPath":"template/tmpl/1/4","templateSize":0,"templatePhySicalSize":0,"checkSum":"ed0e788280ff2912ea40f7f91ca7a249","result":true,"details":"Downloaded 472312242 bytes (download comple te remote=472312242bytes)","wait":0}}] } 2016-05-12 03:26:17,865 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) Request:Seq 3-7000845620747442560: { Cmd , MgmtId: 181122461655966, via: 3, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DownloadProgress Command":{"jobId":"49e4c4dd-7e12-455f-9293-f7c50659992d","request":"GET_STATUS","hvm":true,"description":"Ubuntu 14.04 LTS (64-bit) ISO","maxDownloadSizeInBytes":53687091200,"id":203,"resourceType":"TEMPLATE","installPath":"template/tmpl/ 2/203","_store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://X.X.0.242/export/cloudstack","_role":"Image"}},"url":" http://xxx.xxxxxxx.com/iso/Ubuntu/Ubuntu%2014.04%20LTS/ubuntu-14.04-server-amd64.iso ","format":"ISO","accountId" :2,"name":"203-2-72045f68-861c-31b8-adf6-19df376a7411","secUrl":"nfs://X.X.0.242/export/cloudstack","wait":0}}] } 2016-05-12 03:26:17,865 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) Processing command: org.apache.cloudstack.storage.command.DownloadProgressCommand 2016-05-12 03:26:17,866 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) Seq 3-7000845620747442560: { Ans: , MgmtId: 181122461655966, via: 3, Ver: v1, Flags: 10, [{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"49e4c4dd-7e 12-455f-9293-f7c50659992d","downloadPct":100,"errorString":"Downloaded 591396864 bytes (download complete remote=591396864bytes)","downloadStatus":"DOWNLOAD_IN_PROGRESS","downloadPath":"/mnt/SecStorage/1b8d363c-1a86-325d-b1b8-3c030ea93856 /template/tmpl/2/203/dnld7251963221322463805tmp_","installPath":"template/tmpl/2/203","templateSize":0,"templatePhySicalSize":0,"result":true,"details":"Downloaded 591396864 bytes (download complete remote=591396864bytes)","wait":0}}] } 2016-05-12 03:26:17,868 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:null) Request:Seq 3-7000845620747442561: { Cmd , MgmtId: 181122461655966, via: 3, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DownloadProgress Command":{"jobId":"2a1b2549-6af5-4e75-913c-7f6ac264f2ae","request":"GET_STATUS","hvm":false,"description":"CentOS 5.5(64-bit) no GUI (KVM)","checksum":"ed0e788280ff2912ea40f7f91ca7a249","maxDownloadSizeInBytes":53687091200,"id":4,"resourc eType":"TEMPLATE","installPath":"template/tmpl/1/4","_store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://X.X.0.242/export/cloudstack","_role":"Image"}},"url":" http://download.cloud.com/releases/2.2.0/eec2209b-9875-3c8d-92be-c001bd8 a0faf.qcow2.bz2","format":"QCOW2","accountId":1,"name":"centos55-x86_64","secUrl":"nfs://X.X.0.242/export/cloudstack","wait":0}}] } 2016-05-12 03:26:17,868 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:null) Processing command: org.apache.cloudstack.storage.command.DownloadProgressCommand 2016-05-12 03:26:17,869 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:null) Seq 3-7000845620747442561: { Ans: , MgmtId: 181122461655966, via: 3, Ver: v1, Flags: 10, [{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"2a1b2549-6a f5-4e75-913c-7f6ac264f2ae","downloadPct":100,"errorString":"Downloaded 472312242 bytes (download complete remote=472312242bytes)","downloadStatus":"DOWNLOAD_IN_PROGRESS","downloadPath":"/mnt/SecStorage/1b8d363c-1a86-325d-b1b8-3c030ea93856 /template/tmpl/1/4/dnld1030975883579196189tmp_","installPath":"template/tmpl/1/4","templateSize":0,"templatePhySicalSize":0,"checkSum":"ed0e788280ff2912ea40f7f91ca7a249","result":true,"details":"Downloaded 472312242 bytes (download comple te remote=472312242bytes)","wait":0}}] } 2016-05-12 03:26:27,344 DEBUG [cloud.agent.Agent] (UgentTask-5:null) Sending ping: Seq 3-935: { Cmd , MgmtId: -1, via: 3, Ver: v1, Flags: 11, [{"com.cloud.agent.api.PingStorageCommand":{"changes":{},"hostType":"Storage","hostId":3,"wait" :0}}] } 2016-05-12 03:26:27,387 DEBUG [cloud.agent.Agent] (Agent-Handler-4:null) Received response: Seq 3-935: { Ans: , MgmtId: 181122461655966, via: 3, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.PingAnswer":{"_command":{"hostType":"Storage" ,"hostId":3,"wait":0},"result":true,"wait":0}}] } 2016-05-12 03:26:37,879 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:null) Request:Seq 3-7000845620747442562: { Cmd , MgmtId: 181122461655966, via: 3, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DownloadProgressCommand":{"jobId":"49e4c4dd-7e12-455f-9293-f7c50659992d","request":"GET_STATUS","hvm":true,"description":"Ubuntu 14.04 LTS (64-bit) ISO","maxDownloadSizeInBytes":53687091200,"id":203,"resourceType":"TEMPLATE","installPath":"template/tmpl/2/203","_store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://X.X.0.242/export/cloudstack","_role":"Image"}},"url":" http://xxx.xxxxxxx.com/iso/Ubuntu/Ubuntu%2014.04%20LTS/ubuntu-14.04-server-amd64.iso","format":"ISO","accountId":2,"name":"203-2-72045f68-861c-31b8-adf6-19df376a7411","secUrl":"nfs://X.X.0.242/export/cloudstack","wait":0}}] } 2016-05-12 03:26:37,880 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:null) Processing command: org.apache.cloudstack.storage.command.DownloadProgressCommand 2016-05-12 03:26:37,880 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:null) Seq 3-7000845620747442562: { Ans: , MgmtId: 181122461655966, via: 3, Ver: v1, Flags: 10, [{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"49e4c4dd-7e12-455f-9293-f7c50659992d","downloadPct":100,"errorString":"Downloaded 591396864 bytes (download complete remote=591396864bytes)","downloadStatus":"DOWNLOAD_IN_PROGRESS","downloadPath":"/mnt/SecStorage/1b8d363c-1a86-325d-b1b8-3c030ea93856/template/tmpl/2/203/dnld7251963221322463805tmp_","installPath":"template/tmpl/2/203","templateSize":0,"templatePhySicalSize":0,"result":true,"details":"Downloaded 591396864 bytes (download complete remote=591396864bytes)","wait":0}}] } 2016-05-12 03:26:37,882 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) Request:Seq 3-7000845620747442563: { Cmd , MgmtId: 181122461655966, via: 3, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DownloadProgressCommand":{"jobId":"2a1b2549-6af5-4e75-913c-7f6ac264f2ae","request":"GET_STATUS","hvm":false,"description":"CentOS 5.5(64-bit) no GUI (KVM)","checksum":"ed0e788280ff2912ea40f7f91ca7a249","maxDownloadSizeInBytes":53687091200,"id":4,"resourceType":"TEMPLATE","installPath":"template/tmpl/1/4","_store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://X.X.0.242/export/cloudstack","_role":"Image"}},"url":" http://download.cloud.com/releases/2.2.0/eec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2","format":"QCOW2","accountId":1,"name":"centos55-x86_64","secUrl":"nfs://X.X.0.242/export/cloudstack","wait":0}}] } 2016-05-12 03:26:37,882 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) Processing command: org.apache.cloudstack.storage.command.DownloadProgressCommand 2016-05-12 03:26:37,883 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) Seq 3-7000845620747442563: { Ans: , MgmtId: 181122461655966, via: 3, Ver: v1, Flags: 10, [{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"2a1b2549-6a f5-4e75-913c-7f6ac264f2ae","downloadPct":100,"errorString":"Downloaded 472312242 bytes (download complete remote=472312242bytes)","downloadStatus":"DOWNLOAD_IN_PROGRESS","downloadPath":"/mnt/SecStorage/1b8d363c-1a86-325d-b1b8-3c030ea93856/template/tmpl/1/4/dnld1030975883579196189tmp_","installPath":"template/tmpl/1/4","templateSize":0,"templatePhySicalSize":0,"checkSum":"ed0e788280ff2912ea40f7f91ca7a249","result":true,"details":"Downloaded 472312242 bytes (download complete remote=472312242bytes)","wait":0}}] } 2016-05-12 03:26:50,268 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) Processing command: com.cloud.agent.api.GetStorageStatsCommand 2016-05-12 03:26:50,269 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:null) Determined host X.X.0.242 corresponds to IP X.X.0.242 2016-05-12 03:26:50,270 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:null) Mounting device with nfs-style path of X.X.0.242:/export/cloudstack 2016-05-12 03:26:50,270 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:null) mount nfs://X.X.0.242/export/cloudstack on /mnt/SecStorage/1b8d363c-1a86-325d-b1b8-3c030ea93856 2016-05-12 03:26:50,270 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:null) making available /mnt/SecStorage/1b8d363c-1a86-325d-b1b8-3c030ea93856 on nfs://X.X.0.242/export/cloudstack 2016-05-12 03:26:50,270 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:null) local folder for mount will be /mnt/SecStorage/1b8d363c-1a86-325d-b1b8-3c030ea93856 2016-05-12 03:26:50,272 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:null) Executing: mount 2016-05-12 03:26:50,277 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:null) Execution is successful. 2016-05-12 03:26:50,277 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-5:null) Some device already mounted at /mnt/SecStorage/1b8d363c-1a86-325d-b1b8-3c030ea93856, no need to mount nfs://X.X.0.242/export/cloudstack ==== SSVM health check shows everything is OK: ==== root@s-3-VM:/var/log# /usr/local/cloud/systemvm/ssvm-check.sh ================================================ First DNS server is 8.8.8.8 PING 8.8.8.8 (8.8.8.8): 48 data bytes 56 bytes from 8.8.8.8: icmp_seq=0 ttl=50 time=4.599 ms 56 bytes from 8.8.8.8: icmp_seq=1 ttl=50 time=3.762 ms --- 8.8.8.8 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max/stddev = 3.762/4.181/4.599/0.419 ms Good: Can ping DNS server ================================================ Good: DNS resolves download.cloud.com ================================================ nfs is currently mounted Mount point is /mnt/SecStorage/1b8d363c-1a86-325d-b1b8-3c030ea93856 Good: Can write to mount point ================================================ Management server is X.X.0.222. Checking connectivity. Good: Can connect to management server port 8250 ================================================ Good: Java process is running ================================================ Tests Complete. Look for ERROR or WARNING above. root@s-3-VM:/var/log# ==== Any hints on what could be the reason why the template and ISO are not able to be installed / registered successfully? Looking forward to your reply, thank you. Cheers. On Wed, May 11, 2016 at 7:37 PM, Indra Pramana <in...@sg.or.id> wrote: > Dear all, > > I setup a test environment using CloudStack 4.8.0 (latest from > repository), one management server and two KVM agent hosts running Ubuntu > 14.04 LTS. Setup a zone with basic networking because the test environment > doesn't support VLAN and the test management server only have 1 NIC. The > private and public network are using the same subnet. Using Ceph RBD as > primary storage and NFS as secondary storage. > > Zone is created successfully, the two hosts are added successfully and > connected to the management servers. SSVM and CPVM are created without any > issues. However, the default > "CentOS 5.5(64-bit) no GUI (KVM)" template is stuck at "Installing > Template" status and Ready = "No" although it seems to be downloaded > successfully. I also tried registering an Ubuntu ISO and it faces the same > problem, it's downloaded successfully but then will stuck at "Installing > ISO" status. > > I ran the SSVM health check and everything is OK. Also go through the > troubleshooting steps as per below documentation but still unable to > determine the root cause: > > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/SSVM,+templates,+Secondary+storage+troubleshooting > > I tried restarting the SSVM service, stop and start the SSVM and > restarting the management-server service, it will cause the download of the > ISO and template to be restarted, but it will then stuck at installing > again. > > Logs from management server shows below lines which keeps on appearing / > looping every several seconds: > > === > 2016-05-11 19:31:29,031 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] > (Timer-6:ctx-369bfc57) (logid:0b1754a2) getCommandHostDelegation: class > org.apache.cloudstack.storage.command.DownloadProgressCommand > 2016-05-11 19:31:29,031 DEBUG [c.c.h.XenServerGuru] (Timer-6:ctx-369bfc57) > (logid:0b1754a2) getCommandHostDelegation: class > org.apache.cloudstack.storage.command.DownloadProgressCommand > 2016-05-11 19:31:29,031 DEBUG [o.a.c.s.RemoteHostEndPoint] > (Timer-6:ctx-369bfc57) (logid:0b1754a2) Sending command > org.apache.cloudstack.storage.command.DownloadProgressCommand to host: 3 > 2016-05-11 19:31:29,034 DEBUG [c.c.a.t.Request] (Timer-6:ctx-369bfc57) > (logid:0b1754a2) Seq 3-3028107799453237894: Sending { Cmd , MgmtId: > 181122461655966, via: 3(s-3-VM), Ver: v1, Flags: 100011, > [{"org.apache.cloudstack.storage.command > .DownloadProgressCommand":{"jobId":"cab4ada2-e52f-4c25-9c30-9ec23dc3fcd4","request":"GET_STATUS","hvm":false,"description":"CentOS > 5.5(64-bit) no GUI > (KVM)","checksum":"ed0e788280ff2912ea40f7f91ca7a249","maxDownloadSizeInBytes":536870912 > > 00,"id":4,"resourceType":"TEMPLATE","installPath":"template/tmpl/1/4","_store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://x.x.0.242/export/cloudstack","_role":"Image"}},"url":" > http://download.cloud.com/releases/2.2.0/eec2209b-987 > 5-3c8d-92be-c001bd8a0faf.qcow2.bz2","format":"QCOW2","accountId":1,"name":"centos55-x86_64","secUrl":"nfs://x.x.0.242/export/cloudstack","wait":0}}] > } > 2016-05-11 19:31:29,081 DEBUG [c.c.a.t.Request] > (AgentManager-Handler-8:null) (logid:) Seq 3-3028107799453237894: > Processing: { Ans: , MgmtId: 181122461655966, via: 3, Ver: v1, Flags: 10, > [{"com.cloud.agent.api.storage.DownloadAnswer":{ > "jobId":"cab4ada2-e52f-4c25-9c30-9ec23dc3fcd4","downloadPct":100,"errorString":"Downloaded > 472312242 bytes (download complete > remote=472312242bytes)","downloadStatus":"DOWNLOAD_IN_PROGRESS","downloadPath":"/mnt/SecStorage/1b8d363c-1a86-3 > 25d-b1b8-3c030ea93856/template/tmpl/1/4/dnld3801843575178925235tmp_","installPath":"template/tmpl/1/4","templateSize":0,"templatePhySicalSize":0,"checkSum":"ed0e788280ff2912ea40f7f91ca7a249","result":true,"details":"Downloaded > 472312242 > bytes (download complete remote=472312242bytes)","wait":0}}] } > 2016-05-11 19:31:33,039 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager > Timer:ctx-b98dc395) (logid:46e91a46) Resetting hosts suitable for reconnect > 2016-05-11 19:31:33,041 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager > Timer:ctx-b98dc395) (logid:46e91a46) Completed resetting hosts suitable for > reconnect > 2016-05-11 19:31:33,041 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager > Timer:ctx-b98dc395) (logid:46e91a46) Acquiring hosts for clusters already > owned by this management server > 2016-05-11 19:31:33,042 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager > Timer:ctx-b98dc395) (logid:46e91a46) Completed acquiring hosts for clusters > already owned by this management server > 2016-05-11 19:31:33,042 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager > Timer:ctx-b98dc395) (logid:46e91a46) Acquiring hosts for clusters not owned > by any management server > 2016-05-11 19:31:33,043 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager > Timer:ctx-b98dc395) (logid:46e91a46) Completed acquiring hosts for clusters > not owned by any management server > 2016-05-11 19:31:33,520 DEBUG [c.c.a.m.AgentManagerImpl] > (AgentManager-Handler-10:null) (logid:) SeqA 4-3326: Processing Seq > 4-3326: { Cmd , MgmtId: -1, via: 4, Ver: v1, Flags: 11, > [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{ > "_proxyVmId":4,"_loadInfo":"{\n \"connections\": []\n}","wait":0}}] } > 2016-05-11 19:31:33,608 DEBUG [c.c.a.m.AgentManagerImpl] > (AgentManager-Handler-10:null) (logid:) SeqA 4-3326: Sending Seq 4-3326: { > Ans: , MgmtId: 181122461655966, via: 4, Ver: v1, Flags: 100010, > [{"com.cloud.agent.api.AgentControlAnswe > r":{"result":true,"wait":0}}] } > 2016-05-11 19:31:37,910 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] > (AsyncJobMgr-Heartbeat-1:ctx-e8a869c3) (logid:8c9bcefd) Begin cleanup > expired async-jobs > 2016-05-11 19:31:37,916 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] > (AsyncJobMgr-Heartbeat-1:ctx-e8a869c3) (logid:8c9bcefd) End cleanup expired > async-jobs > 2016-05-11 19:31:37,966 DEBUG [o.a.c.s.i.BaseImageStoreDriverImpl] > (RemoteHostEndPoint-4:ctx-02c29196) (logid:cc402ab0) Performing image store > createTemplate async callback > 2016-05-11 19:31:38,522 DEBUG [c.c.a.m.AgentManagerImpl] > (AgentManager-Handler-11:null) (logid:) SeqA 4-3327: Processing Seq > 4-3327: { Cmd , MgmtId: -1, via: 4, Ver: v1, Flags: 11, > [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{ > "_proxyVmId":4,"_loadInfo":"{\n \"connections\": []\n}","wait":0}}] } > 2016-05-11 19:31:38,604 DEBUG [c.c.a.m.AgentManagerImpl] > (AgentManager-Handler-11:null) (logid:) SeqA 4-3327: Sending Seq 4-3327: { > Ans: , MgmtId: 181122461655966, via: 4, Ver: v1, Flags: 100010, > [{"com.cloud.agent.api.AgentControlAnswe > r":{"result":true,"wait":0}}] } > 2016-05-11 19:31:39,081 DEBUG [o.a.c.s.i.BaseImageStoreDriverImpl] > (RemoteHostEndPoint-4:ctx-7ac61961) (logid:f3094baf) Performing image store > createTemplate async callback > 2016-05-11 19:31:39,219 DEBUG [c.c.s.d.DownloadListener] > (Timer-5:ctx-a9ee8fd5) (logid:b445ac5e) Scheduling timeout at 30000 ms, > TEMPLATE: 203 at host 3 > 2016-05-11 19:31:39,472 DEBUG [c.c.s.d.DownloadListener] > (Timer-6:ctx-e44234cf) (logid:bb7ce357) Scheduling timeout at 30000 ms, > TEMPLATE: 4 at host 3 > 2016-05-11 19:31:41,786 DEBUG [o.s.b.f.a.InjectionMetadata] > (secstorage-1:ctx-1a3a9719) (logid:6bcbe476) Processing injected method of > bean 'org.apache.cloudstack.storage.image.store.ImageStoreImpl': > AutowiredFieldElement for com.cloud.s > torage.dao.VMTemplateDao > org.apache.cloudstack.storage.image.store.ImageStoreImpl.imageDao > 2016-05-11 19:31:41,786 DEBUG [o.s.b.f.s.DefaultListableBeanFactory] > (secstorage-1:ctx-1a3a9719) (logid:6bcbe476) Returning cached instance of > singleton bean 'VMTemplateDaoImpl' > 2016-05-11 19:31:41,786 DEBUG [o.s.b.f.a.InjectionMetadata] > (secstorage-1:ctx-1a3a9719) (logid:6bcbe476) Processing injected method of > bean 'org.apache.cloudstack.storage.image.store.ImageStoreImpl': > AutowiredFieldElement for private org > .apache.cloudstack.storage.datastore.ObjectInDataStoreManager > org.apache.cloudstack.storage.image.store.ImageStoreImpl.objectInStoreMgr > 2016-05-11 19:31:41,786 DEBUG [o.s.b.f.s.DefaultListableBeanFactory] > (secstorage-1:ctx-1a3a9719) (logid:6bcbe476) Returning cached instance of > singleton bean 'objectInDataStoreManagerImpl' > 2016-05-11 19:31:41,786 DEBUG [o.s.b.f.a.InjectionMetadata] > (secstorage-1:ctx-1a3a9719) (logid:6bcbe476) Processing injected method of > bean 'org.apache.cloudstack.storage.image.store.ImageStoreImpl': > AutowiredFieldElement for private com > .cloud.capacity.dao.CapacityDao > org.apache.cloudstack.storage.image.store.ImageStoreImpl.capacityDao > 2016-05-11 19:31:41,787 DEBUG [o.s.b.f.s.DefaultListableBeanFactory] > (secstorage-1:ctx-1a3a9719) (logid:6bcbe476) Returning cached instance of > singleton bean 'capacityDaoImpl' > 2016-05-11 19:31:41,789 DEBUG [o.s.b.f.a.InjectionMetadata] > (secstorage-1:ctx-1a3a9719) (logid:6bcbe476) Processing injected method of > bean 'org.apache.cloudstack.storage.image.store.ImageStoreImpl': > AutowiredFieldElement for com.cloud.storage.dao.VMTemplateDao > org.apache.cloudstack.storage.image.store.ImageStoreImpl.imageDao > 2016-05-11 19:31:41,789 DEBUG [o.s.b.f.s.DefaultListableBeanFactory] > (secstorage-1:ctx-1a3a9719) (logid:6bcbe476) Returning cached instance of > singleton bean 'VMTemplateDaoImpl' > 2016-05-11 19:31:41,789 DEBUG [o.s.b.f.a.InjectionMetadata] > (secstorage-1:ctx-1a3a9719) (logid:6bcbe476) Processing injected method of > bean 'org.apache.cloudstack.storage.image.store.ImageStoreImpl': > AutowiredFieldElement for private > org.apache.cloudstack.storage.datastore.ObjectInDataStoreManager > org.apache.cloudstack.storage.image.store.ImageStoreImpl.objectInStoreMgr > 2016-05-11 19:31:41,789 DEBUG [o.s.b.f.s.DefaultListableBeanFactory] > (secstorage-1:ctx-1a3a9719) (logid:6bcbe476) Returning cached instance of > singleton bean 'objectInDataStoreManagerImpl' > 2016-05-11 19:31:41,789 DEBUG [o.s.b.f.a.InjectionMetadata] > (secstorage-1:ctx-1a3a9719) (logid:6bcbe476) Processing injected method of > bean 'org.apache.cloudstack.storage.image.store.ImageStoreImpl': > AutowiredFieldElement for private com.cloud.capacity.dao.CapacityDao > org.apache.cloudstack.storage.image.store.ImageStoreImpl.capacityDao > 2016-05-11 19:31:41,790 DEBUG [o.s.b.f.s.DefaultListableBeanFactory] > (secstorage-1:ctx-1a3a9719) (logid:6bcbe476) Returning cached instance of > singleton bean 'capacityDaoImpl' > 2016-05-11 19:31:41,792 DEBUG [o.s.b.f.a.InjectionMetadata] > (secstorage-1:ctx-1a3a9719) (logid:6bcbe476) Processing injected method of > bean 'org.apache.cloudstack.storage.image.store.ImageStoreImpl': > AutowiredFieldElement for com.cloud.storage.dao.VMTemplateDao > org.apache.cloudstack.storage.image.store.ImageStoreImpl.imageDao > 2016-05-11 19:31:41,792 DEBUG [o.s.b.f.s.DefaultListableBeanFactory] > (secstorage-1:ctx-1a3a9719) (logid:6bcbe476) Returning cached instance of > singleton bean 'VMTemplateDaoImpl' > 2016-05-11 19:31:41,792 DEBUG [o.s.b.f.a.InjectionMetadata] > (secstorage-1:ctx-1a3a9719) (logid:6bcbe476) Processing injected method of > bean 'org.apache.cloudstack.storage.image.store.ImageStoreImpl': > AutowiredFieldElement for private > org.apache.cloudstack.storage.datastore.ObjectInDataStoreManager > org.apache.cloudstack.storage.image.store.ImageStoreImpl.objectInStoreMgr > 2016-05-11 19:31:41,793 DEBUG [o.s.b.f.s.DefaultListableBeanFactory] > (secstorage-1:ctx-1a3a9719) (logid:6bcbe476) Returning cached instance of > singleton bean 'objectInDataStoreManagerImpl' > 2016-05-11 19:31:41,793 DEBUG [o.s.b.f.a.InjectionMetadata] > (secstorage-1:ctx-1a3a9719) (logid:6bcbe476) Processing injected method of > bean 'org.apache.cloudstack.storage.image.store.ImageStoreImpl': > AutowiredFieldElement for private com.cloud.capacity.dao.CapacityDao > org.apache.cloudstack.storage.image.store.ImageStoreImpl.capacityDao > 2016-05-11 19:31:41,793 DEBUG [o.s.b.f.s.DefaultListableBeanFactory] > (secstorage-1:ctx-1a3a9719) (logid:6bcbe476) Returning cached instance of > singleton bean 'capacityDaoImpl' > 2016-05-11 19:31:41,794 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] > (secstorage-1:ctx-1a3a9719) (logid:6bcbe476) Zone 1 is ready to launch > secondary storage VM > 2016-05-11 19:31:42,212 DEBUG [o.s.b.f.a.InjectionMetadata] > (consoleproxy-1:ctx-0006665c) (logid:35d3c02d) Processing injected method > of bean 'org.apache.cloudstack.storage.image.store.ImageStoreImpl': > AutowiredFieldElement for com.cloud.storage.dao.VMTemplateDao > org.apache.cloudstack.storage.image.store.ImageStoreImpl.imageDao > 2016-05-11 19:31:42,212 DEBUG [o.s.b.f.s.DefaultListableBeanFactory] > (consoleproxy-1:ctx-0006665c) (logid:35d3c02d) Returning cached instance of > singleton bean 'VMTemplateDaoImpl' > 2016-05-11 19:31:42,212 DEBUG [o.s.b.f.a.InjectionMetadata] > (consoleproxy-1:ctx-0006665c) (logid:35d3c02d) Processing injected method > of bean 'org.apache.cloudstack.storage.image.store.ImageStoreImpl': > AutowiredFieldElement for private > org.apache.cloudstack.storage.datastore.ObjectInDataStoreManager > org.apache.cloudstack.storage.image.store.ImageStoreImpl.objectInStoreMgr > 2016-05-11 19:31:42,212 DEBUG [o.s.b.f.s.DefaultListableBeanFactory] > (consoleproxy-1:ctx-0006665c) (logid:35d3c02d) Returning cached instance of > singleton bean 'objectInDataStoreManagerImpl' > 2016-05-11 19:31:42,212 DEBUG [o.s.b.f.a.InjectionMetadata] > (consoleproxy-1:ctx-0006665c) (logid:35d3c02d) Processing injected method > of bean 'org.apache.cloudstack.storage.image.store.ImageStoreImpl': > AutowiredFieldElement for private com.cloud.capacity.dao.CapacityDao > org.apache.cloudstack.storage.image.store.ImageStoreImpl.capacityDao > 2016-05-11 19:31:42,213 DEBUG [o.s.b.f.s.DefaultListableBeanFactory] > (consoleproxy-1:ctx-0006665c) (logid:35d3c02d) Returning cached instance of > singleton bean 'capacityDaoImpl' > 2016-05-11 19:31:42,216 DEBUG [c.c.c.ConsoleProxyManagerImpl] > (consoleproxy-1:ctx-0006665c) (logid:35d3c02d) Zone 1 is ready to launch > console proxy > 2016-05-11 19:31:43,523 DEBUG [c.c.a.m.AgentManagerImpl] > (AgentManager-Handler-15:null) (logid:) SeqA 4-3328: Processing Seq > 4-3328: { Cmd , MgmtId: -1, via: 4, Ver: v1, Flags: 11, > [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":4,"_loadInfo":"{\n > \"connections\": []\n}","wait":0}}] } > 2016-05-11 19:31:43,606 DEBUG [c.c.a.m.AgentManagerImpl] > (AgentManager-Handler-15:null) (logid:) SeqA 4-3328: Sending Seq 4-3328: { > Ans: , MgmtId: 181122461655966, via: 4, Ver: v1, Flags: 100010, > [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] } > 2016-05-11 19:31:47,908 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] > (AsyncJobMgr-Heartbeat-1:ctx-7a3cd65f) (logid:2832c3eb) Begin cleanup > expired async-jobs > 2016-05-11 19:31:47,917 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] > (AsyncJobMgr-Heartbeat-1:ctx-7a3cd65f) (logid:2832c3eb) End cleanup expired > async-jobs > ==== > > Anyone can advise what could be the problem? > > Looking forward to your reply, thank you. > > Cheers. > > -ip- >