Sanjeev N created CLOUDSTACK-4400:
-------------------------------------
Summary: [object_store_refactor] Three entries for one template in
template_store_ref when MS was restarting during template download
Key: CLOUDSTACK-4400
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4400
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Storage Controller, Template
Affects Versions: 4.2.0
Environment: Latest build from ACS 4.2 branch
Reporter: Sanjeev N
Priority: Critical
Fix For: 4.2.0
Three entries for one template in template_store_ref when MS was restarting
during template download.
Restarting management server when the template download was in progress,
created three entries in template_store_ref table for the same template and in
UI tamplate Ready is set to No even though it is downloaded to S3
Steps to Reproduce:
================
1.Bring up CS with two or three zones
2.Register template any one of the zones
3.Restart the management server when the template download was in progres
Result:
=====
After restart template sync initiated template download via all the SSVMs and
created multiple intries in the template_store_ref for the same template.
In UI template is not showing as ready even though it is download to S3 via one
of the ssvms.
Observations:
=============
After management server restart, it found template 208 is not downlaoded
2013-08-19 08:54:01,695 INFO [storage.image.TemplateServiceImpl]
(AgentConnectTaskPool-3:null) Template Sync did not find
208-2-0fe7ea1d-5e21-33cd-b328-e4510821608d on image store 3, may request
download based on available hypervisor types
2013-08-19 08:54:01,709 INFO [storage.image.TemplateServiceImpl]
(AgentConnectTaskPool-3:null) Removing leftover template
208-2-0fe7ea1d-5e21-33cd-b328-e4510821608d entry from template store table
2013-08-19 08:54:01,722 INFO [storage.image.TemplateServiceImpl]
(AgentConnectTaskPool-6:null) Template Sync did not find
208-2-0fe7ea1d-5e21-33cd-b328-e4510821608d on image store 3, may request
download based on available hypervisor types
2013-08-19 08:54:01,722 INFO [storage.image.TemplateServiceImpl]
(AgentConnectTaskPool-6:null) Removing leftover template
208-2-0fe7ea1d-5e21-33cd-b328-e4510821608d entry from template store table
2013-08-19 08:54:01,735 INFO [storage.image.TemplateServiceImpl]
(AgentConnectTaskPool-4:null) Template Sync did not find
208-2-0fe7ea1d-5e21-33cd-b328-e4510821608d on image store 3, may request
download based on available hypervisor types
2013-08-19 08:54:01,735 INFO [storage.image.TemplateServiceImpl]
(AgentConnectTaskPool-4:null) Removing leftover template
208-2-0fe7ea1d-5e21-33cd-b328-e4510821608d entry from template store table
2013-08-19 08:54:01,747 INFO [storage.image.TemplateServiceImpl]
(AgentConnectTaskPool-6:null) Downloading template
208-2-0fe7ea1d-5e21-33cd-b328-e4510821608d to image store imagestore
2013-08-19 08:54:01,751 INFO [storage.image.TemplateServiceImpl]
(AgentConnectTaskPool-4:null) Downloading template
208-2-0fe7ea1d-5e21-33cd-b328-e4510821608d to image store imagestore
2013-08-19 08:54:01,753 INFO [storage.image.TemplateServiceImpl]
(AgentConnectTaskPool-3:null) Downloading template
208-2-0fe7ea1d-5e21-33cd-b328-e4510821608d to image store imagestore
2013-08-19 08:54:02,068 DEBUG [storage.image.TemplateDataFactoryImpl]
(AgentConnectTaskPool-4:null) template 208 is already in store:3, type:Image
2013-08-19 08:54:02,080 DEBUG [storage.image.TemplateDataFactoryImpl]
(AgentConnectTaskPool-3:null) template 208 is already in store:3, type:Image
2013-08-19 08:54:02,069 DEBUG [storage.image.TemplateDataFactoryImpl]
(AgentConnectTaskPool-6:null) template 208 is already in store:3, type:Image
2013-08-19 08:54:02,163 DEBUG [storage.image.BaseImageStoreDriverImpl]
(AgentConnectTaskPool-6:null) Downloading template to data store 3
2013-08-19 08:54:02,164 DEBUG [storage.image.BaseImageStoreDriverImpl]
(AgentConnectTaskPool-4:null) Downloading template to data store 3
2013-08-19 08:54:02,219 DEBUG [agent.transport.Request]
(AgentConnectTaskPool-3:null) Seq 3-1325465607: Sending { Cmd , MgmtId:
6615759585382, via: 3, Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.ReadyCommand":{"dcId":1,"hostId":3,"wait":0}}] }
2013-08-19 08:54:02,233 DEBUG [agent.transport.Request]
(AgentManager-Handler-3:null) Seq 3-1325465607: Processing: { Ans: , MgmtId:
6615759585382, via: 3, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.ReadyAnswer":{"result":true,"wait":0}}] }
2013-08-19 08:54:02,233 DEBUG [agent.manager.AgentAttache]
(AgentManager-Handler-3:null) Seq 3-1325465607: No more commands found
2013-08-19 08:54:02,233 DEBUG [agent.transport.Request]
(AgentConnectTaskPool-3:null) Seq 3-1325465607: Received: { Ans: , MgmtId:
6615759585382, via: 3, Ver: v1, Flags: 110, { ReadyAnswer } }
2013-08-19 08:54:02,233 DEBUG [cloud.host.Status] (AgentConnectTaskPool-3:null)
Transition:[Resource state = Enabled, Agent event = Ready, Host id = 3, name =
s-1-VM]
2013-08-19 08:54:02,277 DEBUG [cloud.host.Status] (AgentConnectTaskPool-3:null)
Agent status update: [id = 3; name = s-1-VM; old status = Connecting; event =
Ready; new status = Up; old update count = 11; new update count = 12]
2013-08-19 08:54:02,347 DEBUG [agent.transport.Request]
(AgentConnectTaskPool-3:null) Seq 3-1325465608: Sending { Cmd , MgmtId:
6615759585382, via: 3, Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.ReadyCommand":{"dcId":1,"hostId":3,"wait":0}}] }
2013-08-19 08:54:02,374 DEBUG [storage.image.BaseImageStoreDriverImpl]
(AgentConnectTaskPool-6:null) Performing image store createTemplate async
callback
2013-08-19 08:54:02,380 DEBUG [storage.image.BaseImageStoreDriverImpl]
(AgentConnectTaskPool-4:null) Performing image store createTemplate async
callback
2013-08-19 08:54:02,390 DEBUG [agent.transport.Request]
(AgentManager-Handler-5:null) Seq 3-1325465608: Processing: { Ans: , MgmtId:
6615759585382, via: 3, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.ReadyAnswer":{"result":true,"wait":0}}] }
2013-08-19 08:54:02,391 DEBUG [agent.manager.AgentAttache]
(AgentManager-Handler-5:null) Seq 3-1325465608: No more commands found
2013-08-19 08:54:02,391 DEBUG [agent.transport.Request]
(AgentConnectTaskPool-3:null) Seq 3-1325465608: Received: { Ans: , MgmtId:
6615759585382, via: 3, Ver: v1, Flags: 110, { ReadyAnswer } }
2013-08-19 08:54:02,423 DEBUG [cloudstack.storage.RemoteHostEndPoint]
(AgentConnectTaskPool-6:null) Sending command
org.apache.cloudstack.storage.command.DownloadCommand to host: 9
2013-08-19 08:54:02,432 DEBUG [cloudstack.storage.RemoteHostEndPoint]
(AgentConnectTaskPool-4:null) Sending command
org.apache.cloudstack.storage.command.DownloadCommand to host: 3
2013-08-19 08:54:02,465 DEBUG [agent.transport.Request]
(AgentConnectTaskPool-6:null) Seq 9-275972102: Sending { Cmd , MgmtId:
6615759585382, via: 9, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DownloadCommand":{"hvm":true,"description":"win8","maxDownloadSizeInBytes":53687091200,"id":208,"resourceType":"TEMPLATE","installPath":"template/tmpl/2/208/208-2-0fe7ea1d-5e21-33cd-b328-e4510821608d","_store":{"com.cloud.agent.api.to.S3TO":{"id":3,"uuid":"3c9432fe-ff7c-42d0-b886-5006386950ae","endPoint":"10.147.29.56:8080","bucketName":"imagestore","httpsFlag":false,"created":"Aug
16, 2013 7:53:59
AM","enableRRS":false}},"url":"http://10.147.28.7/templates/Windows8/Windows864.vhd","format":"VHD","accountId":2,"name":"208-2-0fe7ea1d-5e21-33cd-b328-e4510821608d","wait":0}}]
}
2013-08-19 08:54:02,476 DEBUG [agent.transport.Request]
(AgentConnectTaskPool-4:null) Seq 3-1325465609: Sending { Cmd , MgmtId:
6615759585382, via: 3, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DownloadCommand":{"hvm":true,"description":"win8","maxDownloadSizeInBytes":53687091200,"id":208,"resourceType":"TEMPLATE","installPath":"template/tmpl/2/208/208-2-0fe7ea1d-5e21-33cd-b328-e4510821608d","_store":{"com.cloud.agent.api.to.S3TO":{"id":3,"uuid":"3c9432fe-ff7c-42d0-b886-5006386950ae","endPoint":"10.147.29.56:8080","bucketName":"imagestore","httpsFlag":false,"created":"Aug
16, 2013 7:53:59
AM","enableRRS":false}},"url":"http://10.147.28.7/templates/Windows8/Windows864.vhd","format":"VHD","accountId":2,"name":"208-2-0fe7ea1d-5e21-33cd-b328-e4510821608d","wait":0}}]
}
mysql> select * from template_store_ref where template_id=208;
+----+----------+-------------+---------------------+---------------------+--------------------------------------+--------------+------------+------------+---------------+----------------------+-----------+-------------------------------------------------------------------------------+-------------------------------------------------------------------------------+------+--------------+----------------------+-----------+-----------+---------+--------------+---------+---------------------+
| id | store_id | template_id | created | last_updated |
job_id | download_pct | size | store_role |
physical_size | download_state | error_str | local_path
| install_path
| url | download_url |
download_url_created | state | destroyed | is_copy | update_count | ref_cnt
| updated |
+----+----------+-------------+---------------------+---------------------+--------------------------------------+--------------+------------+------------+---------------+----------------------+-----------+-------------------------------------------------------------------------------+-------------------------------------------------------------------------------+------+--------------+----------------------+-----------+-----------+---------+--------------+---------+---------------------+
| 24 | 3 | 208 | 2013-08-19 12:54:02 | 2013-08-19 13:14:48 |
bd84f245-2fef-4f1c-8d6b-33de36b1843d | 100 | 9436754432 | Image |
9436754432 | DOWNLOADED | |
template/tmpl/2/208/208-2-0fe7ea1d-5e21-33cd-b328-e4510821608d/Windows864.vhd |
template/tmpl/2/208/208-2-0fe7ea1d-5e21-33cd-b328-e4510821608d/Windows864.vhd |
NULL | NULL | NULL | Creating | 0 | 0 |
1 | 0 | 2013-08-19 12:54:02 |
| 25 | 3 | 208 | 2013-08-19 12:54:02 | 2013-08-19 13:14:18 |
a43fb0d4-d973-4086-8c78-9b643cf492d3 | 98 | 0 | Image |
0 | DOWNLOAD_IN_PROGRESS | |
template/tmpl/2/208/208-2-0fe7ea1d-5e21-33cd-b328-e4510821608d/Windows864.vhd |
template/tmpl/2/208/208-2-0fe7ea1d-5e21-33cd-b328-e4510821608d |
NULL | NULL | NULL | Ready | 0 | 0 |
3 | 0 | 2013-08-19 13:14:48 |
| 26 | 3 | 208 | 2013-08-19 12:54:02 | 2013-08-19 13:14:38 |
bd84f245-2fef-4f1c-8d6b-33de36b1843d | 100 | 9436754432 | Image |
9436754432 | DOWNLOADED | |
template/tmpl/2/208/208-2-0fe7ea1d-5e21-33cd-b328-e4510821608d/Windows864.vhd |
template/tmpl/2/208/208-2-0fe7ea1d-5e21-33cd-b328-e4510821608d/Windows864.vhd |
NULL | NULL | NULL | Allocated | 0 | 0 |
0 | 0 | NULL |
+----+----------+-------------+---------------------+---------------------+--------------------------------------+--------------+------------+------------+---------------+----------------------+-----------+-------------------------------------------------------------------------------+-------------------------------------------------------------------------------+------+--------------+----------------------+-----------+-----------+---------+--------------+---------+---------------------+
3 rows in set (0.00 sec)
Attaching management server log file and cloud DB.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
