Pearl1594 opened a new pull request #4551:
URL: https://github.com/apache/cloudstack/pull/4551
### Description
This PR aims at fixing the issue caused when:
- download a Volume (such that there is a record in the volume_store_ref
table)
- migrate the same volume from one primary store to the another
When migrating between primary storage pools ACS temporarily copies the
volume to the secondary store and then copies it from there to the destination
primary store and then the secondary store volume directory is cleared up and
the volume (volumes table) is marked as Expunged. BUT the corresponding
volume's secondary store record(created when we downloaded the volume) still
appears to be ready
This PR fixes this issue by clearing all records on the secondary store
corresponding to that volume and also enhances the garbage collection logic to
delete any volumes that were marked as Expunged but have entries in the
volume_store_ref table.
Observations made:
Entries in volume_store_ref after downloading the volume and initiating
migration:
```
*************************** 2. row ***************************
id: 113
store_id: 1
volume_id: 127
zone_id: 1
created: 2020-12-15 05:45:40
last_updated: NULL
job_id: NULL
download_pct: 100
size: 0
physical_size: 0
download_state: DOWNLOADED
checksum: NULL
error_str: NULL
local_path: NULL
install_path:
volumes/2/127/a73cb20a-9ab3-4c4d-9605-31a3850e1fdc.qcow2
url: NULL
download_url:
http://10.1.34.20/userdata/38406061-14aa-499a-92c7-6f25978759ae.qcow2
state: Ready
destroyed: 0
update_count: 2
ref_cnt: 0
updated: 2020-12-15 05:45:52
download_url_created: 2020-12-15 05:45:52
*************************** 3. row ***************************
id: 114
store_id: 1
volume_id: 127
zone_id: 0
created: 2020-12-15 05:46:10
last_updated: NULL
job_id: NULL
download_pct: 0
size: 0
physical_size: 0
download_state: NULL
checksum: NULL
error_str: NULL
local_path: NULL
install_path:
volumes/2/127/7b5ba7e3-bc06-4c31-8eda-74cda3fab182.qcow2
url: NULL
download_url: NULL
state: Copying
destroyed: 0
update_count: 3
ref_cnt: 0
updated: 2020-12-15 05:46:20
download_url_created: NULL
```
Once migration completes, the directory is cleared up from the secondary
store, the record corresponding to the migrating volume is also cleared up -
but the entry for the downloaded volume is still left behind:
```
*************************** 2. row ***************************
id: 113
store_id: 1
volume_id: 127
zone_id: 1
created: 2020-12-15 05:45:40
last_updated: NULL
job_id: NULL
download_pct: 100
size: 0
physical_size: 0
download_state: DOWNLOADED
checksum: NULL
error_str: NULL
local_path: NULL
install_path:
volumes/2/127/a73cb20a-9ab3-4c4d-9605-31a3850e1fdc.qcow2
url: NULL
download_url:
http://10.1.34.20/userdata/38406061-14aa-499a-92c7-6f25978759ae.qcow2
state: Ready
destroyed: 0
update_count: 2
ref_cnt: 0
updated: 2020-12-15 05:45:52
download_url_created: 2020-12-15 05:45:52
2 rows in set (0.00 sec)
MariaDB [cloud]> select
id,account_id,name,folder,path,volume_type,created,updated,removed,state from
volumes where id = 127\G
*************************** 1. row ***************************
id: 127
account_id: 2
name: ROOT-46
folder:
/acs/primary/ref-trl-1977-k-M7-pearl-dsilva/ref-trl-1977-k-M7-pearl-dsilva-kvm-pri1
path: 87ff45f2-e466-42f2-9f75-b772fc26a4f1
volume_type: ROOT
created: 2020-12-15 05:33:09
updated: 2020-12-15 05:46:30
removed: NULL
state: Expunged
1 row in set (0.00 sec)
```
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] New feature (non-breaking change which adds functionality)
- [X] Bug fix (non-breaking change which fixes an issue)
- [ ] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
### Feature/Enhancement Scale or Bug Severity
#### Bug Severity
- [ ] BLOCKER
- [ ] Critical
- [ ] Major
- [X] Minor
- [ ] Trivial
### How Has This Been Tested?
The same steps mentioned above were repeated with the fix in place and the
following are observed:
Firstly, the garbage collector identifies any left behind volume entries and
clears them
```
020-12-15 06:07:14,285 DEBUG [c.c.s.StorageManagerImpl]
(StorageManager-Scavenger-1:ctx-c21cc4c6) (logid:95f250bd) Secondary storage
garbage collector found 1 volumes to cleanup on volume_store_ref for store:
NFS://10.10.0.16/acs/secondary/ref-trl-1977-k-M7-pearl-dsilva/ref-trl-1977-k-M7-pearl-dsilva-sec1
2020-12-15 06:07:14,286 DEBUG [c.c.s.StorageManagerImpl]
(StorageManager-Scavenger-1:ctx-c21cc4c6) (logid:95f250bd) Deleting volume
store DB entry:
VolumeDataStore[113-127-1volumes/2/127/a73cb20a-9ab3-4c4d-9605-31a3850e1fdc.qcow2]
```
Downloaded a volume:
```
MariaDB [cloud]> select * from volume_store_ref\G
id: 134
store_id: 2
volume_id: 121
zone_id: 1
created: 2020-12-17 05:48:52
last_updated: NULL
job_id: NULL
download_pct: 100
size: 0
physical_size: 0
download_state: DOWNLOADED
checksum: NULL
error_str: NULL
local_path: NULL
install_path:
volumes/2/121/afaf7b46-59d4-4e1b-b592-d1ce16d60b14.qcow2
url: NULL
download_url:
http://10.1.34.15/userdata/8f111dea-e1f7-4351-9155-6ec43487019a.qcow2
state: Ready
destroyed: 0
update_count: 2
ref_cnt: 0
updated: 2020-12-17 05:49:04
download_url_created: 2020-12-17 05:49:04
2 rows in set (0.00 sec)
MariaDB [cloud]> select
id,account_id,name,folder,path,volume_type,created,updated,removed,state from
volumes where id = 121\G
*************************** 1. row ***************************
id: 121
account_id: 2
name: ROOT-51
folder:
/acs/primary/ref-trl-1977-k-M7-pearl-dsilva/ref-trl-1977-k-M7-pearl-dsilva-kvm-pri2
path: 2e30ed4b-9bdd-4e97-8f45-8051a1e28b16
volume_type: ROOT
created: 2020-12-14 11:54:15
updated: 2020-12-17 05:49:03
removed: NULL
state: Ready
1 row in set (0.00 sec)
```
Initiate migration - here we see another entry created for the same volume:
```
*************************** 2. row ***************************
id: 134
store_id: 2
volume_id: 121
zone_id: 1
created: 2020-12-17 05:48:52
last_updated: NULL
job_id: NULL
download_pct: 100
size: 0
physical_size: 0
download_state: DOWNLOADED
checksum: NULL
error_str: NULL
local_path: NULL
install_path:
volumes/2/121/afaf7b46-59d4-4e1b-b592-d1ce16d60b14.qcow2
url: NULL
download_url:
http://10.1.34.15/userdata/8f111dea-e1f7-4351-9155-6ec43487019a.qcow2
state: Ready
destroyed: 0
update_count: 2
ref_cnt: 0
updated: 2020-12-17 05:49:04
download_url_created: 2020-12-17 05:49:04
*************************** 3. row ***************************
id: 135
store_id: 2
volume_id: 121
zone_id: 0
created: 2020-12-17 05:50:10
last_updated: NULL
job_id: NULL
download_pct: 0
size: 0
physical_size: 0
download_state: NULL
checksum: NULL
error_str: NULL
local_path: NULL
install_path: volumes/2/121
url: NULL
download_url: NULL
state: Creating
destroyed: 0
update_count: 1
ref_cnt: 0
updated: 2020-12-17 05:50:10
download_url_created: NULL
```
Post migration - notice there is no entry left behind for the expunged
volume:
```
MariaDB [cloud]> select * from volume_store_ref where volume_id = 121 \G
Empty set (0.00 sec)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
