sounds painful. thanks for sharing this though. Hopefully someone can gain value from this, and incorporate this into their operations.
On Thu, Feb 7, 2013 at 1:19 PM, Bryan Whitehead <dri...@megahappy.net>wrote: > This is on cloudstack 3.0.2. I had already removed all the snapshots in > the cloudstack UI. So from the cloudstack point of view there was zero > snapshots when in fact there was many more. > > The libvirt guys in #virt IRC were able to help me clean up all the > internal snapshots on the qcow2 file via "virsh snapshot-delete [...] > [...]". Unfortunately I cannot recover the 200G of space the qcow2 file has > grown to without shutting down the VM since I'm running qemu/libvirt that > is too old (CentOS 6.3 + some patches). > > According to libvirt there is 0 snapshots left but directly running > "qemu-img info [qcow2 file]" shows there are still ~10 snapshots. (look at > my original email - you can see the difference between "qemu-img info" > output and "virsh snapshot-list" > > Anyhow, scheduled snapshots is working again just fine... including > auto-rotation of a limit of 3-backups. But I will need to more closely > monitor logs for systemvm storage problems... I can deal with a 220G qcow2 > file. Since many internal snapshots have been deleted there is a lot of > "free" space in the qcow2 file - so hopefully it won't grow again if this > happens again. > > > On Tue, Feb 5, 2013 at 1:33 PM, Ahmad Emneina <aemne...@gmail.com> wrote: > >> Bryan* ... sorry >> >> >> On Tue, Feb 5, 2013 at 1:33 PM, Ahmad Emneina <aemne...@gmail.com> wrote: >> >> > I'd also file a bug for this Brian. The process was failing, and not >> > recovering gracefully, by cleaning up after itself. >> > >> > >> > On Tue, Feb 5, 2013 at 1:18 PM, Ahmad Emneina <aemne...@gmail.com> >> wrote: >> > >> >> I think in 4.0 and master this was broken. Marcus or Edison could >> >> probably comment to that. Can you try deleting the snapshots from the >> >> cloudstack UI? If that doesnt work, we can work to devise a method >> where we >> >> query the db, identify valid snapshots, then using virsh commands >> delete >> >> the failed snapshots. >> >> >> >> >> >> On Fri, Feb 1, 2013 at 4:02 PM, Bryan Whitehead <dri...@megahappy.net >> >wrote: >> >> >> >>> For some reason my storageVM was in a state where things "worked" but >> >>> automated snapshots were not working. >> >>> >> >>> Basically the step where libvirt created a stapshot on the primary >> >>> storage >> >>> worked - but the step to have the snapshot parent copied to the >> secondary >> >>> storage was not working. I got automated snapshots working again by >> >>> rebooting the storage systemVM. >> >>> >> >>> However, when i look at libvirt I can see there are many snapshots on >> the >> >>> primary storage. A 12G filesystem has a qcow2 size of 220GB. >> >>> >> >>> virsh snapshot-list --parent i-3-14-VM >> >>> Name Creation Time State Parent >> >>> ------------------------------------------------------------ >> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130125025313 >> 2013-01-25 >> >>> 02:53:13 +0000 running >> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130126025313 >> 2013-01-26 >> >>> 02:53:13 +0000 running >> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130125025313 >> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130127025313 >> 2013-01-27 >> >>> 02:53:13 +0000 running >> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130126025313 >> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130128025313 >> 2013-01-28 >> >>> 02:53:13 +0000 running >> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130127025313 >> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130129025313 >> 2013-01-29 >> >>> 02:53:13 +0000 running >> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130128025313 >> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130130025313 >> 2013-01-30 >> >>> 02:53:13 +0000 running >> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130129025313 >> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130131025313 >> 2013-01-31 >> >>> 02:53:13 +0000 running >> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130130025313 >> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201025313 >> 2013-02-01 >> >>> 02:53:13 +0000 running >> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130131025313 >> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201223643 >> 2013-02-01 >> >>> 22:36:43 +0000 running >> >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201025313 >> >>> >> >>> The actual qcow2 file looks like a much bigger wreck: >> >>> >> >>> qemu-img info >> /gluster/qcow2/images/7ec192c1-ecee-4f5e-8e3d-567f7878ceb1 >> >>> image: /gluster/qcow2/images/7ec192c1-ecee-4f5e-8e3d-567f7878ceb1 >> >>> file format: qcow2 >> >>> virtual size: 100G (107374182400 bytes) >> >>> disk size: 211G >> >>> cluster_size: 65536 >> >>> backing file: >> /gluster/qcow2/images/cb151441-209c-4f43-a2a4-b390c9bb9768 >> >>> (actual path: >> /gluster/qcow2/images/cb151441-209c-4f43-a2a4-b390c9bb9768) >> >>> Snapshot list: >> >>> ID TAG VM SIZE DATE VM >> CLOCK >> >>> 1 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20121230025415 >> >>> 1.0G 2012-12-30 02:54:16 785:39:01.680 >> >>> 2 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20121231025415 >> >>> 1.0G 2012-12-31 02:54:16 809:35:35.417 >> >>> 3 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130101025415 >> >>> 1.0G 2013-01-01 02:54:16 833:24:20.046 >> >>> 4 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130102025415 >> >>> 1.0G 2013-01-02 02:54:15 857:13:02.766 >> >>> 5 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130103025415 >> >>> 1.0G 2013-01-03 02:54:15 881:01:42.467 >> >>> 6 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130104025415 >> >>> 1.0G 2013-01-04 02:54:15 904:50:29.840 >> >>> 7 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130105025415 >> >>> 1.0G 2013-01-05 02:54:16 928:39:17.670 >> >>> 8 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130106025415 >> >>> 1.0G 2013-01-06 02:54:16 952:28:07.139 >> >>> 9 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130107025415 >> >>> 1.1G 2013-01-07 02:54:16 976:16:58.396 >> >>> 10 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130108025415 >> >>> 1.1G 2013-01-08 02:54:15 1000:05:45.624 >> >>> 11 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130109025415 >> >>> 1.1G 2013-01-09 02:54:15 1023:54:25.482 >> >>> 12 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130110025415 >> >>> 1.1G 2013-01-10 02:54:16 1047:43:17.503 >> >>> 13 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130111025415 >> >>> 1.1G 2013-01-11 02:54:16 1071:32:03.747 >> >>> 14 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130112025415 >> >>> 1.1G 2013-01-12 02:54:16 1095:20:52.220 >> >>> 15 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130113025415 >> >>> 1.1G 2013-01-13 02:54:16 1119:09:43.387 >> >>> 16 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130114025415 >> >>> 1.1G 2013-01-14 02:54:16 1142:58:32.657 >> >>> 17 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130115025415 >> >>> 1.1G 2013-01-15 02:54:16 1166:47:20.084 >> >>> 18 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130116025416 >> >>> 1.1G 2013-01-16 02:54:16 1190:36:06.729 >> >>> 19 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130117025416 >> >>> 1.1G 2013-01-17 02:54:16 1214:24:53.904 >> >>> 20 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130118025415 >> >>> 1.1G 2013-01-18 02:54:16 1238:13:35.640 >> >>> 21 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130119025415 >> >>> 1.1G 2013-01-19 02:54:16 1262:02:17.839 >> >>> 22 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130120025415 >> >>> 1.1G 2013-01-20 02:54:16 1285:51:01.114 >> >>> 23 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130121025415 >> >>> 1.1G 2013-01-21 02:54:16 1309:39:49.956 >> >>> 24 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130122025416 >> >>> 1.1G 2013-01-22 02:54:16 1333:28:39.703 >> >>> 25 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130123025415 >> >>> 1.1G 2013-01-23 02:54:16 1357:17:25.426 >> >>> 26 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130124025416 >> >>> 1.1G 2013-01-24 02:54:16 1381:06:04.731 >> >>> 27 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130125025313 >> >>> 517M 2013-01-25 02:53:13 04:46:52.307 >> >>> 28 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130126025313 >> >>> 678M 2013-01-26 02:53:13 28:44:35.134 >> >>> 29 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130127025313 >> >>> 918M 2013-01-27 02:53:13 52:42:00.249 >> >>> 30 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130128025313 >> >>> 1.1G 2013-01-28 02:53:13 76:38:55.428 >> >>> 31 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130129025313 >> >>> 1.2G 2013-01-29 02:53:13 100:35:27.978 >> >>> 32 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130130025313 >> >>> 1.2G 2013-01-30 02:53:13 124:31:55.179 >> >>> 33 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130131025313 >> >>> 1.3G 2013-01-31 02:53:13 148:28:17.744 >> >>> 34 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201025313 >> >>> 1.3G 2013-02-01 02:53:13 172:24:33.630 >> >>> 35 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201223643 >> >>> 1.3G 2013-02-01 22:36:43 192:04:17.498 >> >>> >> >>> Suggestions on how I can clean up these extra snapshots? >> >>> >> >>> -Bryan >> >>> >> >> >> >> >> > >> > >
