This is on cloudstack 3.0.2. I had already removed all the snapshots in the cloudstack UI. So from the cloudstack point of view there was zero snapshots when in fact there was many more.
The libvirt guys in #virt IRC were able to help me clean up all the internal snapshots on the qcow2 file via "virsh snapshot-delete [...] [...]". Unfortunately I cannot recover the 200G of space the qcow2 file has grown to without shutting down the VM since I'm running qemu/libvirt that is too old (CentOS 6.3 + some patches). According to libvirt there is 0 snapshots left but directly running "qemu-img info [qcow2 file]" shows there are still ~10 snapshots. (look at my original email - you can see the difference between "qemu-img info" output and "virsh snapshot-list" Anyhow, scheduled snapshots is working again just fine... including auto-rotation of a limit of 3-backups. But I will need to more closely monitor logs for systemvm storage problems... I can deal with a 220G qcow2 file. Since many internal snapshots have been deleted there is a lot of "free" space in the qcow2 file - so hopefully it won't grow again if this happens again. On Tue, Feb 5, 2013 at 1:33 PM, Ahmad Emneina <aemne...@gmail.com> wrote: > Bryan* ... sorry > > > On Tue, Feb 5, 2013 at 1:33 PM, Ahmad Emneina <aemne...@gmail.com> wrote: > > > I'd also file a bug for this Brian. The process was failing, and not > > recovering gracefully, by cleaning up after itself. > > > > > > On Tue, Feb 5, 2013 at 1:18 PM, Ahmad Emneina <aemne...@gmail.com> > wrote: > > > >> I think in 4.0 and master this was broken. Marcus or Edison could > >> probably comment to that. Can you try deleting the snapshots from the > >> cloudstack UI? If that doesnt work, we can work to devise a method > where we > >> query the db, identify valid snapshots, then using virsh commands delete > >> the failed snapshots. > >> > >> > >> On Fri, Feb 1, 2013 at 4:02 PM, Bryan Whitehead <dri...@megahappy.net > >wrote: > >> > >>> For some reason my storageVM was in a state where things "worked" but > >>> automated snapshots were not working. > >>> > >>> Basically the step where libvirt created a stapshot on the primary > >>> storage > >>> worked - but the step to have the snapshot parent copied to the > secondary > >>> storage was not working. I got automated snapshots working again by > >>> rebooting the storage systemVM. > >>> > >>> However, when i look at libvirt I can see there are many snapshots on > the > >>> primary storage. A 12G filesystem has a qcow2 size of 220GB. > >>> > >>> virsh snapshot-list --parent i-3-14-VM > >>> Name Creation Time State Parent > >>> ------------------------------------------------------------ > >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130125025313 2013-01-25 > >>> 02:53:13 +0000 running > >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130126025313 2013-01-26 > >>> 02:53:13 +0000 running > >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130125025313 > >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130127025313 2013-01-27 > >>> 02:53:13 +0000 running > >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130126025313 > >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130128025313 2013-01-28 > >>> 02:53:13 +0000 running > >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130127025313 > >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130129025313 2013-01-29 > >>> 02:53:13 +0000 running > >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130128025313 > >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130130025313 2013-01-30 > >>> 02:53:13 +0000 running > >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130129025313 > >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130131025313 2013-01-31 > >>> 02:53:13 +0000 running > >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130130025313 > >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201025313 2013-02-01 > >>> 02:53:13 +0000 running > >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130131025313 > >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201223643 2013-02-01 > >>> 22:36:43 +0000 running > >>> 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201025313 > >>> > >>> The actual qcow2 file looks like a much bigger wreck: > >>> > >>> qemu-img info > /gluster/qcow2/images/7ec192c1-ecee-4f5e-8e3d-567f7878ceb1 > >>> image: /gluster/qcow2/images/7ec192c1-ecee-4f5e-8e3d-567f7878ceb1 > >>> file format: qcow2 > >>> virtual size: 100G (107374182400 bytes) > >>> disk size: 211G > >>> cluster_size: 65536 > >>> backing file: > /gluster/qcow2/images/cb151441-209c-4f43-a2a4-b390c9bb9768 > >>> (actual path: > /gluster/qcow2/images/cb151441-209c-4f43-a2a4-b390c9bb9768) > >>> Snapshot list: > >>> ID TAG VM SIZE DATE VM > CLOCK > >>> 1 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20121230025415 > >>> 1.0G 2012-12-30 02:54:16 785:39:01.680 > >>> 2 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20121231025415 > >>> 1.0G 2012-12-31 02:54:16 809:35:35.417 > >>> 3 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130101025415 > >>> 1.0G 2013-01-01 02:54:16 833:24:20.046 > >>> 4 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130102025415 > >>> 1.0G 2013-01-02 02:54:15 857:13:02.766 > >>> 5 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130103025415 > >>> 1.0G 2013-01-03 02:54:15 881:01:42.467 > >>> 6 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130104025415 > >>> 1.0G 2013-01-04 02:54:15 904:50:29.840 > >>> 7 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130105025415 > >>> 1.0G 2013-01-05 02:54:16 928:39:17.670 > >>> 8 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130106025415 > >>> 1.0G 2013-01-06 02:54:16 952:28:07.139 > >>> 9 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130107025415 > >>> 1.1G 2013-01-07 02:54:16 976:16:58.396 > >>> 10 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130108025415 > >>> 1.1G 2013-01-08 02:54:15 1000:05:45.624 > >>> 11 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130109025415 > >>> 1.1G 2013-01-09 02:54:15 1023:54:25.482 > >>> 12 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130110025415 > >>> 1.1G 2013-01-10 02:54:16 1047:43:17.503 > >>> 13 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130111025415 > >>> 1.1G 2013-01-11 02:54:16 1071:32:03.747 > >>> 14 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130112025415 > >>> 1.1G 2013-01-12 02:54:16 1095:20:52.220 > >>> 15 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130113025415 > >>> 1.1G 2013-01-13 02:54:16 1119:09:43.387 > >>> 16 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130114025415 > >>> 1.1G 2013-01-14 02:54:16 1142:58:32.657 > >>> 17 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130115025415 > >>> 1.1G 2013-01-15 02:54:16 1166:47:20.084 > >>> 18 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130116025416 > >>> 1.1G 2013-01-16 02:54:16 1190:36:06.729 > >>> 19 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130117025416 > >>> 1.1G 2013-01-17 02:54:16 1214:24:53.904 > >>> 20 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130118025415 > >>> 1.1G 2013-01-18 02:54:16 1238:13:35.640 > >>> 21 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130119025415 > >>> 1.1G 2013-01-19 02:54:16 1262:02:17.839 > >>> 22 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130120025415 > >>> 1.1G 2013-01-20 02:54:16 1285:51:01.114 > >>> 23 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130121025415 > >>> 1.1G 2013-01-21 02:54:16 1309:39:49.956 > >>> 24 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130122025416 > >>> 1.1G 2013-01-22 02:54:16 1333:28:39.703 > >>> 25 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130123025415 > >>> 1.1G 2013-01-23 02:54:16 1357:17:25.426 > >>> 26 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130124025416 > >>> 1.1G 2013-01-24 02:54:16 1381:06:04.731 > >>> 27 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130125025313 > >>> 517M 2013-01-25 02:53:13 04:46:52.307 > >>> 28 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130126025313 > >>> 678M 2013-01-26 02:53:13 28:44:35.134 > >>> 29 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130127025313 > >>> 918M 2013-01-27 02:53:13 52:42:00.249 > >>> 30 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130128025313 > >>> 1.1G 2013-01-28 02:53:13 76:38:55.428 > >>> 31 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130129025313 > >>> 1.2G 2013-01-29 02:53:13 100:35:27.978 > >>> 32 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130130025313 > >>> 1.2G 2013-01-30 02:53:13 124:31:55.179 > >>> 33 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130131025313 > >>> 1.3G 2013-01-31 02:53:13 148:28:17.744 > >>> 34 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201025313 > >>> 1.3G 2013-02-01 02:53:13 172:24:33.630 > >>> 35 24661080-ae91-468d-b8f0-030b5b36f786_ROOT-14_20130201223643 > >>> 1.3G 2013-02-01 22:36:43 192:04:17.498 > >>> > >>> Suggestions on how I can clean up these extra snapshots? > >>> > >>> -Bryan > >>> > >> > >> > > >
