Yes, it is a big problem. I have not checked the code that creates the template, but I am not sure if the problem you are experiencing is related to 9025. It might be a problem in some other place.
My suggestion is for you to create a Jira ticket detailing the situation and wait until someone fixes it. Have you tested on ACS 4.10? If 4.9.2.0 works, it might be something that went only into 4.9.3.0. On Tue, Oct 31, 2017 at 8:04 PM, Lotic Lists <li...@lotic.com.br> wrote: > I tested on a fresh environment, acs 4.9.3.0, xenserver 6.5 with all > patches and primary storage NFS, we have same problem > > - Snapshot files > [root@acs01 203]# ls -lh /exports/secondary/snapshots/2/6/ > total 2.8G > -rw-r--r-- 1 root root 1.7G Oct 31 16:48 > cad96f0c-de49-4255-b0a0-9aea5e2297cb.vhd > <-- snap-base > -rw-r--r-- 1 root root 1.1G Oct 31 16:49 > fa4e7e4d-531d-42dd-8a80-df6b274584a8.vhd > <-- snap-diff > https://i.imgur.com/bWupsAS.png > > [root@acs01 203]# md5sum /exports/secondary/snapshots/ > 2/6/fa4e7e4d-531d-42dd-8a80-df6b274584a8.vhd > 17eec8d4f6d4c128b34b9e2e1876ceb7 /exports/secondary/snapshots/ > 2/6/fa4e7e4d-531d-42dd-8a80-df6b274584a8.vhd > > - Template files > [root@acs01 203]# ls -lh /exports/secondary/template/tmpl/2/203 > total 2.0G > -rw-r--r-- 1 root root 1.1G Oct 31 16:50 e9def45d-1b9e-4c2e-b843- > 852fe40f00b2.vhd > -rw-r--r-- 1 root root 303 Oct 31 16:50 template.properties > > [root@acs01 203]# md5sum /exports/secondary/template/ > tmpl/2/203/e9def45d-1b9e-4c2e-b843-852fe40f00b2.vhd > 17eec8d4f6d4c128b34b9e2e1876ceb7 /exports/secondary/template/ > tmpl/2/203/e9def45d-1b9e-4c2e-b843-852fe40f00b2.vhd > > It's a huge problem!!! I think issue 9025 is not resolved > > Regards > Marcelo > > -----Original Message----- > From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com] > Sent: terça-feira, 31 de outubro de 2017 16:00 > To: us...@cloudstack.apache.org > Subject: Re: Strange size of template from snapshot on XenServer > > Well, reading the description of the issue it seems to be related to the > problem you are describing. However, I compared the classes that were > changed with PR (https://github.com/apache/cloudstack/pull/1176) and they > are the same in ACS 4.9.2.0 and 4.9.3.0. > > On Tue, Oct 31, 2017 at 6:36 PM, Lotic Lists <li...@lotic.com.br> wrote: > > > Hi Rafael, thanks for test. > > > > I found the issue > > https://issues.apache.org/jira/browse/CLOUDSTACK-9025 > > I think is a same case. I will up a new lab environment now for > > simulate the issue > > > > Regards. > > Marcelo > > > > -----Original Message----- > > From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com] > > Sent: segunda-feira, 30 de outubro de 2017 12:08 > > To: us...@cloudstack.apache.org > > Subject: Re: Strange size of template from snapshot on XenServer > > > > I just did, and the size of the template is the size of the root disk > > + the 1GB file I created in the test vm. > > > > The system I tested this is an ACS 4.9.2, XenServer 6.5, Primary Stg > > ISCSI > > > > On Mon, Oct 30, 2017 at 10:20 AM, Lotic Lists <li...@lotic.com.br> > wrote: > > > > > Good morning > > > > > > Guys, who could execute the test below? > > > > > > 1. Create a manual snapshot of a volume 2. Create a file (1GB) with > > > dd on VM 3. Create second snapshot of a volume 4. Convert latest > > > snapshot to a template 5. Verify the size of vhd of new template in > > > secondary storage. > > > > > > Here the template have 1GB, exactly the size of latest snapshot. > > > > > > ACS 4.9.3, XenServer 6.5, Primary Stg ISCSI > > > > > > > > > -----Original Message----- > > > From: Lotic Lists [mailto:li...@lotic.com.br] > > > Sent: quinta-feira, 26 de outubro de 2017 15:03 > > > To: us...@cloudstack.apache.org > > > Subject: RE: Strange size of template from snapshot on XenServer > > > > > > I test the same case on other environment, create template from > > > snapshot, acs 4.9.2.0 and XenServer 6.5/iscsi > > > > > > 1. Was created two snapshots > > > > > > # ls -lh /var/cloud_mount/e5921d9a-70b0-3501-9fda-6097e721fffb/ > > > snapshots/131/617/ > > > total 3.0G > > > -rw-r--r-- 1 4294967294 root 13G Oct 26 13:46 > > > 42c7c51d-4aa7-40d2-b5eb- cb56fc963974.vhd > > > -rw-r--r-- 1 4294967294 root 1.2G Oct 26 14:42 > > > 53b33a5a-a7d9-4efa-b126- 38eec5482b05.vhd > > > > > > https://i.imgur.com/cSHZAJ3.png > > > > > > 2. create a template from latest snapshot > > > > > > # ls -lh > > > total 37M > > > -rw-r--r-- 1 4294967294 root 1.2G Oct 26 14:43 > > > 637604a6-3085-444e-ba09- b9aab11a2c16.vhd > > > -rw-r--r-- 1 4294967294 root 303 Oct 26 14:43 template.properties > > > > > > # cat template.properties > > > #Thu Oct 26 16:43:54 UTC 2017 > > > filename=637604a6-3085-444e-ba09-b9aab11a2c16.vhd > > > id=1 > > > vhd=true > > > vhd.filename=637604a6-3085-444e-ba09-b9aab11a2c16.vhd > > > public=true > > > uniquename=637604a6-3085-444e-ba09-b9aab11a2c16 > > > vhd.virtualsize=21474836480 > > > virtualsize=21474836480 > > > hvm= > > > vhd.size=1197752832 > > > size=1197752832 > > > > > > # md5sum /var/cloud_mount/e5921d9a-70b0-3501-9fda-6097e721fffb/ > > > snapshots/131/617/53b33a5a-a7d9-4efa-b126-38eec5482b05.vhd > > > 9d2078a6f41ca5e1eab824848c06df53 /var/cloud_mount/e5921d9a- > > > 70b0-3501-9fda-6097e721fffb/snapshots/131/617/53b33a5a- > > > a7d9-4efa-b126-38eec5482b05.vhd > > > > > > # md5sum /var/cloud_mount/e5921d9a-70b0-3501-9fda-6097e721fffb/ > > > template/tmpl/2/294/637604a6-3085-444e-ba09-b9aab11a2c16.vhd > > > 9d2078a6f41ca5e1eab824848c06df53 /var/cloud_mount/e5921d9a- > > > 70b0-3501-9fda-6097e721fffb/template/tmpl/2/294/637604a6- > > > 3085-444e-ba09-b9aab11a2c16.vhd > > > > > > ACS not merge files of snapshots in a single file for template, I > > > think it's a bug. > > > > > > > > > -----Original Message----- > > > From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com] > > > Sent: quarta-feira, 25 de outubro de 2017 22:51 > > > To: us...@cloudstack.apache.org > > > Subject: Re: Strange size of template from snapshot on XenServer > > > > > > It is easy to test, get one of these files and try to use them in > > > Virtualbox. If they run, they do not require anything else. > > > > > > On Wed, Oct 25, 2017 at 10:49 PM, Lotic Lists <li...@lotic.com.br> > > wrote: > > > > > > > Sure but files on secondary storage don't have parents, I don't > > > > know if ACS can connect standalone files. > > > > > > > > vhd-util scan -f -m'*.vhd' -p > > > > vhd=14818597-55bb-49be-9ace-761e9e01c074.vhd capacity=311385128960 > > > > size=304124744192 hidden=0 parent=none > > > > vhd=bbd9e9b5-53fb-44a2-afbe-f262396a2d84.vhd capacity=311385128960 > > > > size=17770869248 hidden=0 parent=none > > > > vhd=3b90bbb2-7ce5-41f6-9f7e-fd0ea061cb2d.vhd capacity=311385128960 > > > > size=52199817728 hidden=0 parent=none > > > > vhd=4b450217-c9ad-45a2-946b-de3cb323469b.vhd capacity=311385128960 > > > > size=51615670784 hidden=0 parent=none > > > > > > > > > > > > -----Original Message----- > > > > From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com] > > > > Sent: quarta-feira, 25 de outubro de 2017 22:35 > > > > To: us...@cloudstack.apache.org > > > > Subject: Re: Strange size of template from snapshot on XenServer > > > > > > > > If it was not executing coalesce you would see the parent > > > > reference when you listed the hierarchy, right? > > > > > > > > On Wed, Oct 25, 2017 at 10:25 PM, Lotic Lists <li...@lotic.com.br> > > > wrote: > > > > > > > > > I think ACS preserve all VHDs for coalesce but coalesce not > > > > > occur when create a template or before starting next snapshot > scheduled. > > > > > If I create a template from any snapshots (2) showing in GUI, > > > > > the template have 49GB. 49 GB is the size of the 2 latest files > > > > > on secondary > > > > storage. > > > > > > > > > > What do you think? Bug? > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com] > > > > > Sent: quarta-feira, 25 de outubro de 2017 22:07 > > > > > To: us...@cloudstack.apache.org > > > > > Subject: Re: Strange size of template from snapshot on XenServer > > > > > > > > > > Aha, The two destroyed entries are showing " removed: NULL". > > > > > > > > > > In ACS one thing is the "destroying" of a resource, another is > > > > > the removal of this resource from the system. I believe there is > > > > > something to do with the expunge interval, but I am not sure if > > > > > the snapshots complete removal also happens within the expunge > > > > > interval of user VMs or if it is configured by something else. > > > > > > > > > > > > > > > On Wed, Oct 25, 2017 at 10:03 PM, Lotic Lists > > > > > <li...@lotic.com.br> > > > > wrote: > > > > > > > > > > > I not start manual snapshot, just scheduled with keep=2 > > > > > > > > > > > > GUI showing 2 snapshots, database showing 4 entries, 2 > > > > > > Destroyed and > > > > > > 2 BackedUp > > > > > > > > > > > > *************************** 132. row *************************** > > > > > > id: 6380 > > > > > > data_center_id: 2 > > > > > > account_id: 64 > > > > > > domain_id: 49 > > > > > > volume_id: 2102 > > > > > > disk_offering_id: 230 > > > > > > status: Destroyed > > > > > > path: NULL > > > > > > name: FullCRM_ROOT-542_20171022030205 > > > > > > uuid: 1294cdb7-01bf-4e76-b4c0-11b3c6b395e9 > > > > > > snapshot_type: 4 > > > > > > type_description: DAILY > > > > > > size: 311385128960 > > > > > > created: 2017-10-22 03:02:05 > > > > > > removed: NULL > > > > > > backup_snap_id: NULL > > > > > > swift_id: NULL > > > > > > sechost_id: NULL > > > > > > prev_snap_id: NULL > > > > > > hypervisor_type: XenServer > > > > > > version: 2.2 > > > > > > s3_id: NULL > > > > > > min_iops: NULL > > > > > > max_iops: NULL > > > > > > *************************** 133. row *************************** > > > > > > id: 6416 > > > > > > data_center_id: 2 > > > > > > account_id: 64 > > > > > > domain_id: 49 > > > > > > volume_id: 2102 > > > > > > disk_offering_id: 230 > > > > > > status: Destroyed > > > > > > path: NULL > > > > > > name: FullCRM_ROOT-542_20171023030205 > > > > > > uuid: 0a8067a3-ad15-40b6-8b16-c1881d7d9f41 > > > > > > snapshot_type: 4 > > > > > > type_description: DAILY > > > > > > size: 311385128960 > > > > > > created: 2017-10-23 03:02:05 > > > > > > removed: NULL > > > > > > backup_snap_id: NULL > > > > > > swift_id: NULL > > > > > > sechost_id: NULL > > > > > > prev_snap_id: NULL > > > > > > hypervisor_type: XenServer > > > > > > version: 2.2 > > > > > > s3_id: NULL > > > > > > min_iops: NULL > > > > > > max_iops: NULL > > > > > > *************************** 134. row *************************** > > > > > > id: 6455 > > > > > > data_center_id: 2 > > > > > > account_id: 64 > > > > > > domain_id: 49 > > > > > > volume_id: 2102 > > > > > > disk_offering_id: 230 > > > > > > status: BackedUp > > > > > > path: NULL > > > > > > name: FullCRM_ROOT-542_20171024030207 > > > > > > uuid: 1ca646e5-a155-4a59-aa31-5889eec7536f > > > > > > snapshot_type: 4 > > > > > > type_description: DAILY > > > > > > size: 311385128960 > > > > > > created: 2017-10-24 03:02:07 > > > > > > removed: NULL > > > > > > backup_snap_id: NULL > > > > > > swift_id: NULL > > > > > > sechost_id: NULL > > > > > > prev_snap_id: NULL > > > > > > hypervisor_type: XenServer > > > > > > version: 2.2 > > > > > > s3_id: NULL > > > > > > min_iops: NULL > > > > > > max_iops: NULL > > > > > > *************************** 135. row *************************** > > > > > > id: 6497 > > > > > > data_center_id: 2 > > > > > > account_id: 64 > > > > > > domain_id: 49 > > > > > > volume_id: 2102 > > > > > > disk_offering_id: 230 > > > > > > status: BackedUp > > > > > > path: NULL > > > > > > name: FullCRM_ROOT-542_20171025030208 > > > > > > uuid: 759d61dc-e3db-4815-b68b-b90904ccbda5 > > > > > > snapshot_type: 4 > > > > > > type_description: DAILY > > > > > > size: 311385128960 > > > > > > created: 2017-10-25 03:02:08 > > > > > > removed: NULL > > > > > > backup_snap_id: NULL > > > > > > swift_id: NULL > > > > > > sechost_id: NULL > > > > > > prev_snap_id: NULL > > > > > > hypervisor_type: XenServer > > > > > > version: 2.2 > > > > > > s3_id: NULL > > > > > > min_iops: NULL > > > > > > max_iops: NULL > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com] > > > > > > Sent: quarta-feira, 25 de outubro de 2017 21:50 > > > > > > To: us...@cloudstack.apache.org > > > > > > Subject: Re: Strange size of template from snapshot on > > > > > > XenServer > > > > > > > > > > > > The file "14818597-55bb-49be-9ace-761e9e01c074.vhd" seems more > > > > > > like a manual snapshot. It was created 2 hours after the "normal" > > > > > > time. I would check these snapshots in ACS and see how they > > > > > > are > > > presented. > > > > > > BTW, how many snapshots is ACS showing for this volume? I > > > > > > would also check the "snapshots" table for the volume you have > > > > > > configured > > > them. > > > > > > select * from snapshots where volume_id = ? > > > > > > > > > > > > you may also filter for removed: > > > > > > select * from snapshots where volume_id = ? and removed is > > > > > > null > > > > > > > > > > > > On Wed, Oct 25, 2017 at 9:39 PM, Lotic Lists > > > > > > <li...@lotic.com.br> > > > > wrote: > > > > > > > > > > > > > Hi Rafael, I check > > > > > > > > > > > > > > # ls -lrt > > > > > > > total 285368382 > > > > > > > -rw-r--r-- 1 root root 304124744192 Oct 22 03:28 > > > > > > > 14818597-55bb-49be-9ace- 761e9e01c074.vhd > > > > > > > -rw-r--r-- 1 root root 17770869248 Oct 23 01:10 > > > > > > > bbd9e9b5-53fb-44a2-afbe- f262396a2d84.vhd > > > > > > > -rw-r--r-- 1 root root 52199817728 Oct 24 01:30 > > > > > > > 3b90bbb2-7ce5-41f6-9f7e- fd0ea061cb2d.vhd > > > > > > > -rw-r--r-- 1 root root 51615670784 Oct 25 01:28 > > > > > > > 4b450217-c9ad-45a2-946b- de3cb323469b.vhd > > > > > > > > > > > > > > # vhd-util scan -f -m'*.vhd' -p > > > > > > > vhd=14818597-55bb-49be-9ace-761e9e01c074.vhd > > > > > > > capacity=311385128960 > > > > > > > size=304124744192 hidden=0 parent=none > > > > > > > vhd=bbd9e9b5-53fb-44a2-afbe-f262396a2d84.vhd > > > > > > > capacity=311385128960 > > > > > > > size=17770869248 hidden=0 parent=none > > > > > > > vhd=3b90bbb2-7ce5-41f6-9f7e-fd0ea061cb2d.vhd > > > > > > > capacity=311385128960 > > > > > > > size=52199817728 hidden=0 parent=none > > > > > > > vhd=4b450217-c9ad-45a2-946b-de3cb323469b.vhd > > > > > > > capacity=311385128960 > > > > > > > size=51615670784 hidden=0 parent=none > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Rafael Weingärtner > > > > > > > [mailto:rafaelweingart...@gmail.com] > > > > > > > Sent: quarta-feira, 25 de outubro de 2017 20:44 > > > > > > > To: us...@cloudstack.apache.org > > > > > > > Subject: Re: Strange size of template from snapshot on > > > > > > > XenServer > > > > > > > > > > > > > > Did you check the hierarchy of those snapshot files in your > > > > > > > secondary storage? > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Oct 25, 2017 at 7:44 PM, Lotic Lists > > > > > > > <li...@lotic.com.br> > > > > > wrote: > > > > > > > > > > > > > > > Hi all > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Anyone identify problems with templates from snapshots on > > > > XenServer? > > > > > > > > > > > > > > > > I created a recurrent snapshot, first VHD on secondary > > > > > > > > storage have a similar size of volume. If I create a > > > > > > > > template from latest snapshot, the size of VHD of template > > > > > > > > is smaller equal from last snapshot, I think > > > > > > > > CloudStack/Xenserver not coalesce VHD files of secondary > > > > > > > > storage for create the template > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I keep two recurrent snapshots, on secondary storage have > > > > > > > > four > > > > files. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Environment is ACS 4.9.3.0 and XenServer 6.5, primary > > > > > > > > storage is PreSetup with ISCSI. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > Marcelo > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Rafael Weingärtner > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Rafael Weingärtner > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Rafael Weingärtner > > > > > > > > > > > > > > > > > > > > > > -- > > > > Rafael Weingärtner > > > > > > > > > > > > > > > > > -- > > > Rafael Weingärtner > > > > > > > > > > > > > > > -- > > Rafael Weingärtner > > > > > > > -- > Rafael Weingärtner > > -- Rafael Weingärtner
