As of around CentOS 6.5 (and for sure in RHEL 6.4) that there is a
thinlvm utility that seems to take care of supporting thinly-provisioned
LVM volumes. There is an associated snapshot design based on LVM thin
provisioning that even supports the ability to do snapshots of
snapshots, etc. down the chain.
Unfortunately, from a cursory look, it doesn't look like a back port to
CentOS 5 would be that easy or even possible, but that it is integrated
into both CentOS 6 and CentOS 7 gives some hope for a possibly
standardized support of it in a future XenServer release, doesn't it?
The other big question would be how readily something like this could be
integrated into XenServer.
-=Tobias
On 1/8/2015 9:12 AM, Tobias Kreidl wrote:
Hello, David:
Thanks very much for that feedback. Simple often means good with an
easier implementation and fewer complexities to worry about, so as
suggested some time ago on the Xen Project Web site
https://xenorg.uservoice.com/forums/172169-xen-development?filter=hot&page=1
the thin provisioning of LVM does appear to be a good contender. The
running out of space issue is, of course, endemic in any storage
mechanism, so it's more a question of how to prevent corruption and
recover in that event. As with ext file systems, perhaps keeping a
certain percentage of the volume reserved for emergency issues could
help while refusing additional write commits at some point when all
storage capacity is exhausted (going into "RO" mode). Of course, being
able to copy out and recover lost space would be a useful means of
being able to keep VMs running by doing storage Xenmotions
periodically and recovering the space through some means from the
original disk (if need be though a re-initialization process or an LVM
resize operation). With thin provisioning, it's less of an issue for
some scenarios like a XenDesktop environment containing a large number
of Windows clients because Microsoft comes out with monthly patches
and it is way more efficient to just apply those to your golden images
and reprovision all your VMs. You're then back to your base storage.
It would also be nice to se VHDX adopted for Windows VMs at some
point. And while on the topic of storage, where do things stand with
Ceph? The flexibility looks tremendous, but my understanding is that
there are still performance issues. It has the capability of natively
supporting several different storage options, addressing your comment
about plug-in support for virtually any sort of file system one might
want. It would also be interesting to get an update on both NFSv4
(which works under Creedence, just isn't officially supported) as well
as with potential pNFS support. Some vendors like NetApp showed early
on that NFS is still a very viable option for many storage needs and
our own tests of iSCSI vs. NFS were convincing enough that with an
LACP bond, we have a very solid NFS connection to our NexentaStor SDS
and with thin provisioning plus on-the-fly compression/decompression,
get anywhere from a 20:1 to 50:1 reduction in utilized storage. And as
mentioned already, this is with the equal I/O performance of iSCSI.
It's hard to beat that, especially if you can convince those who hold
the purse strings that you've just increased your storage capacity by
a factor of 20 or more without needing any additional disks.
Again, your efforts in this area are most appreciated and thank you
for continuing to investigate and address these points. Citrix has
been really great about being open to suggestions and feedback from
the user community, so many kudos to you and your colleagues for
lending your ears.
-=Tobias
On 1/8/2015 7:57 AM, Dave Scott wrote:
On 8 Jan 2015, at 14:26, Lorscheider Santiago
<lorscheider.santi...@gmail.com> wrote:
Hi Tobias,
Congratulations analysis. Really if used LVMoSCSI or LVMoHBA will
not have much gain.
That’s right — our block storage types (LVMoiSCSI and LVMoHBA) are
“thick provisioned”
and hence VDIs take up a lot of space. As Tobias says below, this
shows how important
thin provisioning is.
Within the upstream xapi-project (cc:d xen-api) we’ve been looking
into thin provisioning
options for block storage. I’ve written two draft designs:
1. thin LVHD:
http://xapi-project.github.io/xapi/futures/thin-lvhd/thin-lvhd.html
This proposes to extend the existing LVM-based SRs, allowing LVs to
be resized at runtime
as more space is needed. Every LV could start out small (like a file)
and grow as new
blocks are written. The trick is to make sure allocation can still
happen when there
is a network partition in the pool and avoid timing out VM I/O. The
design proposes to
cache free blocks locally on each host, allocate from there, and
replay an allocation
journal against the LVM metadata periodically. [ This is all based on
previous work by
Germano Percossi and Jon Ludlam, so I can’t claim credit :) ]
2. OCFS2:
http://xapi-project.github.io/xapi/futures/ocfs2/ocfs2.html
This proposes to manage OCFS2 clustered filesystem instances and to
store VDIs as files
(format flexible) on top. Most of the design so far is to deal with
the complexity of
managing the configuration and state of the O2CB cluster.
My gut feeling at the moment is that thin LVHD is simpler to
implement and will cause
less “behaviour churn”, since we don’t have to change how HA behaves,
how maintenance mode
is used etc.
Hopefully at some point in the future Xapi will be able to use any
user-managed existing
filesystem, so a Linux expert power-user could use OCFS2 if they wanted.
Cheers,
Dave Scott
Best Regards,
Cordialmente,
Lorscheider Santiago
Visite meu blog: www.centralcloud.info
Twitter: @lsantiagos
Antes de imprimir, pense em sua responsabilidade e compromisso com o
meio ambiente
On Tue, Jan 6, 2015 at 4:40 PM, Tobias Kreidl
<tobias.kre...@nau.edu> wrote:
Thank you very much, Dave, for the updates about this nice addition.
Here is my one concern, namely that for incremental backups to work
properly, you need to continuously keep a base snapshot on the
server, hence you will -- even if sparse storage is utilized --
still take up a fair amount of storage as long as you want to
maintain the snapshot and backup mechanism.
This led me to thinking, so, what happens if you have SRs with
different underlying properties? What kind of space requirements are
being realized?
As a test, I took both a Linux RHEL 7 and a Windows 8.1 VM and
checked how much space was really being used as well as allocated on
the SR, as well as in the export files. Please note that here, some
extra space savings take place on the NexentaStor NFS mount because
of some built-in automatic compression taking place.
For a Linux VM with 8 GB of allocated storage on a VDI and around
1.63 GB of space in use and a Windows VM with 40 GB of allocated
storage and around 20 GB in actual use, here are the numbers I came
up with:
RHEL 7 RHEL 7 Windows 8.1
Windows 8.1
Function LVM NFS+compression LVM
NFS+compression
base used: 8.0 GB 1.1 GB 40.0 GB
20.2 GB
base allocated: 8.0 GB 8.0 GB 40.0 GB
40.0 GB
pre-snapshot
virtual-size: 8589923591 8589934592 42049672960
42949672960
physical-utilisation: 8615100416 5984371200 43041947648
20136346112
snapshot used: 13.6 GB 1.1 GB 59.0 GB
20.2 GB
snapshot allocated: 16.0 GB 16.0 GB 80.1 GB
80.0 GB
full snapshot
virtual-size: 8589934592 8589934592 42949672960
42949672960
physical-virtualisation: 8388608 39944704 8388608 88576
snapshot
export size (VHD): 5.97 GB 5.97 GB 20.24 GB
20.22 GB
delta snapshot used: 13.6 GB 1.1 GB 59.1 GB
0.3 GB
delta snap allocated: 24.0 GB 24.0 GB 120.1 GB
120.0 GB
delta snapshot
virtual-size: 8589934592 8589934592 42949672960
42949672960
physical-virtualistion: 8388608 20992 8388608 88576
delta snapshot
export size: 0.126 GB 0.044 GB 0.078 GB
0.074 GB
Not surprising is that the exports all came our roughly the same
(they are, after all, creating VHD files in all cases), with minor
difference due to some activity between when the exports were
created. However, what is particularly interesting in the NFS-based
SR is that the initial snapshot seems to have taken up virtually no
discernible space at all. Why? Evidently because it creates it's own
initial "difference" disk. Running "xe vdi-list uuid=(UUID_of_VDI)
params=all" on the VDI yields details on the space
utilization/allocation. This is actually quite interesting, as I had
not tried this with an already thinly-provisioned (as opposed to
sparely populated) storage before.
Noting how an NFS-based SR is so much more space efficient, it does
however leave the issue that if create both a full and delta
snapshot, you will no longer be able to storage Xenmotion the VM
until you bring the snapshot count back down to two or lower. One
option would be to immediately clean this up after the latest delta
vdi-export has taken place; alternatively, you could just defer this
until a storage Xenmotion or other action is called for that
requires prior cleanup.
This brings us back to the LVM case. The huge difference here is
that you need readily three times the size of the VDI to be
allocated and at least double the size to be able to retain a base
copy and triple the size to hold in addition a delta snapshot.
Plus, unlike a NFS-based SR, you cannot over-commit your storage
allocation on the SR. it would hence be of great benefit in space
savings if one could do one of the following: (1) store a
thinly-provisioned and compressed version of the initial snapshot
similar to what NexentaStor does with the NFS VM, (2) in creating an
incremental snapshot, access somehow an off-line VDI file containing
the base, (3) had the means to efficiently temporarily pull in an
offlined VDI so you'd not have to keep the base snapshot on the SR
constantly, (4) be able to store the base snapshot on a different SR
than the VDI you want to snapshot, or (5) some other clever, unnamed
mechanism.
The NAU VMbackup mechanism we have used in-house for years is
efficient mainly because it creates a full snapshot on the fly for
the purpose of a full backup and once completed, deletes it. Hence,
you never need more additional space at any given time than that of
a copy of the largest storage associated with an particular VM. The
disadvantage is that this is of course a sequential operation and
hence takes quite some time if you have a lot of VMs. However, being
able to snapshot and keep a spare image would still take up a large
amount of extra space. While having the means to very efficiently
create incremental (delta) snapshots and back them up, this still
leaves the large storage requirement issue open for LVM-based
storage, should one want to retain baseline snapshots over longer
periods of time.
To me, this is an indication how thin provisioning (in contrast to
just sparse storage) can make a huge difference. It would be really
interesting to see what other options could be implemented to help
address the LVM limitations outlined above.
Thanks for taking the time to look over these thoughts, and I most
certainly welcome feedback, in particular if I have overlooked
something blatant.
-=Tobias
On 1/5/2015 8:09 AM, Dave Scott wrote:
On 31 Dec 2014, at 11:27, Lorscheider Santiago
<lorscheider.santi...@gmail.com>
wrote:
Hi Dave!
Very good xapi the project site. Thanks for the tip!
http://xapi-project.github.io/
About recuros of "export and import only the blocks Which have
changed between two snapshots" this is a sensational news! It's
been times that I miss this feature, which brings tremendous
agility to backup. From what you wrote, similar operation to
VMware CBT.
By your tests, you confirmed that this resource is in the build
90383c. We can not test since the Release Candidate build is in
90239c. Although he had already noticed changes in the snapshot in
earlier builds the 90239c.
Congratulations for the work and thanks for sharing this
information with us!
I’m glad you liked the new site! Credit for the site itself and
much of the content should also go to: (extracted from git history
in no particular order)
Euan Harris
John Else
Jonathan Davies
Rob Hoes
Si Beaumont
If you find any problems with it or have suggestions for
improvement (or new content), feel free to make pull requests or
make issues on the tracker:
https://github.com/xapi-project/xapi-project.github.io
Cheers,
Dave
Cordialmente,
Lorscheider Santiago
Visite meu blog:
www.centralcloud.info
Twitter: @lsantiagos
Antes de imprimir, pense em sua responsabilidade e compromisso com
o meio ambiente
On Mon, Dec 29, 2014 at 7:47 PM, Dave Scott
<dave.sc...@citrix.com>
wrote:
On 29 Dec 2014, at 15:31, Dennis Duane Booher <duane.boo...@nau.edu>
wrote:
Hi Dave,
This looks very interesting. Since this is in the xapi-project
does this mean it is intended for some future release? Is it in
the current Creedence pre-release build?
I believe the code for this is present in the recent creedence
builds. Just to check I installed build number ‘90383c’ and did a
bit of smoke testing— the xe commands listed in the xapi-project
page ran ok and produced the correct output.
HTH,
Dave
Duane
-----Original Message-----
From:
xs-devel-requ...@lists.xenserver.org
[mailto:xs-devel-requ...@lists.xenserver.org
] On Behalf Of Dave Scott
Sent: Sunday, December 28, 2014 4:05 AM
To:
xs-de...@lists.xenserver.org
Subject: Re: [xs-devel] Backup solutions for XenServer
Hi,
On 27 Dec 2014, at 12:07, Lorscheider Santiago
<lorscheider.santi...@gmail.com>
wrote:
Hi Duane,
I do not mean backup tools but the improvements that XenServer
6.5 in your api to facilitate the work of backup tools.
I've written up some of the recent XenAPI improvements here:
http://xapi-project.github.io/features/snapshots/snapshots.html
The new APIs allow you to
- export and import disks in .vhd format (previously we only
supported raw). This means the files are sparse.
- export and import only the blocks which have changed between
two snapshots. This allows you to avoid re-copying the same data
again and again.
Let me know what you think.
Cheers,
Dave
Cordialmente,
Lorscheider Santiago
Visite meu blog:
www.centralcloud.info
Twitter: @lsantiagos
Antes de imprimir, pense em sua responsabilidade e compromisso
com o meio ambiente
On Fri, Dec 19, 2014 at 8:27 PM, Dennis Duane Booher
<duane.boo...@nau.edu>
wrote:
Hi Lorscheider,
We use
https://github.com/NAUbackup
for all our production Xenserver VM backups. It is freely
available if you would like to give it a try.
Duane
From:
xs-devel-requ...@lists.xenserver.org
[mailto:xs-devel-requ...@lists.xenserver.org
] On Behalf Of Lorscheider Santiago
Sent: Friday, December 19, 2014 3:33 AM
To: xs-devel
Subject: [xs-devel] Backup solutions for XenServer
Hi All,
Looking backup solutions available on the market, solutions with
good support XenServer are limited. The changes made in
creedence api will make it easier and more attractive than more
backup siluçoes have support XenServer?
Regards,
Lorscheider Santiago
Visite meu blog:
www.centralcloud.info
Twitter: @lsantiagos
Antes de imprimir, pense em sua responsabilidade e compromisso
com o meio ambiente
_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
