Re: [one-users] experiences with distributed FS?
Hi,
I'm one of the developers of the distributed file system XtreemFS. I
just noticed this thread since one of the posters followed us on
Twitter. So I just joint the mailing list.
Internally we also run a OpenNebula cluster and use XtreemFS as shared
file system there. Since XtreemFS is POSIX compatible, you can use the
tm_nfs and just point OpenNebula to the mounted POSIX volume.
A colleague of mine did setup the installation which is based on
OpenNebula 2.2. So I'm not fully aware of the details. He told me that
he has also pointed the variable VM_DIR (in $ONE_LOCATION/etc/oned.conf)
to a XtreemFS volume, so our KVM-based installation does support live
migration. So far we haven't seen any problems with using XtreemFS in
OpenNebula, otherwise we would have fixed it ;-)
Regarding the performance: We did not do any measurements so far. Any
numbers or suggestions how to benchmark different distributed file
systems are welcome.
In XtreemFS the metadata of a volume is stored on one metadata server.
If you like, you can set it up replicated and then your file system will
have no single point of failure.
Regarding support: Although we cannot offer commercial support, we
provide support through our mailing list and are always eager to help.
Best regards,
Michael
On 02/09/2012 09:52 AM, Gian Uberto Lauri wrote:
HE == Hans-Joachim Ehlershansjoachim.ehl...@eumetsat.int
writes:
HE Hi Joao,
We're looking into the possibility of using a distributed FS, like
glusterfs or similar solution for the image repository
HE We are looking forward to use GPFS as a clustered FS for our
HE upcoming KVM OpenNebula solution since we are already a happy
HE GPFS user.
Beware of the clustering software, be sure to use a very recent
release (i.e. the on in Debian Stable).
I tried it with an older, broken version of the clustering software and
initially it worked wonderfully well... Until the cluster software began
to malfunction and kick the servers out.
But we plan to use something like this in the future (since we have
sysadmin much better than me available :) ).
--
ing. Gian Uberto Lauri
Ricercatore / Reasearcher
Laboratorio Ricerca e Sviluppo / Research Development Lab.
Area Calcolo Distribuito / Distributed Computation Area
gianuberto.la...@eng.it
Engineering Ingegneria Informatica spa
Corso Stati Uniti 23/C, 35127 Padova (PD)
Tel. +39-049.8283.571 | main(){printf(unix[\021%six\012\0],
Fax +39-049.8283.569 |(unix)[have]+fun-0x60);}
Skype: gian.uberto.lauri | David Korn, ATT Bell Labs
http://www.eng.it | ioccc best One Liner, 1987
___
Users mailing list
Users@lists.opennebula.org
http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
___
Users mailing list
Users@lists.opennebula.org
http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
Re: [one-users] experiences with distributed FS?
On Thu, Feb 9, 2012 at 11:17 AM, Michael Berlin Regarding the performance: We did not do any measurements so far. Any numbers or suggestions how to benchmark different distributed file systems are welcome. Hmm, you are using FUSE. Performance measurements would be really nice to have. Writing fancy file systems using FUSE is easy. Making them fast and scalable is a damn hard job and often impossible. -- Thanks, //richard ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
Re: [one-users] experiences with distributed FS?
Hi Joao. I made a similar search recently. Here is my results: Lustre - does not have redundancy, If you will use file striping between nodes and one nodes go offline all data are not available. Gluster - does not support KVM virtualization. Software developer lead mentioned that it will be fixed in next release (April). Shipping Dog - Working only with images does not allow to store plain files on it. one image can be connected to only one VM per a time. eXtremFS - does not have support. If something not working it is your problem even you ready to pay for fixing. And my choice: MooseFS - redundant any node can go offline and data will be available, scalable, with striping, with CoW, Create copy of huge images in a second, has internal checksum correction, has commercial support, data deduplication (commercial version only) and many other features. there are plugging for OpenNebula. It has some bugs but you can start from this point. Regards, Max On 02/08/2012 08:18 AM, João Pagaime wrote: Hello all, We're looking into the possibility of using a distributed FS, like glusterfs or similar solution for the image repository Can anyone share his experiences on this topic? any hints would be nice... Existing specific documentation? REcommendend configurations? Tested configurations? Things to watch out for? Things particular to open-nebula? Problems? Stability (is it maintainable without a general shutdown)? Effort/learning curve (large, small, hours, days, weeks)? Any other information? Thanks any way, best regards, João ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
Re: [one-users] experiences with distributed FS?
Our research project is currently using GlusterFS for our distributed NFS storage system. We're leveraging the distributed-replica configuration in which every two pairs of servers is a replica pair, and all pairs form a distributed cluster. We do not do data stripping since to achieve up-time reliability with stripping would require too many servers. Furthermore, another nice feature of GlusterFS, is that you can just install it into a VM, clone it a few times, and distribute them across VMMs. However, we utilize physical servers using RAID-1. One issue with MooseFS is that it relies upon a single Metadata server. Therefore, if that Metadata server fails, the cluster fails. GlusterFS does not have a Metadata server. -- Sincerely, Hutson Betts (Computer Science and Engineering - Texas AM University) Key ID: F98BFC1E http://pgp.mit.edu/ On Wed, 2012-02-08 at 11:15 -0500, Maxim Mikheev wrote: Hi Joao. I made a similar search recently. Here is my results: Lustre - does not have redundancy, If you will use file striping between nodes and one nodes go offline all data are not available. Gluster - does not support KVM virtualization. Software developer lead mentioned that it will be fixed in next release (April). Shipping Dog - Working only with images does not allow to store plain files on it. one image can be connected to only one VM per a time. eXtremFS - does not have support. If something not working it is your problem even you ready to pay for fixing. And my choice: MooseFS - redundant any node can go offline and data will be available, scalable, with striping, with CoW, Create copy of huge images in a second, has internal checksum correction, has commercial support, data deduplication (commercial version only) and many other features. there are plugging for OpenNebula. It has some bugs but you can start from this point. Regards, Max On 02/08/2012 08:18 AM, João Pagaime wrote: Hello all, We're looking into the possibility of using a distributed FS, like glusterfs or similar solution for the image repository Can anyone share his experiences on this topic? any hints would be nice... Existing specific documentation? REcommendend configurations? Tested configurations? Things to watch out for? Things particular to open-nebula? Problems? Stability (is it maintainable without a general shutdown)? Effort/learning curve (large, small, hours, days, weeks)? Any other information? Thanks any way, best regards, João ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org signature.asc Description: This is a digitally signed message part ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
Re: [one-users] experiences with distributed FS?
2012/2/8 João Pagaime j...@fccn.pt: Can anyone share his experiences on this topic? any hints would be nice... I've written a small article few months ago after a successful deployment of OpenNebula on top of a MooseFS volume: http://blog.opennebula.org/?p=1512 Short anwser: yes, they works, but you need to fully understand how they work before using them with OpenNebula. Build a test environment and try different configurations, make some test plugging randomly power-chord and try to recover by yourself :) -- Giovanni Toraldo http://gionn.net/ ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
Re: [one-users] experiences with distributed FS?
On 2012/02/08 5:42 PM, Alberto Zuin - Liste wrote: In past we used OpenNebula 2.2 with MooseFS driver for immediate deployment via snapshot, but the driver wasn't updated to OpenNebula 3.0. We are very happy with MooseFS: it's very easy and robust. The only consideration is on disk side: SATA is a little slow expecially in write operation, then we use a lot of RAM on each storage server (8 GB) to cache data. I can agree with the point on slowness I had a shared moosefs setup running (with only 4 moosefs servers, with 2x 1Tb SATA drives in each), and found disk writes to be far too slow to host our test environment correctly I have migrated this to a custom tm, which uses moosefs for image store, and copies vm images to local storage to actually run. I am busy investigating moosefs slowness on a separate testing cluster (and have so far improved bonnie++ rewrite speed from 1.5MB/s to 18MB/s). I should have something I am willing to put back into use as a proper shared filesystem soon (I really like using the snapshot capabilites to deploy vms) Chris ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
Re: [one-users] experiences with distributed FS?
On 2012/02/08 7:54 PM, Chris Picton wrote: On 2012/02/08 5:42 PM, Alberto Zuin - Liste wrote: In past we used OpenNebula 2.2 with MooseFS driver for immediate deployment via snapshot, but the driver wasn't updated to OpenNebula 3.0. We are very happy with MooseFS: it's very easy and robust. The only consideration is on disk side: SATA is a little slow expecially in write operation, then we use a lot of RAM on each storage server (8 GB) to cache data. I can agree with the point on slowness Just to clarify - moosefs is fast in almost all respects, but is slow when continually reading from and writing to the same block of a file. Unfortunately the workload on some of my vms seems to hit this case fairly often, and that is what I am trying to optimise. I had a shared moosefs setup running (with only 4 moosefs servers, with 2x 1Tb SATA drives in each), and found disk writes to be far too slow to host our test environment correctly I have migrated this to a custom tm, which uses moosefs for image store, and copies vm images to local storage to actually run. I am busy investigating moosefs slowness on a separate testing cluster (and have so far improved bonnie++ rewrite speed from 1.5MB/s to 18MB/s). I should have something I am willing to put back into use as a proper shared filesystem soon (I really like using the snapshot capabilites to deploy vms) Chris ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
