Re: Storage cluster with FreeBSD
Norberto Meijome wrote: > indeed. Well, as I said in OP, similar to what Lustre offers. I don't know what Lustre has so my response might or might not be what you need... > Let's see : what I am after is a way to hook up a few computers ( say, 6) with > a few HD (say, 6 x 400 GB), and setup across the whole thing a RAID that > would let me : > - keep running with no loss of data in case of a disk or node failure (or > minimal in case of a node failure) > > - can be accessed by clients over standard network protocols (NFS, CIFS, etc) > > - the clients see volumes of x GB (maybe 1 of the total size, maybe 3 of > different sizes...doesn't matter) > > All this until hear seems doable with FBSD + GEOM architecture. Now the tricky > ones : May or may not. Do you want to have the drives attached to individual machines, then exported via ggate, then mounted on each machine and then RAID-ed together? This looks like it needs a large overhead in administration. If you have the drives attached to individual machines and then one machine that imports them all in some way (ggate) and exports them via NFS and CIFS, then you have a single point of failure. In truth, there appears to be only one volume manager that's close to being usable, and that's ZFS in 7. But there's nothing automatic in the way it can use the drives and (re) export them via NFS, etc. There is no distributed file systems for FreeBSD. > - scales linearly - if I add 90 storage hosts, the system doesnt get bogged > down with the management of the disks / stripe distributoin The only things you have with GEOM are low-level building blocks: RAID transformations and network devices. If you calculate for yourself that, (a silly example) if you create a RAID1 out of 90 hosts you can survive the data being explicitely sent to each of the hosts individually, then go ahead. There is no magic or smart behaviour involved anywhere. > - I can add new hosts (from 6 to 10) and the new storage is available for the > cluster. If I had LVM, I could simply add the new disks to the physical group, > and then grow the logical volume... I don't see how 7.0's recursive > partitioning can help me here Can you please explain ? Only ZFS can do that on 7. In truth, FreeBSD is really bad for storage works, and probably noone has ever done what you need (it's just not supported). The reasons are: - there are no distributed file systems for freebsd (meaning ones that have built-in support for operation on multiple nodes) - UFS panics on the slightest IO error and is not mountable by multiple hosts at the same time *at all*, even over gmirror - gmirror over ggate sort-of works but ggate network and IO errors will (at best) disconnect one of the drives and you'll need to manually reconnect them to the gmirror. After that you'll need to rebuild the whole drives (which is slow over the network). I.e. there's no automatic failover. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Storage cluster with FreeBSD
On Fri, 26 Oct 2007 13:51:36 +0200 (CEST) Wojciech Puchar <[EMAIL PROTECTED]> wrote: > > cool, thanks for the pointers... I had a side note to review these in > > conjunction with ggate. > So, is there any logical volume manager that works properly under 6.x or 7, > ala LVM2 in Linux? > > there is gvinum, but i can't say "it works right". it's quite easy to > crash, at least was for me. so i've read... > > but - freebsd allows 7 partitions, but partitions can be recurrent - i > mean you can use bsdlabel on partitions and get things like ad0aa etc. > > i used this to make 30 partitions. interesting... > > so it's "manual LVM" as then you can ggate, gmirror or gconcat or gstripe > with that partitions. sorry, i don't see how this recursive formatting of partitions can be used as a 'manual lvm'... see below > > precise more clearly what you need. as always - simple solutions are best > solutions. and unix == simplicity. indeed. Well, as I said in OP, similar to what Lustre offers. Let's see : what I am after is a way to hook up a few computers ( say, 6) with a few HD (say, 6 x 400 GB), and setup across the whole thing a RAID that would let me : - keep running with no loss of data in case of a disk or node failure (or minimal in case of a node failure) - can be accessed by clients over standard network protocols (NFS, CIFS, etc) - the clients see volumes of x GB (maybe 1 of the total size, maybe 3 of different sizes...doesn't matter) All this until hear seems doable with FBSD + GEOM architecture. Now the tricky ones : - scales linearly - if I add 90 storage hosts, the system doesnt get bogged down with the management of the disks / stripe distributoin - I can add new hosts (from 6 to 10) and the new storage is available for the cluster. If I had LVM, I could simply add the new disks to the physical group, and then grow the logical volume... I don't see how 7.0's recursive partitioning can help me here Can you please explain ? thanks! B _ {Beto|Norberto|Numard} Meijome "There are two kinds of stupid people. One kind says,'This is old and therefore good'. The other kind says, 'This is new, and therefore better.'" John Brunner, 'The Shockwave Rider'. I speak for myself, not my employer. Contents may be hot. Slippery when wet. Reading disclaimers makes you go blind. Writing them is worse. You have been Warned. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Storage cluster with FreeBSD
On Fri, 26 Oct 2007 10:27:16 +0200 (CEST) Wojciech Puchar <[EMAIL PROTECTED]> wrote: > > > > I have this pet project, playing with the Lustre Cluster FS. Is there > > anything similar to this that would run with FreeBSD as the host? ( Clients > > supported would HAVE to be linux,freebsd, Win32, OSX, ideally over standard > > protocols with no need for special driver). > > > > I seem to remember that ggate, in the GEOM stack, allows for storage over > > different nodes, but reading about ggate i am not sure it provides what I'm > > after : > > > > - storage over a number of nodes (few or large number). > > ggate provides remote volume. simply. Right. > > > - abstract view of the storage from the client's point of view ( 1 TB > > storage, doesnt matter how this is setup). > yes > > > - dynamic sizing : add servers, storage grows. > > no. you may have more remote/virtual volumes, > > > - resilience to node loss. > > > no. > > as every good unix tool, doesn't do everything, it does one thing just > right, remote volumes for ggate. > > use ggate with gmirror and gconcat :) cool, thanks for the pointers... I had a side note to review these in conjunction with ggate. So, is there any logical volume manager that works properly under 6.x or 7, ala LVM2 in Linux? ( I know of vinum, but the way I understand it is is pre-GEOM ). thanks Wojciech! :) B _ {Beto|Norberto|Numard} Meijome Contrary to popular belief, Unix is user friendly. It just happens to be very selective about who it decides to make friends with. I speak for myself, not my employer. Contents may be hot. Slippery when wet. Reading disclaimers makes you go blind. Writing them is worse. You have been Warned. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Storage cluster with FreeBSD
I have this pet project, playing with the Lustre Cluster FS. Is there anything similar to this that would run with FreeBSD as the host? ( Clients supported would HAVE to be linux,freebsd, Win32, OSX, ideally over standard protocols with no need for special driver). I seem to remember that ggate, in the GEOM stack, allows for storage over different nodes, but reading about ggate i am not sure it provides what I'm after : - storage over a number of nodes (few or large number). ggate provides remote volume. simply. - abstract view of the storage from the client's point of view ( 1 TB storage, doesnt matter how this is setup). yes - dynamic sizing : add servers, storage grows. no. you may have more remote/virtual volumes, - resilience to node loss. no. as every good unix tool, doesn't do everything, it does one thing just right, remote volumes for ggate. use ggate with gmirror and gconcat :) ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Storage cluster with FreeBSD
Hi everyone, I have this pet project, playing with the Lustre Cluster FS. Is there anything similar to this that would run with FreeBSD as the host? ( Clients supported would HAVE to be linux,freebsd, Win32, OSX, ideally over standard protocols with no need for special driver). I seem to remember that ggate, in the GEOM stack, allows for storage over different nodes, but reading about ggate i am not sure it provides what I'm after : - storage over a number of nodes (few or large number). - abstract view of the storage from the client's point of view ( 1 TB storage, doesnt matter how this is setup). - dynamic sizing : add servers, storage grows. - resilience to node loss. thanks!! B _ {Beto|Norberto|Numard} Meijome "The greatest dangers to liberty lurk in insidious encroachment by men of zeal, well-meaning but without understanding." Justice Louis D. Brandeis I speak for myself, not my employer. Contents may be hot. Slippery when wet. Reading disclaimers makes you go blind. Writing them is worse. You have been Warned. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"