Re: Storage cluster with FreeBSD

2007-10-27 Thread Ivan Voras
Norberto Meijome wrote:

 indeed. Well, as I said in OP, similar to what Lustre offers. 

I don't know what Lustre has so my response might or might not be what
you need...

 Let's see : what I am after is a way to hook up a few computers ( say, 6) with
 a few  HD (say,  6 x 400 GB), and setup across the whole thing a RAID that
 would let me :
  - keep running with no loss of data  in case of a disk or node failure (or
 minimal in case of a node failure)
 
  - can be accessed by clients over standard network protocols (NFS, CIFS, etc)
 
  - the clients see volumes of x GB (maybe 1 of the total size, maybe 3 of
 different sizes...doesn't matter)
 
 All this until hear seems doable with FBSD + GEOM architecture. Now the tricky
 ones :

May or may not. Do you want to have the drives attached to individual
machines, then exported via ggate, then mounted on each machine and then
RAID-ed together? This looks like it needs a large overhead in
administration. If you have the drives attached to individual machines
and then one machine that imports them all in some way (ggate) and
exports them via NFS and CIFS, then you have a single point of failure.

In truth, there appears to be only one volume manager that's close to
being usable, and that's ZFS in 7. But there's nothing automatic in the
way it can use the drives and (re) export them via NFS, etc.

There is no distributed file systems for FreeBSD.

  - scales linearly - if I add 90 storage hosts, the system doesnt get bogged
 down with the management of the disks / stripe distributoin

The only things you have with GEOM are low-level building blocks: RAID
transformations and network devices. If you calculate for yourself that,
(a silly example) if you create a RAID1 out of 90 hosts you can survive
the data being explicitely sent to each of the hosts individually, then
go ahead. There is no magic or smart behaviour involved anywhere.

  - I can add new hosts (from 6 to 10) and the new storage is available for the
 cluster. If I had LVM, I could simply add the new disks to the physical group,
 and then grow the logical volume... I don't see how 7.0's recursive
 partitioning can help me here Can you please explain ? 

Only ZFS can do that on 7.

In truth, FreeBSD is really bad for storage works, and probably noone
has ever done what you need (it's just not supported). The reasons are:

- there are no distributed file systems for freebsd (meaning ones that
have built-in support for operation on multiple nodes)
- UFS panics on the slightest IO error and is not mountable by multiple
hosts at the same time *at all*, even over gmirror
- gmirror over ggate sort-of works but ggate network and IO errors will
(at best) disconnect one of the drives and you'll need to manually
reconnect them to the gmirror. After that you'll need to rebuild the
whole drives (which is slow over the network). I.e. there's no automatic
failover.


___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Storage cluster with FreeBSD

2007-10-26 Thread Wojciech Puchar


I have this pet project, playing with the Lustre Cluster FS. Is there anything 
similar to this that would run with FreeBSD as the host? ( Clients supported 
would HAVE to be linux,freebsd, Win32, OSX, ideally over standard protocols 
with no need for special driver).

I seem to remember that ggate,  in the GEOM stack, allows for storage over 
different nodes, but reading about ggate i am not sure it provides what I'm 
after :

- storage over a number of nodes (few or large number).


ggate provides remote volume. simply.


- abstract view of the storage from the client's point of view ( 1 TB storage, 
doesnt matter how this is setup).

yes


- dynamic sizing : add servers, storage grows.


no. you may have more remote/virtual volumes,


- resilience to node loss.


no.

as every good unix tool, doesn't do everything, it does one thing just 
right, remote volumes for ggate.


use ggate with gmirror and gconcat :)
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Storage cluster with FreeBSD

2007-10-26 Thread Norberto Meijome
On Fri, 26 Oct 2007 10:27:16 +0200 (CEST)
Wojciech Puchar [EMAIL PROTECTED] wrote:

 
  I have this pet project, playing with the Lustre Cluster FS. Is there 
  anything similar to this that would run with FreeBSD as the host? ( Clients 
  supported would HAVE to be linux,freebsd, Win32, OSX, ideally over standard 
  protocols with no need for special driver).
 
  I seem to remember that ggate,  in the GEOM stack, allows for storage over 
  different nodes, but reading about ggate i am not sure it provides what I'm 
  after :
 
  - storage over a number of nodes (few or large number).
 
 ggate provides remote volume. simply.

Right.

 
  - abstract view of the storage from the client's point of view ( 1 TB 
  storage, doesnt matter how this is setup).
 yes
 
  - dynamic sizing : add servers, storage grows.
 
 no. you may have more remote/virtual volumes,
 
  - resilience to node loss.
 
 no.
 
 as every good unix tool, doesn't do everything, it does one thing just 
 right, remote volumes for ggate.
 
 use ggate with gmirror and gconcat :)

cool, thanks for the pointers... I had a side note to review these in 
conjunction with ggate. So, is there any logical volume manager that works 
properly under 6.x or 7, ala LVM2 in Linux? ( I know of vinum, but the way I 
understand it is is pre-GEOM ).

thanks Wojciech! :)
B

_
{Beto|Norberto|Numard} Meijome

Contrary to popular belief, Unix is user friendly. It just happens to be very 
selective about who it decides to make friends with.

I speak for myself, not my employer. Contents may be hot. Slippery when wet. 
Reading disclaimers makes you go blind. Writing them is worse. You have been 
Warned.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Storage cluster with FreeBSD

2007-10-26 Thread Norberto Meijome
On Fri, 26 Oct 2007 13:51:36 +0200 (CEST)
Wojciech Puchar [EMAIL PROTECTED] wrote:

  cool, thanks for the pointers... I had a side note to review these in 
  conjunction with ggate.  
 So, is there any logical volume manager that works properly under 6.x or 7, 
 ala LVM2 in Linux?
 
 there is gvinum, but i can't say it works right. it's quite easy to 
 crash, at least was for me.

so i've read...

 
 but - freebsd allows 7 partitions, but partitions can be recurrent - i 
 mean you can use bsdlabel on partitions and get things like ad0aa etc.
 
 i used this to make 30 partitions.

interesting...

 
 so it's manual LVM as then you can ggate, gmirror or gconcat or gstripe 
 with that partitions.

sorry, i don't see how this recursive formatting of partitions can be used as a
'manual lvm'... see below

 
 precise more clearly what you need. as always - simple solutions are best 
 solutions. and unix == simplicity.

indeed. Well, as I said in OP, similar to what Lustre offers. 

Let's see : what I am after is a way to hook up a few computers ( say, 6) with
a few  HD (say,  6 x 400 GB), and setup across the whole thing a RAID that
would let me :
 - keep running with no loss of data  in case of a disk or node failure (or
minimal in case of a node failure)

 - can be accessed by clients over standard network protocols (NFS, CIFS, etc)

 - the clients see volumes of x GB (maybe 1 of the total size, maybe 3 of
different sizes...doesn't matter)

All this until hear seems doable with FBSD + GEOM architecture. Now the tricky
ones :

 - scales linearly - if I add 90 storage hosts, the system doesnt get bogged
down with the management of the disks / stripe distributoin

 - I can add new hosts (from 6 to 10) and the new storage is available for the
cluster. If I had LVM, I could simply add the new disks to the physical group,
and then grow the logical volume... I don't see how 7.0's recursive
partitioning can help me here Can you please explain ? 

thanks!
B
_
{Beto|Norberto|Numard} Meijome

There are two kinds of stupid people. One kind says,'This is old and therefore
good'. The other kind says, 'This is new, and therefore better.'
 John Brunner, 'The Shockwave Rider'.

I speak for myself, not my employer. Contents may be hot. Slippery when wet.
Reading disclaimers makes you go blind. Writing them is worse. You have been
Warned.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Storage cluster with FreeBSD

2007-10-25 Thread Norberto Meijome
Hi everyone,

I have this pet project, playing with the Lustre Cluster FS. Is there anything 
similar to this that would run with FreeBSD as the host? ( Clients supported 
would HAVE to be linux,freebsd, Win32, OSX, ideally over standard protocols 
with no need for special driver).

I seem to remember that ggate,  in the GEOM stack, allows for storage over 
different nodes, but reading about ggate i am not sure it provides what I'm 
after : 

 - storage over a number of nodes (few or large number).
 - abstract view of the storage from the client's point of view ( 1 TB storage, 
doesnt matter how this is setup).
 - dynamic sizing : add servers, storage grows.
 - resilience to node loss.

thanks!!
B
_
{Beto|Norberto|Numard} Meijome

The greatest dangers to liberty lurk in insidious encroachment by men of zeal, 
well-meaning but without understanding.
   Justice Louis D. Brandeis

I speak for myself, not my employer. Contents may be hot. Slippery when wet. 
Reading disclaimers makes you go blind. Writing them is worse. You have been 
Warned.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]