[SLUG] Network Real-Time Hot Filesystem Replication?
I've just spent some time quickly researching this to no real satisfaction. What I'm looking for is a way to do real-time hot-replication of a whole filesystem or filesystem tree over 2 nodes (and strictly 2 nodes) without STOMITH[1]. The scenario is I have two identical systems with local (software) RAID1. They will be tethered onto their internet feed via ethernet, and can optionally be tethered to each other via Gig. I want to be able to set it up so /home (and maybe other filesystems) are replicated from one to the other, in both directions, in real time so they can run in an all-hot redundant cluster. The environment should be mostly read-oriented, so I can live with write-latent solutions as long as they handle the race/collision gracefully (preferably by actually detecting and reporting it if they can't avoid it). The options I've investigated so far: * Lustre (MDS requirements[2] make this not an option) * GlobalFS (STOMITH requirements make this not an option. Oriented towards shared media too, which I am not using) * tsync (Naive concurrent operation model, but otherwise viable) * MogileFS (not quite what I was looking for, but none the less useful). * OpenAFS (read-only replication only, loss of the node hosting the write volume still renders the volume unwritable). Is anybody aware of any other options that I've missed? C. [1] Shoot The Other Machine In The Head - the ability for any node to forcibly powerdown any other node believed to be malfunctioning. [2] Single instance MDS only, only clusterable through shared storage. d'oh. [3] People suggesting rsync will be taken out back and shot for not reading the requirements. -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Network Real-Time Hot Filesystem Replication?
On Sat, 05 Apr 2008 09:52:55 +1100 Crossfire [EMAIL PROTECTED] wrote: I've just spent some time quickly researching this to no real satisfaction. What I'm looking for is a way to do real-time hot-replication of a whole filesystem or filesystem tree over 2 nodes (and strictly 2 nodes) without STOMITH[1]. The scenario is I have two identical systems with local (software) RAID1. They will be tethered onto their internet feed via ethernet, and can optionally be tethered to each other via Gig. Have you had a look at http://www.drbd.org/ ? It basically mirrors a blockdevice over ethernet. A raid1 of sorts. -- Regards Mick Pollard ( lunix ) BOFH Excuse of the day: Intermittant Checksum Invalidation Error pgpPPkdV5wNbV.pgp Description: PGP signature -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Network Real-Time Hot Filesystem Replication?
Crossfire wrote: I want to be able to set it up so /home (and maybe other filesystems) are replicated from one to the other, in both directions, in real time so they can run in an all-hot redundant cluster. The environment should be mostly read-oriented, so I can live with write-latent solutions as long as they handle the race/collision gracefully (preferably by actually detecting and reporting it if they can't avoid it). isn't this just a description of a network filesytem... say NFS? I am also interested in what you come up with, but haven't seen anything that matchs. DRBD is not RW from both nodes. I have also used RAID1 over AoE and iSCSI, but not sure if this would help you at all either with only two nodes. I was thinking just yesterday some sort of fuse filesystem is what we need :) dave -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Network Real-Time Hot Filesystem Replication?
Mick Pollard wrote: On Sat, 05 Apr 2008 09:52:55 +1100 Crossfire [EMAIL PROTECTED] wrote: I've just spent some time quickly researching this to no real satisfaction. What I'm looking for is a way to do real-time hot-replication of a whole filesystem or filesystem tree over 2 nodes (and strictly 2 nodes) without STOMITH[1]. The scenario is I have two identical systems with local (software) RAID1. They will be tethered onto their internet feed via ethernet, and can optionally be tethered to each other via Gig. Have you had a look at http://www.drbd.org/ ? It basically mirrors a blockdevice over ethernet. A raid1 of sorts. DRBD doesn't actually solve the problem - it either provides a warm replication of a normal filesystem, or provides an pseudo-shared storage device. Warm replication is a no-go since I will need effectively local access to the filesystem on both nodes so there isn't a song-and-dance routine to perform w.r.t mounts, etc, during a failure. As a psuedo-shared storage device, I doubt it's of any particular use due to (drastically-higher) latency incurred by having to replicate the changes between the storage pools rather than the storage pool being shared. I'd also be concerned about split-brain recovery with cluster filesystems (split-brain is not actually possible with a real shared drive, but completely possible with DRBD). Initial comments I've seen from people trying to use it with OCFS2 also seem poor. C. -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Network Real-Time Hot Filesystem Replication?
Dave Kempe wrote: I was thinking just yesterday some sort of fuse filesystem is what we need :) dave haven't tried it, but this is fuse http://www.furquim.org/chironfs/ dave -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Network Real-Time Hot Filesystem Replication?
Dave Kempe wrote: Crossfire wrote: I want to be able to set it up so /home (and maybe other filesystems) are replicated from one to the other, in both directions, in real time so they can run in an all-hot redundant cluster. The environment should be mostly read-oriented, so I can live with write-latent solutions as long as they handle the race/collision gracefully (preferably by actually detecting and reporting it if they can't avoid it). isn't this just a description of a network filesytem... say NFS? No. Network Filesystems still have a distinct single storage location. If that storage is taken offline, clients can only error or hang. With a hot real-time replicated filesystem, all involved nodes would have a full local copy at all times and would be able to continue operation. C. -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Network Real-Time Hot Filesystem Replication?
On Sat, Apr 05, 2008 at 09:52:55AM +1100, Crossfire wrote: I've just spent some time quickly researching this to no real satisfaction. What I'm looking for is a way to do real-time hot-replication of a whole filesystem or filesystem tree over 2 nodes (and strictly 2 nodes) without STOMITH[1]. The scenario is I have two identical systems with local (software) RAID1. They will be tethered onto their internet feed via ethernet, and can optionally be tethered to each other via Gig. I want to be able to set it up so /home (and maybe other filesystems) are replicated from one to the other, in both directions, in real time so they can run in an all-hot redundant cluster. The environment should be mostly read-oriented, so I can live with write-latent solutions as long as they handle the race/collision gracefully (preferably by actually detecting and reporting it if they can't avoid it). The options I've investigated so far: * Lustre (MDS requirements[2] make this not an option) * GlobalFS (STOMITH requirements make this not an option. Oriented towards shared media too, which I am not using) * tsync (Naive concurrent operation model, but otherwise viable) * MogileFS (not quite what I was looking for, but none the less useful). * OpenAFS (read-only replication only, loss of the node hosting the write volume still renders the volume unwritable). Is anybody aware of any other options that I've missed? I think once you ask for no STOMITH and also read/write access from more than one location, you have made it really hard for yourself. Unless you go to something NFS. Lustre doesn't really met you requirements. its more for file striping and speed then for HA if 1 can be readonly then you could setup a raid 1 with one device being a network block device C. [1] Shoot The Other Machine In The Head - the ability for any node to forcibly powerdown any other node believed to be malfunctioning. [2] Single instance MDS only, only clusterable through shared storage. d'oh. [3] People suggesting rsync will be taken out back and shot for not reading the requirements. -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html -- I also understand how tender the free enterprise system can be. - George W. Bush 07/08/2002 White House press conference, Washington, D.C. signature.asc Description: Digital signature -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Network Real-Time Hot Filesystem Replication?
On Sat, Apr 05, 2008 at 10:11:00AM +1100, John Ferlito wrote: Keeping in mind I've never done this so no idea how well it works. I'd say a combination of Global File System - http://sources.redhat.com/cluster/gfs/ I think the requirements where for no STOMITH and GFS uses that in both incarnations, so does ocfs, psfs and Global Network Block Device - http://sourceware.org/cluster/gnbd/ should do the trick, this document explains how http://sources.redhat.com/cluster/wiki/DRBD_Cookbook?highlight=(CategoryHowTo) On Sat, Apr 05, 2008 at 09:52:55AM +1100, Crossfire wrote: I've just spent some time quickly researching this to no real satisfaction. What I'm looking for is a way to do real-time hot-replication of a whole filesystem or filesystem tree over 2 nodes (and strictly 2 nodes) without STOMITH[1]. The scenario is I have two identical systems with local (software) RAID1. They will be tethered onto their internet feed via ethernet, and can optionally be tethered to each other via Gig. I want to be able to set it up so /home (and maybe other filesystems) are replicated from one to the other, in both directions, in real time so they can run in an all-hot redundant cluster. The environment should be mostly read-oriented, so I can live with write-latent solutions as long as they handle the race/collision gracefully (preferably by actually detecting and reporting it if they can't avoid it). The options I've investigated so far: * Lustre (MDS requirements[2] make this not an option) * GlobalFS (STOMITH requirements make this not an option. Oriented towards shared media too, which I am not using) * tsync (Naive concurrent operation model, but otherwise viable) * MogileFS (not quite what I was looking for, but none the less useful). * OpenAFS (read-only replication only, loss of the node hosting the write volume still renders the volume unwritable). Is anybody aware of any other options that I've missed? C. [1] Shoot The Other Machine In The Head - the ability for any node to forcibly powerdown any other node believed to be malfunctioning. [2] Single instance MDS only, only clusterable through shared storage. d'oh. [3] People suggesting rsync will be taken out back and shot for not reading the requirements. -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html -- John http://www.inodes.org/ -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html -- It is clear our nation is reliant upon big foreign oil. More and more of our imports come from overseas. - George W. Bush 09/25/2000 Beaverton, OR signature.asc Description: Digital signature -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
