Re: [SLUG] Network Real-Time Hot Filesystem Replication?
Crossfire wrote: I want to be able to set it up so /home (and maybe other filesystems) are replicated from one to the other, in both directions, in real time so they can run in an all-hot redundant cluster. I think I found something that might help you: http://gluster.org/docs/index.php/GlusterFS http://gluster.org/docs/index.php/GlusterFS_Translators_v1.3#Automatic_File_Replication_Translator_.28AFR.29 I haven't tried it yet, but it looks good, so I might. sorry to revive an old thread :) dave -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Network Real-Time Hot Filesystem Replication?
Crossfire wrote: I've just spent some time quickly researching this to no real satisfaction. What I'm looking for is a way to do real-time hot-replication of a whole filesystem or filesystem tree over 2 nodes (and strictly 2 nodes) without STOMITH[1]. The scenario is I have two identical systems with local (software) RAID1. They will be tethered onto their internet feed via ethernet, and can optionally be tethered to each other via Gig. I want to be able to set it up so /home (and maybe other filesystems) are replicated from one to the other, in both directions, in real time so they can run in an all-hot redundant cluster. The environment should be mostly read-oriented, so I can live with write-latent solutions as long as they handle the race/collision gracefully (preferably by actually detecting and reporting it if they can't avoid it). I've had some success with Software iSCSI targets on Linux to date. I'm currently using software iSCSI over Gigabit Ethernet to back a VMware ESX cluster[0]. Software iSCSI targets (I have experience only with tgtd - the only one that seemed current) present a Linux block device as an iSCSI target over the network. I present an LVM logical volume. One could conceive of an eventuality where you made both machines iSCSI targets and initiators and ran RAID1 over the local and remote iSCSI targets[1]. I have no idea what sort of (terrible) performance you might get out of this sort of setup, but it would meet your requirements, and with enough RAM for read-caching in each node, it might not be too bad. You would need that Gigabit cross-connect. There are large warnings in the scsi_tgt code about using it in production, however. I suspect this problem space isn't addressed terribly often because, well, (1) it's Hard, (2) most people who care about this stuff buy shared storage (check ebay), (3) It's even Harder once you start talking file systems that do this[2]. Cheers, Matt 0. I can post my recipe for the target bits, if anyone cares. 1. With a global filesystem, of course. 2. Ceph, which Robert Collins suggested above, is a really good example of a brilliantly designed distributed file system (much better than MogileFS, which is more an on-disk hash table with quirks), but I have my doubts about it's suitability for production systems (though I hope it gets there). -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Network Real-Time Hot Filesystem Replication?
I don't know whether it would suit you at all, but I'll mention http://allmydata.org/trac/tahoe for the simple reason it looks interesting and it only just announced version 1.0 RC's mention of Ceph jogged my memory on this. Matt -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Network Real-Time Hot Filesystem Replication?
This one time, at band camp, Amos Shapira wrote: 1. You CAN'T mount a non-cluster-aware file system even read-only on the secondary node since the primary will change FS-structs under the feet of the read-only node and cause it to crash (because non-cluster-aware filesystems assume that they are the only ones who touch that partition). 2. You CAN mount read-write on multiple nodes if you use one of the cluster-aware filesystems (GFS and OCFS are regularly mentioned, but if you find any other cluster-aware file system then it sounds like it will work too). You're right, the example I was thinking of does not mount the filesystem on the secondary nodes until the primary goes down; once the FS is not mounted one of the secondaries takes over and mounts it read/write. -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Network Real-Time Hot Filesystem Replication?
Adrian Chadd wrote: I looked into it about a year ago and I couldn't find any simple way of doing this using free software. There's CODA/AFS as possible solutions but they still push the notion of master/slave rather than equal peers, which Chris mentions he needs (ie, constant synchronisation between each member rather than periodic pushback..) Chris, try looking at CODA/AFS support? OpenAFS was already considered. R/O replication is a pain, as is the whole volume host death problem. (ie: write volume goes away if the host holding the volume dies). I haven't looked at Coda recently. They still seem to be active (I thought they'd all abandoned ship for Intermezzo - seems I was wrong). I'll check it out sometime soon. C. -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Network Real-Time Hot Filesystem Replication?
On Sat, 2008-04-05 at 09:52 +1100, Crossfire wrote: I've just spent some time quickly researching this to no real satisfaction. What I'm looking for is a way to do real-time hot-replication of a whole filesystem or filesystem tree over 2 nodes (and strictly 2 nodes) without STOMITH[1]. The scenario is I have two identical systems with local (software) RAID1. They will be tethered onto their internet feed via ethernet, and can optionally be tethered to each other via Gig. I want to be able to set it up so /home (and maybe other filesystems) are replicated from one to the other, in both directions, in real time so they can run in an all-hot redundant cluster. The environment should be mostly read-oriented, so I can live with write-latent solutions as long as they handle the race/collision gracefully (preferably by actually detecting and reporting it if they can't avoid it). The options I've investigated so far: * Lustre (MDS requirements[2] make this not an option) * GlobalFS (STOMITH requirements make this not an option. Oriented towards shared media too, which I am not using) * tsync (Naive concurrent operation model, but otherwise viable) * MogileFS (not quite what I was looking for, but none the less useful). * OpenAFS (read-only replication only, loss of the node hosting the write volume still renders the volume unwritable). Is anybody aware of any other options that I've missed? http://sourceforge.net/projects/ceph/ -Rob -- GPG key available at: http://www.robertcollins.net/keys.txt. signature.asc Description: This is a digitally signed message part -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Network Real-Time Hot Filesystem Replication?
On Sun, Apr 6, 2008 at 2:47 PM, Jamie Wilkinson [EMAIL PROTECTED] wrote: This one time, at band camp, Crossfire wrote: Dave Kempe wrote: Crossfire wrote: I want to be able to set it up so /home (and maybe other filesystems) are replicated from one to the other, in both directions, in real time so they can run in an all-hot redundant cluster. The environment should be mostly read-oriented, so I can live with write-latent solutions as long as they handle the race/collision gracefully (preferably by actually detecting and reporting it if they can't avoid it). isn't this just a description of a network filesytem... say NFS? No. Network Filesystems still have a distinct single storage location. If that storage is taken offline, clients can only error or hang. With a hot real-time replicated filesystem, all involved nodes would have a full local copy at all times and would be able to continue operation. I agreed with your earlier decision about not using drbd because you wouldn't be able to write from multiple nodes to the filesystem; all the slaves would have to be mounted read-only. However if you wanted to get Can you provide links which support this? I've been using DRBD for a few months now (just in stand-by mode, but been following the forums and docs during that time) and all indications are that: 1. You CAN'T mount a non-cluster-aware file system even read-only on the secondary node since the primary will change FS-structs under the feet of the read-only node and cause it to crash (because non-cluster-aware filesystems assume that they are the only ones who touch that partition). 2. You CAN mount read-write on multiple nodes if you use one of the cluster-aware filesystems (GFS and OCFS are regularly mentioned, but if you find any other cluster-aware file system then it sounds like it will work too). Ref: http://www.linux-ha.org/DRBD/FAQ#head-2cad8caa095cfb6e2935261cb595390c742ebd86 fancy you could still use drbd (which is a great fit for all your other requirements) on a multi-node fileserver, and do some nifty failover using IP takeover. Or if you're trying to share the local disk of a lot of nodes, then what if you used DRBD on them all to replicate the block device, and run a NFS server on the nodes thremselves? Yes you'd get a lot of network traffic between them, but it'd work, no? :) Have you tried this suggestions? From all I read about DRBD this will cause all secondary nodes to crash. Cheers, --Amos -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Network Real-Time Hot Filesystem Replication?
On Sun, Apr 06, 2008, Amos Shapira wrote: I've been using DRBD for a few months now (just in stand-by mode, but been following the forums and docs during that time) and all indications are that: 1. You CAN'T mount a non-cluster-aware file system even read-only on the secondary node since the primary will change FS-structs under the feet of the read-only node and cause it to crash (because non-cluster-aware filesystems assume that they are the only ones who touch that partition). 2. You CAN mount read-write on multiple nodes if you use one of the cluster-aware filesystems (GFS and OCFS are regularly mentioned, but if you find any other cluster-aware file system then it sounds like it will work too). IIRC they assume a single back-end device. Does DRBD give you a journaling block device which will stall updates until they've been pushed? How will the FSes tolerate the device IO being possibly milliseconds later than the master? Have you tried this suggestions? From all I read about DRBD this will cause all secondary nodes to crash. I looked into it about a year ago and I couldn't find any simple way of doing this using free software. There's CODA/AFS as possible solutions but they still push the notion of master/slave rather than equal peers, which Chris mentions he needs (ie, constant synchronisation between each member rather than periodic pushback..) Chris, try looking at CODA/AFS support? Adrian -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Network Real-Time Hot Filesystem Replication?
On Sun, Apr 6, 2008 at 9:25 PM, Adrian Chadd [EMAIL PROTECTED] wrote: On Sun, Apr 06, 2008, Amos Shapira wrote: I've been using DRBD for a few months now (just in stand-by mode, but been following the forums and docs during that time) and all indications are that: 1. You CAN'T mount a non-cluster-aware file system even read-only on the secondary node since the primary will change FS-structs under the feet of the read-only node and cause it to crash (because non-cluster-aware filesystems assume that they are the only ones who touch that partition). 2. You CAN mount read-write on multiple nodes if you use one of the cluster-aware filesystems (GFS and OCFS are regularly mentioned, but if you find any other cluster-aware file system then it sounds like it will work too). IIRC they assume a single back-end device. Does DRBD give you a journaling block device which will stall updates until they've been pushed? How will the FSes tolerate the device IO being possibly milliseconds later than the master? Again - I haven't got around to actually use it (as much as I'd like to just sit down and try it) but you can see in the link that I sent with my previous reply that they clearly claim that it is supported. Have you tried this suggestions? From all I read about DRBD this will cause all secondary nodes to crash. I looked into it about a year ago and I couldn't find any simple way of Could it be that you looked at 0.7? I always used 0.8+ and got the impression that there were major improvements introduced in it over 0.7. doing this using free software. There's CODA/AFS as possible solutions but they still push the notion of master/slave rather than equal peers, which Chris mentions he needs (ie, constant synchronisation between each member rather than periodic pushback..) That's what DRBD 0.8+GFS/OCFS is promoted as . Cheers, --Amos -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Network Real-Time Hot Filesystem Replication?
This one time, at band camp, Crossfire wrote: Dave Kempe wrote: Crossfire wrote: I want to be able to set it up so /home (and maybe other filesystems) are replicated from one to the other, in both directions, in real time so they can run in an all-hot redundant cluster. The environment should be mostly read-oriented, so I can live with write-latent solutions as long as they handle the race/collision gracefully (preferably by actually detecting and reporting it if they can't avoid it). isn't this just a description of a network filesytem... say NFS? No. Network Filesystems still have a distinct single storage location. If that storage is taken offline, clients can only error or hang. With a hot real-time replicated filesystem, all involved nodes would have a full local copy at all times and would be able to continue operation. I agreed with your earlier decision about not using drbd because you wouldn't be able to write from multiple nodes to the filesystem; all the slaves would have to be mounted read-only. However if you wanted to get fancy you could still use drbd (which is a great fit for all your other requirements) on a multi-node fileserver, and do some nifty failover using IP takeover. Or if you're trying to share the local disk of a lot of nodes, then what if you used DRBD on them all to replicate the block device, and run a NFS server on the nodes thremselves? Yes you'd get a lot of network traffic between them, but it'd work, no? :) -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
[SLUG] Network Real-Time Hot Filesystem Replication?
I've just spent some time quickly researching this to no real satisfaction. What I'm looking for is a way to do real-time hot-replication of a whole filesystem or filesystem tree over 2 nodes (and strictly 2 nodes) without STOMITH[1]. The scenario is I have two identical systems with local (software) RAID1. They will be tethered onto their internet feed via ethernet, and can optionally be tethered to each other via Gig. I want to be able to set it up so /home (and maybe other filesystems) are replicated from one to the other, in both directions, in real time so they can run in an all-hot redundant cluster. The environment should be mostly read-oriented, so I can live with write-latent solutions as long as they handle the race/collision gracefully (preferably by actually detecting and reporting it if they can't avoid it). The options I've investigated so far: * Lustre (MDS requirements[2] make this not an option) * GlobalFS (STOMITH requirements make this not an option. Oriented towards shared media too, which I am not using) * tsync (Naive concurrent operation model, but otherwise viable) * MogileFS (not quite what I was looking for, but none the less useful). * OpenAFS (read-only replication only, loss of the node hosting the write volume still renders the volume unwritable). Is anybody aware of any other options that I've missed? C. [1] Shoot The Other Machine In The Head - the ability for any node to forcibly powerdown any other node believed to be malfunctioning. [2] Single instance MDS only, only clusterable through shared storage. d'oh. [3] People suggesting rsync will be taken out back and shot for not reading the requirements. -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Network Real-Time Hot Filesystem Replication?
On Sat, 05 Apr 2008 09:52:55 +1100 Crossfire [EMAIL PROTECTED] wrote: I've just spent some time quickly researching this to no real satisfaction. What I'm looking for is a way to do real-time hot-replication of a whole filesystem or filesystem tree over 2 nodes (and strictly 2 nodes) without STOMITH[1]. The scenario is I have two identical systems with local (software) RAID1. They will be tethered onto their internet feed via ethernet, and can optionally be tethered to each other via Gig. Have you had a look at http://www.drbd.org/ ? It basically mirrors a blockdevice over ethernet. A raid1 of sorts. -- Regards Mick Pollard ( lunix ) BOFH Excuse of the day: Intermittant Checksum Invalidation Error pgpPPkdV5wNbV.pgp Description: PGP signature -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Network Real-Time Hot Filesystem Replication?
Crossfire wrote: I want to be able to set it up so /home (and maybe other filesystems) are replicated from one to the other, in both directions, in real time so they can run in an all-hot redundant cluster. The environment should be mostly read-oriented, so I can live with write-latent solutions as long as they handle the race/collision gracefully (preferably by actually detecting and reporting it if they can't avoid it). isn't this just a description of a network filesytem... say NFS? I am also interested in what you come up with, but haven't seen anything that matchs. DRBD is not RW from both nodes. I have also used RAID1 over AoE and iSCSI, but not sure if this would help you at all either with only two nodes. I was thinking just yesterday some sort of fuse filesystem is what we need :) dave -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Network Real-Time Hot Filesystem Replication?
Mick Pollard wrote: On Sat, 05 Apr 2008 09:52:55 +1100 Crossfire [EMAIL PROTECTED] wrote: I've just spent some time quickly researching this to no real satisfaction. What I'm looking for is a way to do real-time hot-replication of a whole filesystem or filesystem tree over 2 nodes (and strictly 2 nodes) without STOMITH[1]. The scenario is I have two identical systems with local (software) RAID1. They will be tethered onto their internet feed via ethernet, and can optionally be tethered to each other via Gig. Have you had a look at http://www.drbd.org/ ? It basically mirrors a blockdevice over ethernet. A raid1 of sorts. DRBD doesn't actually solve the problem - it either provides a warm replication of a normal filesystem, or provides an pseudo-shared storage device. Warm replication is a no-go since I will need effectively local access to the filesystem on both nodes so there isn't a song-and-dance routine to perform w.r.t mounts, etc, during a failure. As a psuedo-shared storage device, I doubt it's of any particular use due to (drastically-higher) latency incurred by having to replicate the changes between the storage pools rather than the storage pool being shared. I'd also be concerned about split-brain recovery with cluster filesystems (split-brain is not actually possible with a real shared drive, but completely possible with DRBD). Initial comments I've seen from people trying to use it with OCFS2 also seem poor. C. -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Network Real-Time Hot Filesystem Replication?
Dave Kempe wrote: I was thinking just yesterday some sort of fuse filesystem is what we need :) dave haven't tried it, but this is fuse http://www.furquim.org/chironfs/ dave -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Network Real-Time Hot Filesystem Replication?
Dave Kempe wrote: Crossfire wrote: I want to be able to set it up so /home (and maybe other filesystems) are replicated from one to the other, in both directions, in real time so they can run in an all-hot redundant cluster. The environment should be mostly read-oriented, so I can live with write-latent solutions as long as they handle the race/collision gracefully (preferably by actually detecting and reporting it if they can't avoid it). isn't this just a description of a network filesytem... say NFS? No. Network Filesystems still have a distinct single storage location. If that storage is taken offline, clients can only error or hang. With a hot real-time replicated filesystem, all involved nodes would have a full local copy at all times and would be able to continue operation. C. -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Network Real-Time Hot Filesystem Replication?
On Sat, Apr 05, 2008 at 09:52:55AM +1100, Crossfire wrote: I've just spent some time quickly researching this to no real satisfaction. What I'm looking for is a way to do real-time hot-replication of a whole filesystem or filesystem tree over 2 nodes (and strictly 2 nodes) without STOMITH[1]. The scenario is I have two identical systems with local (software) RAID1. They will be tethered onto their internet feed via ethernet, and can optionally be tethered to each other via Gig. I want to be able to set it up so /home (and maybe other filesystems) are replicated from one to the other, in both directions, in real time so they can run in an all-hot redundant cluster. The environment should be mostly read-oriented, so I can live with write-latent solutions as long as they handle the race/collision gracefully (preferably by actually detecting and reporting it if they can't avoid it). The options I've investigated so far: * Lustre (MDS requirements[2] make this not an option) * GlobalFS (STOMITH requirements make this not an option. Oriented towards shared media too, which I am not using) * tsync (Naive concurrent operation model, but otherwise viable) * MogileFS (not quite what I was looking for, but none the less useful). * OpenAFS (read-only replication only, loss of the node hosting the write volume still renders the volume unwritable). Is anybody aware of any other options that I've missed? I think once you ask for no STOMITH and also read/write access from more than one location, you have made it really hard for yourself. Unless you go to something NFS. Lustre doesn't really met you requirements. its more for file striping and speed then for HA if 1 can be readonly then you could setup a raid 1 with one device being a network block device C. [1] Shoot The Other Machine In The Head - the ability for any node to forcibly powerdown any other node believed to be malfunctioning. [2] Single instance MDS only, only clusterable through shared storage. d'oh. [3] People suggesting rsync will be taken out back and shot for not reading the requirements. -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html -- I also understand how tender the free enterprise system can be. - George W. Bush 07/08/2002 White House press conference, Washington, D.C. signature.asc Description: Digital signature -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Network Real-Time Hot Filesystem Replication?
On Sat, Apr 05, 2008 at 10:11:00AM +1100, John Ferlito wrote: Keeping in mind I've never done this so no idea how well it works. I'd say a combination of Global File System - http://sources.redhat.com/cluster/gfs/ I think the requirements where for no STOMITH and GFS uses that in both incarnations, so does ocfs, psfs and Global Network Block Device - http://sourceware.org/cluster/gnbd/ should do the trick, this document explains how http://sources.redhat.com/cluster/wiki/DRBD_Cookbook?highlight=(CategoryHowTo) On Sat, Apr 05, 2008 at 09:52:55AM +1100, Crossfire wrote: I've just spent some time quickly researching this to no real satisfaction. What I'm looking for is a way to do real-time hot-replication of a whole filesystem or filesystem tree over 2 nodes (and strictly 2 nodes) without STOMITH[1]. The scenario is I have two identical systems with local (software) RAID1. They will be tethered onto their internet feed via ethernet, and can optionally be tethered to each other via Gig. I want to be able to set it up so /home (and maybe other filesystems) are replicated from one to the other, in both directions, in real time so they can run in an all-hot redundant cluster. The environment should be mostly read-oriented, so I can live with write-latent solutions as long as they handle the race/collision gracefully (preferably by actually detecting and reporting it if they can't avoid it). The options I've investigated so far: * Lustre (MDS requirements[2] make this not an option) * GlobalFS (STOMITH requirements make this not an option. Oriented towards shared media too, which I am not using) * tsync (Naive concurrent operation model, but otherwise viable) * MogileFS (not quite what I was looking for, but none the less useful). * OpenAFS (read-only replication only, loss of the node hosting the write volume still renders the volume unwritable). Is anybody aware of any other options that I've missed? C. [1] Shoot The Other Machine In The Head - the ability for any node to forcibly powerdown any other node believed to be malfunctioning. [2] Single instance MDS only, only clusterable through shared storage. d'oh. [3] People suggesting rsync will be taken out back and shot for not reading the requirements. -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html -- John http://www.inodes.org/ -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html -- It is clear our nation is reliant upon big foreign oil. More and more of our imports come from overseas. - George W. Bush 09/25/2000 Beaverton, OR signature.asc Description: Digital signature -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
