Re: [Dorset] Hot Backup Mode
On Tuesday 22 Jul 2014 19:04:03 Ralph Corderoy wrote: No, you mean like a hot spare for failover when the primary goes kaput, or even better, a cluster (of two) fileservers that are in sync and share the load, read and write. This is all still uncertain. It now appears that the current system has two servers, each running the same software and receiving the same data streams, but connected to different networks. The clients are split between the two, so (presumably) there are two clients dealing with each subset of data. Lots need to be found out here. Do you have to fit in with the existing protocol from the clients, i.e. you're replacing just the servers? Do the clients know which of the two servers to contact and learn when they should switch, or is that transparent to them? Do both servers serve requests, or just the primary? Perhaps the primary handles read and write but the secondary just does reads. We have to replace everything. Are you aware of the Linux HA project, high availability. That's the term often used to describe failover systems. There's lots out there on it. If just two servers are needed then a shared hardware solution is possible. Or pure software solutions on commodity hardware can be done. It's a big area! Examples, http://www.dell.com/content/topics/global.aspx/power/en/ps4q00_linux?c=us https://en.wikipedia.org/wiki/Lustre_%28file_system%29 Thanks for the links. We are trying to find how the current system does it. I think that's key. It may well be, especially if it's a bit old, that it's more simplistic than its grand description and that would lessen what you need to do. See above. However, there is uncertainty whether the new requirement is more onerous than the existing one. We hope to find out more by Monday. -- Terry Coles -- Next meeting: Bournemouth, Tuesday, 2014-08-05 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread on mailing list: mailto:dorset@mailman.lug.org.uk How to Report Bugs Effectively: http://goo.gl/4Xue
[Dorset] Hot Backup Mode
Can anyone explain hot backup mode in the context of file server as opposed to a database? Can it be done? Terry Coles -- Next meeting: Bournemouth, Tuesday, 2014-08-05 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread on mailing list: mailto:dorset@mailman.lug.org.uk How to Report Bugs Effectively: http://goo.gl/4Xue
Re: [Dorset] Hot Backup Mode
Hi Terry, Can anyone explain hot backup mode in the context of file server as opposed to a database? Can it be done? You mean you'd like to snapshot a filesystem so its own internal data is in a consitent state for backing up? Although the files' data might be partially complete, e.g. tar creating a large tar file might only be half done. Sure, they're call snapshots, typically, and normally employ copy-on-write so the snapshot is quick to produce and takes little extra storage; only writes modifying the data in the snapshot take extra space until the snapshot is deleted. Obviously, you still have to ensure important files, e.g. database ones, are internally consistent when the snapshot takes place, but since the snapshot is quick this doesn't have such an impact. See https://en.wikipedia.org/wiki/Snapshot_%28computer_storage%29 Cheers, Ralph. -- Next meeting: Bournemouth, Tuesday, 2014-08-05 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread on mailing list: mailto:dorset@mailman.lug.org.uk How to Report Bugs Effectively: http://goo.gl/4Xue
Re: [Dorset] Hot Backup Mode
On Tuesday 22 Jul 2014 12:25:10 Ralph Corderoy wrote: Can anyone explain hot backup mode in the context of file server as opposed to a database? Can it be done? You mean you'd like to snapshot a filesystem so its own internal data is in a consitent state for backing up? Although the files' data might be partially complete, e.g. tar creating a large tar file might only be half done. Sure, they're call snapshots, typically, and normally employ copy-on-write so the snapshot is quick to produce and takes little extra storage; only writes modifying the data in the snapshot take extra space until the snapshot is deleted. Obviously, you still have to ensure important files, e.g. database ones, are internally consistent when the snapshot takes place, but since the snapshot is quick this doesn't have such an impact. See https://en.wikipedia.org/wiki/Snapshot_%28computer_storage%29 We are still trying to understand the requirement, but the current system, (which we are bidding to replace), uses two Sun Sparc Servers with so called hot backup mode. We think that there may be some database files involved. It would appear that this 'mirroring' needs to be continuous, so that both machines remain in step at all times. The servers will be gathering data from a number of clients and serving it up to others (up to 50 in total). The key question that we have is; assuming that a hot backup has been achieved, how do we flip to the backup server seamlessly without losing the data that might have been changed very recently. One idea that we've had is to broadcast all changes to all clients and servers, so when one falls over, the other is already in step. Anyone seen anything like this? We are trying to find how the current system does it. -- Terry Coles -- Next meeting: Bournemouth, Tuesday, 2014-08-05 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread on mailing list: mailto:dorset@mailman.lug.org.uk How to Report Bugs Effectively: http://goo.gl/4Xue
Re: [Dorset] Hot Backup Mode
Hi Terry, Can anyone explain hot backup mode in the context of file server as opposed to a database? Can it be done? You mean you'd like to snapshot a filesystem so its own internal data is in a consitent state for backing up? No, you mean like a hot spare for failover when the primary goes kaput, or even better, a cluster (of two) fileservers that are in sync and share the load, read and write. We are still trying to understand the requirement, but the current system, (which we are bidding to replace), uses two Sun Sparc Servers with so called hot backup mode. We think that there may be some database files involved. It would appear that this 'mirroring' needs to be continuous, so that both machines remain in step at all times. The servers will be gathering data from a number of clients and serving it up to others (up to 50 in total). Lots need to be found out here. Do you have to fit in with the existing protocol from the clients, i.e. you're replacing just the servers? Do the clients know which of the two servers to contact and learn when they should switch, or is that transparent to them? Do both servers serve requests, or just the primary? Perhaps the primary handles read and write but the secondary just does reads. The key question that we have is; assuming that a hot backup has been achieved, how do we flip to the backup server seamlessly without losing the data that might have been changed very recently. Are you aware of the Linux HA project, high availability. That's the term often used to describe failover systems. There's lots out there on it. If just two servers are needed then a shared hardware solution is possible. Or pure software solutions on commodity hardware can be done. It's a big area! Examples, http://www.dell.com/content/topics/global.aspx/power/en/ps4q00_linux?c=us https://en.wikipedia.org/wiki/Lustre_%28file_system%29 One idea that we've had is to broadcast all changes to all clients and servers, so when one falls over, the other is already in step. Anyone seen anything like this? Network broadcasts tend to be unreliable protocols. See item 1; https://en.wikipedia.org/wiki/Fallacies_of_Distributed_Computing :-) You may also find http://ksat.me/a-plain-english-introduction-to-cap-theorem/ interesting if you haven't come across the CAP theorem before. Unless the requirements are simple, I'd avoid doing your own thing from scratch; there are lots of gotchas. We are trying to find how the current system does it. I think that's key. It may well be, especially if it's a bit old, that it's more simplistic than its grand description and that would lessen what you need to do. Cheers, Ralph. -- Next meeting: Bournemouth, Tuesday, 2014-08-05 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread on mailing list: mailto:dorset@mailman.lug.org.uk How to Report Bugs Effectively: http://goo.gl/4Xue
