That's a good question to which I have no answer. I don't know how google do it. I can think of 1. special file system 2. some kind of "scrubber" - a daemon scanning for FS changes and copying whatever changed 3. use a sync tool (rsync?) on adaily (hourly?) basis
I doubt google has 1. This is some good startup material. Dan On 3/18/07, Amos Shapira <[EMAIL PROTECTED]> wrote:
On 07/03/07, Dan Bar Dov <[EMAIL PROTECTED]> wrote: > I suggest you read the latest summary on the reports on storage at > http://storagemojo.com/?p=383 > http://storagemojo.com/?p=378 > > The bottom line conclusions I made out of those are > 1. Do not use RAID5. Stick to RAID 1 (0+1). > 2. Use cheap SATA storage - even build-yourself. The big bucks don't > buy you real reliability. > 3. If at all possible - implement a 3 copies/file scheme rather than > RAID1/5/6. In the long run is is much simpler and more reliable. > > I'd be glad to discuss this more, but man, keep your posts shorter :-) Thanks for the pointers. Now - how would you suggest implementing your point 3, short of having access to an implementation of Google FS? I'm in a position to re-design a system which requires reliable, fault-tolerant access to files over LAN for processing (i.e. read files, calculate stuff on their input, write other files and to a database) and archiving. I'd rather build the solution on proven tools instead of re-inventing a home-grown solution from scratch, but so far I haven't found something I can bet the company's fate on. The current system is built on Windows and though the business owner is open to think about a move to Linux, it will have to be a gradual. Thanks, --Amos
================================================================= To unsubscribe, send mail to [EMAIL PROTECTED] with the word "unsubscribe" in the message body, e.g., run the command echo unsubscribe | mail [EMAIL PROTECTED]
