On February 15, 2011, Black, Michael (IS) wrote: > I'll give you another failure point that most people never see or think of. > > I used to manage numerous Linux systems with RAID-5. One time I had a > drive fail, the spare kicked in, and then during the rebuild a 2nd drive > failed...hosing the RAID (i.e. two failed disks). > > The problem was...normal disk access/backup only scanned the in-use blocks. > The RAID resync scanned the entire disk which had never been done.
I think this is one reason mdraid on linux by default will run a resync regularly. > After that I put in a utility that did a nightly "dd if=/dev/md0 > >/dev/null" job to force a scan of the entire disk set. > > This is one reason why they invented RAID6. > > There's just so many ways to fail...sigh... > > > Michael D. Black > Senior Scientist > NG Information Systems > Advanced Analytics Directorate > > > > ________________________________________ > From: [email protected] [[email protected]] on > behalf of Simon Slavin [[email protected]] Sent: Monday, February 14, > 2011 5:04 PM > To: General Discussion of SQLite Database > Subject: EXT :Re: [sqlite] UPDATE/INSERTing 1-2k rows slower than expected > > On 14 Feb 2011, at 8:50pm, Black, Michael (IS) wrote: > > And if you want to talk about data reliability...BACK UP YOUR DATA. > > And keep the backups off-site. And once a year try to actually restore one > of them. I've earned /lots/ of money from companies with backup systems > that claimed to work fine but were either faulty, or being used to backup > to media with low-level corruption. > > If there was any money in it I'd write a book, but there isn't. Companies > prefer to do it wrong and then hire a consultant after it fails. > > On 14 Feb 2011, at 9:13pm, Jim Wilcoxson wrote: > > I think what would be useful is for drives to have 3 settings: > > > > a) the drive can do whatever it wants to optimize performance > > b) the drive can reorder writes, but not across a sync (ACI, but no D) > > c) the drive has to respect all syncs (ACID) > > > > If the drive mgfr wants to make a) the default, that's fine, but there > > should be an easy way to request the other 2 from an application. > > Users are not usually sophisticated enough to know when it's okay to > > cache writes or not okay. For my use of SQLite, b) would probably be > > fine, but a) apparently is not since it corrupts databases. > > This is a job for someone who wants to contact lots of manufacturers and > ask if they provide models of their drives which conform. Some do, > because they want to sell their drives for use in servers. Often these > drives are physically identical to their mass-market drives, but they have > fewer bad blocks, and have jumpers in a different place or different > firmware. > > > Michael mentioned doing backups and forgetting about all of this, but > > many people are using USB drives to store their backups. So there is > > a bit of a Catch-22 here. Almost all modern-day filesystems are going > > to depend on some kind of journalling to prevent corrupted file > > systems, and as far I as know, journalling filesystems depend on syncs > > to maintain FS consistency. > > If the data is backed up when SQLite has the files closed, things are fine. > The really hard part of this comes when you're working with systems that > must be live at all times. Which is why you'll never see a bank use > SQLite to maintain its live transaction system. The big DBMSs have > systems inside themselves that allow the backing-up of an active > ever-changing database. > > Simon. > _______________________________________________ > sqlite-users mailing list > [email protected] > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > _______________________________________________ > sqlite-users mailing list > [email protected] > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users -- Thomas Fjellstrom [email protected] _______________________________________________ sqlite-users mailing list [email protected] http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
