On Apr 6, 2010, at 10:50 PM, Paul J Davis wrote: > This corruption was quite odd in that there wasn't a conspicuous reason for > it. I didn't dive to deep into the whole thing so it's possible i missed > something obvious.
The instance was unresponsive to ssh for 12 hours. The report from AWS Support was merely a "problem with the underlying host" followed by a recommendation to "launch a replacement at your earliest convenience". I don't know what the gremlins were doing behind the scenes, but I'm not surprised the files are corrupted :) > There are two things at play here. How proactive should we be in provoking > these errors and how much should we check for situations where our data file > got trounced. > > The extreme proactive position would be equivalent to a full table scan per > write which is out of the question. So to some extent we won't be able to > detect some errors until read time which is an unknowable interval. I'm totally comfortable with only detecting them at read-time. > The other aspect is how rigorous should we check reads? This extreme would > basically require a sha1 for every read or write no matter how small, not to > mention the storage overhead. This part I'm not sure about. There's probably > middle ground with crc sums and what not but i don't see a clear answer. We currently store MD5 checksums with document bodies and validate them on reads. It hasn't proven to be an undue burden. Best, Adam > Basically, the question is how much should we attempt to detect when hardware > lies. I reckon that there's probably a middle ground to report when an > assumption is violated and full on table scans. Ideally such things would be > fairly configurable but i sure don't see an obvious answer. > > > On Apr 6, 2010, at 10:06 PM, Randall Leeds <[email protected]> wrote: > >> I immediately want to say 'ini file option' but I'm not sure whether to err >> on safety or speed. >> >> Maybe this is a good candidate for merkle trees or something else we can do >> throughout the view tree that might less overhead than md5 summing all the >> nodes? After all, most inner nodes shouldn't change most of the time. Some >> incremental, cheap checksum might be a worthwhile *option*. >> >> On Apr 6, 2010 6:04 PM, "Adam Kocoloski" <[email protected]> wrote: >> >> Hi all, we recently had an EC2 node go AWOL for about 12 hours. When it >> came back, we noticed after a few days that a number of the view indexes >> stored on that node were not updating. I did some digging into the error >> logs and with Paul's help pieced together what was going on. I won't bother >> you with all the gory details unless you ask for them, but the gist of it is >> that those files are corrupted. >> >> The troubling thing for me is that we only discovered the corruption when it >> completely broke the index updates. In one case, it did this by rearranging >> the bits so that couch_file thought that the btree node it was reading from >> disk had an associated MD5 checksum. It didn't (no btree nodes do), and so >> couch_file threw a file_corruption exception. But if the corruption had >> shown up in another part of the file I might never have known. In fact, >> some of the other indices on that node probably are silently corrupted. >> >> You might wonder how likely it is that a file becomes corrupted but still >> appears to be functioning. I checked the last modified timestamps for three >> broken files. One was last modified when the node went down, but the other >> two had timestamps in between the node's recovery and now. To me, that >> means that the view indexer was able to update those files for quite a while >> (~2 days) before it bumped into a part of the btree that was corrupted. >> >> I wonder what we should do about this. My first thought is to make it >> optional to write btree nodes (possibly only for view index files?) using >> append_term_md5 instead of append_term. It seems like a simple patch, but I >> don't know a priori what the performance hit would be. Other thoughts? >> >> Best, Adam
