Re: [HACKERS] Block-level CRC checks

[Decibel!]

On Sep 30, 2008, at 2:17 PM, [EMAIL PROTECTED] wrote:

A customer of ours has been having trouble with corrupted data for some

time.  Of course, we've almost always blamed hardware (and we've seen

the useful thing to know is when corruption has happened, and where.

RAID controllers have their firmware upgraded, among other actions), but

correct?

That is an important statement, to know when it happens not necessarily to be able to recover the block or where in the block it is corrupt. Is that

Oh, correcting the corruption would be AWESOME beyond belief! But at this point I'd settle for just knowing it had happened.

So we've been tasked with adding CRCs to data files.

CRC or checksum? If the objective is merely general "detection" there
should be some latitude in choosing the methodology for performance.

See above. Perhaps the best win would be a case where you could choose which method you wanted. We generally have extra CPU on the servers, so we could afford to burn some cycles with more complex algorithms.

The idea is that these CRCs are going to be checked just after reading

files from disk, and calculated just before writing it.  They are
just a protection against the storage layer going mad; they are not
intended to protect against faulty RAM, CPU or kernel.

system from hiding.

It will actually find faults in all if it. If the CPU can't add and/ or a RAM location lost a bit, this will blow up just as easily as a bad block. It may cause "false identification" of an error, but it will keep a bad

Well, very likely not, since the intention is to only compute the CRC when we write the block out, at least for now. In the future I would like to be able to detect when a CPU or memory goes bonkers and poops on something, because that's actually happened to us as well.

The implementation I'm envisioning requires the use of a new relation

in the CRC fork; ReadBuffer_common would read the page, calculate the
checksum, and compare it to the one stored in the CRC fork.

fork to store the per-block CRCs. Initially I'm aiming at a CRC32 sum for each block. FlushBuffer would calculate the checksum and store it

I mean, if you just want a sanity test, it doesn't take much. Using a

the block knows its CRC or checksum and that is in error, the block is
bad.

Hell, all that is needed is a long or a short checksum value in the block. second relation creates confusion. If there is a CRC discrepancy between two different blocks, who's wrong? You need a third "control" to know. If

--
Decibel!, aka Jim C. Nasby, Database Architect  [EMAIL PROTECTED]
Give your computer some brain candy! www.distributed.net Team #1828

I believe the idea was to make this as non-invasive as possible. And it would be really nice if this could be enabled without a dump/ reload (maybe the upgrade stuff would make this possible?)

smime.p7s
Description: S/MIME cryptographic signature