On Sun, 15 Jan 2012, Jim Klimov wrote:

It does seem possible that in-memory corruption of data payload
and/or checksum of a block before writing it to disk would render
it invalid on read (data doesn't match checksum, ZFS returns EIO) .
Maybe even worse if the in-memory block is corrupted before the
checksumming, and seemingly valid garbage gets stored on disk,
read afterwards, and used with blind trust.

Please don't under-state the actual issue. ZFS assumes that RAM is 100% reliable. ZFS uses an in-memory cache called the ARC which can span many tens of gigabytes on busy large memory systems. User data is stored in this ARC and the cached data becomes the reference copy of the data until it is evicted. This means that user data can be silently and undetectably corrupted due to memory corruption. The effects that zfs's checksums can detect are just a small subset of the problems which may occur if memory returns wrong values.

In all these cases RAM is the SPOF (single point of failure)
so all ZFS recommendations involve using ECC systems. Alas,
even though ECC chips and chipsets are cheap nowadays, not all
architectures use them anyway (i.e. desktops, laptops, etc.),
and the tagline of running ZFS for "reliable storage on consumer
grade hardware" is poisoned by this fact. Other filesystems

Feel free to blame Intel for this since they seem to be primarily responsible for delivering CPUs and chipsets which don't support ECC. AMD has not been such a perpetrator, although it is possible to buy AMD-based systems which don't provide ECC.

I do wonder, however, if it is possible to make a software ECC
to detect-and/or-repair small memory corruptions on consumer
grade systems. And where would such part fit - in ZFS (i.e.

This could be done for part of the memory but it would obviously result in huge performance loss. I/O to memory would have to become block-oriented rather than random access. It is still necessary for random access to be used in a large part of the memory since it is a requirement in order to run programs and there would no way to defend that part of the memory.

some ECC bits appended in every zfs_*_t structure) or in the
{Solaris} kernel for general VM management. And even then
there's a question whether this would solve more problems than
create a greater one - pose the visibility of solution and
hide problems that actually exist (because there would be
some non-ECC parts of the data path and GIGO principle can
apply at any point). In the bad case, you ECC an invalid
piece of memory, and afterwards trust it as it matches the
checksum. On the good side, there is a smaller window that
data is exposed unprotected, so statistically this solution
should help.

The problem is that with unreliable memory, the software-based ECC would not be able to correct the content of the memory since the ECC itself might have been computed incorrectly (due to unreliable memory). You are then faced with notifications of problems that the user can't fix.

The proper solution (regardless of filesystem used) is to assure that ECC is included in any computer that you buy.

Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
zfs-discuss mailing list

Reply via email to