[zfs-discuss] Checksum error in single device ZFS pool mysterious resilvering after reboot
I have recently been experimenting with ZFS (under FreeBSD 7.0), and I am excited about the possibilities. I have seen, however, two puzzling issues, and I am hoping someone here might have some insight or info. Note that the original zpool that I am discussing is gone now - I had to recreate a new one to make space for a larger root partition, etc... Note that I had one ZFS pool with only a single device/partition (i.e. no mirroring or raidz). Anyway, the two issues are: 1) During a routine test scrub, a checksum error was found. There were also some other errors reported around this time, but not fatal (note that I am investigating potential FreeBSD DMA messages in my log - could be related; the hardware seems to check out fine, BTW). This checksum error was persistent due to the fact that there was no mirror. One file was affected, and upon further investigation, a 64K block (1/2 of a ZFS block?) appeared to be data from some other file. 2) After my first reboot from the above situation, I found that zpool status reported that resilvering completed Also, all error counts were zeroed (not unusual), but the pool now reported that no known errors existed. Rescrubbing revealed the checksum error again, but this time no subsequent reboots ever caused resilvering or hiding of the file error again. The puzzling thing is why would ZFS resilver anything, given that there was only one device in the pool? Plus, the fact that the pool now thought it was OK was disturbing. I am, of course, unsure if 64K of the original file was corrupted at the time of the original file wrote, some time after that (well, at least it appeared to be text data that would have been either from another file, specifically a Mozilla mork DB file, or perhaps on its way to another file and mistakenly written into this file (although no other files had checksum issues)), or if the strange resilver caused the block to become this way. The history of that file is that I had copied my entire home dir to the ZFS pool with no errors reported, and I had not touched that file before the scrub found the problem. I did some forensics later by temporarily disabling the checksum error in the ZFS code in order to read the whole file (this is how I revealed the contents of the bad block). The bad 64K was only a small part of the file, starting at byte 655360 and going for exactly 64K. The rest of the file, up to 3MB or so, was fine. Anyway, does any of the above ring any bells? Thanks, Joe ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS Performance Issue
William Fretts-Saxton wrote: Unfortunately, I don't know the record size of the writes. Is it as simple as looking @ the size of a file, before and after a client request, and noting the difference in size? and The I/O is actually done by RRD4J, [...] a Java version of 'rrdtool' If it behaves like rrdtool, it will limit the size of the file, by consolidating older data. After every n samples, older data will be replaced by an aggregate, freeing space for new samples. To me that implies random I/O. You really need a tool like dtrace (or old fashioned truss) to see the sample rate and size. Cheers, Henk ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] OpenSolaris, ZFS and Hardware RAID, a recipe for success?
I have been tasked with putting together a storage solution for use in a virtualization setup, serving NFS, CIFS, and iSCSI, over GigE. I've inherited a few components to work with: x86 dual core server , 512MB LSI-ELP RAID card 12 x 300GB 15Krpm SAS disks array 2GB Flash to IDE disk/adaptor. The system will be serving virtual hard disks to a range of vmware systems connected by GigE, running enterprise workloads that are impossible to predict at this point. Using the RAID cards capability for RAID6 sounds attractive? Using the Flash RAM for the ZIL? Using zfs for general storage management? Has anyone built a similar system, what is the true path to success? What are the pitfalls? What should I have on my reading list for starters? All advice most gratefully received. MM. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] OpenSolaris, ZFS and Hardware RAID, a recipe for success?
Nick wrote: I have been tasked with putting together a storage solution for use in a virtualization setup, serving NFS, CIFS, and iSCSI, over GigE. I've inherited a few components to work with: x86 dual core server , 512MB LSI-ELP RAID card 12 x 300GB 15Krpm SAS disks array 2GB Flash to IDE disk/adaptor. The system will be serving virtual hard disks to a range of vmware systems connected by GigE, running enterprise workloads that are impossible to predict at this point. Using the RAID cards capability for RAID6 sounds attractive? Assuming the card works well with Solaris, this sounds like a reasonable solution. Using the Flash RAM for the ZIL? I'm not sure why you would want to do this. Just carve off a LUN or slice on the RAID card and use its NVRAM cache. A consumer class flash disk will be slower. Using zfs for general storage management? cool. Has anyone built a similar system, what is the true path to success? Success is at the summit, but there are several paths up the mountain. What are the pitfalls? What should I have on my reading list for starters? Start with the ZFS system admin guide on opensolaris.org. We try to keep the solarisinternals.com wikis up to date also. -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] OpenSolaris, ZFS and Hardware RAID, a recipe for success?
Richard Elling wrote: Nick wrote: Using the RAID cards capability for RAID6 sounds attractive? Assuming the card works well with Solaris, this sounds like a reasonable solution. Careful here. If your workload is unpredictable, RAID 6 (and RAID 5) for that matter will break down under highly randomized write loads. There's a lot of trickery done with hardware RAID cards that can do some read-ahead caching magic, improving the read-paritycalc-paritycalc-write cycle, but you can't beat out the laws of physics. If you do *know* you'll be streaming more than writing random small number of blocks, RAID 6 hardware can work. But with transaction like loads, performance will suck. Jon ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
