Re: [zfs-discuss] Finding disks [was: # disks per vdev]
OK, I have finally settled on hardware; 2x LSI SAS3081E-R controllers 2x Seagate Momentus 5400.6 rpool disks 15x Hitachi 5K3000 'data' disks I am still undecided as to how to group the disks. I have read elsewhere that raid-z1 is best suited with either 3 or 5 disks and raid-z2 is better suited with 6 or 10 disks - is there any truth in this, although I think this was in reference to 4K sector disks; 3x 5 drive z1 = 24t usable 2x 6 drive z2 = 16t usable keeping to those recommendations or 2x 7 disk z2 = 20t usable with 1 cold/warm/hot spare as per my original idea. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Finding disks [was: # disks per vdev]
The LSI2008 chipset is supported and works very well. I would actually use 2 vdevs; 8 disks in each. And I would configure each vdev as raidz2. Maybe use one hot spare. And I also have personal, subjective reasons: I like to use the number of 8 in computers. 7 is an ugly number. Everything is based on powers of 2 in computers. A pocket calculator which only accepts the digits 1-8, but not accept the digit 9, is really ugly (having 7 discs, but not 8, is ugly). Some time ago, there was a problem unless you used even number of discs, that problem is corrected now. I would definitively use raidz2, because resilver time will be very long with 4-5TB disks, potentially several days. During that time, another disk problem such as reas error might occur, which means you loose all your data. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Finding disks [was: # disks per vdev]
Thanks. I ruled out the SAS2008 controller as my motherboard is only PCIe 1.0 so would not have been able to make the most of the difference in increased bandwidth. I can't see myself upgrading every few months (my current WHZ build has lasted over 4 years without a single change) so by the time I do come to upgrade PCIe will probably be obselete!! -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] I/O Currently Suspended Need Help Repairing
Hey guys, I had a zfs system in raidz1 that was working until there was a power outage and now I'm getting this: pool: tank state: DEGRADED status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://www.sun.com/msg/ZFS-8000-HC scrub: scrub in progress for 0h49m, 0.00% done, 7059912h41m to go config: NAMESTATE READ WRITE CKSUM tankDEGRADED28 0 0 raidz1-0 DEGRADED 18313 6 c6d0ONLINE 0 0 0 c7d0DEGRADED 9 21422 too many errors c7d1DEGRADED1327 0 too many errors errors: 28 data errors, use '-v' for a list When I use the -v option to see the errors I get stuff like this: metadata:0x18 metadata:0x19 ... So my question is, when the scrub is done, what should I do to fix the problems? I think those 28 errors refer to 28 different files, so lets say I have no problem deleting those files to save the rest of the drives, how do I go about doing this? Thanks. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] I/O Currently Suspended Need Help Repairing
Ok so now I have no idea what to do. The scrub is not working either. The pool is only 3x 1.5TB drives so it should not take so long. Does anyone know what I should do next? pool: tank state: DEGRADED status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://www.sun.com/msg/ZFS-8000-HC scrub: scrub in progress for 39h40m, 0.00% done, 336791319h2m to go config: NAMESTATE READ WRITE CKSUM tankDEGRADED28 0 0 raidz1-0 DEGRADED 18313 6 c6d0ONLINE 0 0 0 c7d0DEGRADED 9 21422 too many errors c7d1DEGRADED1327 0 too many errors errors: 28 data errors, use '-v' for a list Thanks. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Finding disks [was: # disks per vdev]
On Tue, Jul 5, 2011 at 6:54 AM, Lanky Doodle lanky_doo...@hotmail.com wrote:
OK, I have finally settled on hardware;
2x LSI SAS3081E-R controllers
2x Seagate Momentus 5400.6 rpool disks
15x Hitachi 5K3000 'data' disks
I am still undecided as to how to group the disks. I have read elsewhere that
raid-z1 is best suited with either 3 or 5 disks and raid-z2 is better suited
with 6 or 10 disks - is there any truth in this, although I think this was in
reference to 4K sector disks;
3x 5 drive z1 = 24t usable
2x 6 drive z2 = 16t usable
Take a look at
https://spreadsheets.google.com/spreadsheet/ccc?key=0AtReWsGW-SB1dFB1cmw0QWNNd0RkR1ZnN0JEb2RsLXchl=en_US
I did a bunch of testing with 40 drives. I varied the configuration
between extremes of 10 vdevs of 4 disks each to one vdev of all 40
drives. All vdevs were raidz2, so my net capacity changed, but I was
looking for _relative_ performance differences. I did not test
sequential reads, as that was not one of our expected I/O patterns. I
believe the OS was Solaris 10U8. I know it was at least zpool version
15 and may have been 22.
I used the same 40 drives in all the test cases as I had seen
differences between drives, and choose 40 that all had roughly
matching svc_t values (from iostat). Eventually we had Sun/Oracle come
in and replace any drive who's svc_t was substantially higher than the
others (these drives also usually had lots of added bad blocks
mapped).
keeping to those recommendations or
2x 7 disk z2 = 20t usable with 1 cold/warm/hot spare
The testing was utilizing a portion of our drives, we have 120 x 750
SATA drives in J4400s dual pathed. We ended up with 22 vdevs each a
raidz2 of 5 drives, with one drive in each of the J4400, so we can
lose two complete J4400 chassis and not lose any data.
--
{1-2-3-4-5-6-7-}
Paul Kraus
- Senior Systems Architect, Garnet River ( http://www.garnetriver.com/ )
- Sound Coordinator, Schenectady Light Opera Company (
http://www.sloctheater.org/ )
- Technical Advisor, RPI Players
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Changed to AHCI, can not access disk???
From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Orvar Korvar Here is my problem: I have an 1.5TB disk with OpenSolaris (b134, b151a) using non AHCI. I then changed to AHCI in BIOS, which results in severe problems: I can not boot the system. I suspect the problem is because I changed to AHCI. This is normal, no matter what OS you have. It's the hardware. If you start using a disk in non-AHCI mode, you must always continue to use it in non-AHCI mode. If you switch, it will make the old data inaccessible. Just change it back in BIOS and you'll have your data back. Then backup your data, change to AHCI mode (because it's supposed to perform better) and restore your data. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Changed to AHCI, can not access disk???
On Tue, Jul 5, 2011 at 8:03 PM, Edward Ned Harvey opensolarisisdeadlongliveopensola...@nedharvey.com wrote: From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Orvar Korvar Here is my problem: I have an 1.5TB disk with OpenSolaris (b134, b151a) using non AHCI. I then changed to AHCI in BIOS, which results in severe problems: I can not boot the system. I suspect the problem is because I changed to AHCI. This is normal, no matter what OS you have. It's the hardware. If you start using a disk in non-AHCI mode, you must always continue to use it in non-AHCI mode. If you switch, it will make the old data inaccessible. Really? old data inaccessible? These two links seem to say that the data is there, and it's only a matter whether the correct drivers are loaded or not: http://en.wikipedia.org/wiki/Ahci http://support.microsoft.com/kb/922976 So the question is, does similar workaround exists for (open)solaris? -- Fajar ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Finding disks [was: # disks per vdev]
On Tue, Jul 5, 2011 at 7:47 AM, Lanky Doodle lanky_doo...@hotmail.com wrote: Thanks. I ruled out the SAS2008 controller as my motherboard is only PCIe 1.0 so would not have been able to make the most of the difference in increased bandwidth. Only PCIe 1.0? What chipset is that based on? Might be worthwhile to upgrade as I believe Solaris power-management has a fairly recent cutoff in terms of processor support when it comes to power-management. (AMD Family 16 or better, Intel Nehalem or newer is what I've been told). PCIe 2.0 has been around for quite awhile, PCIe 3.0 will be making an appearance on Ivy Bridge CPUs (and has already been announced by FPGA vendors), but I'm fairly confident that graphics cards will be the first target market to utilize that. Another thing to consider is that you could buy the SAS2008-based cards and move them from motherboard to motherboard for the foreseeable future (copper PCI Express isn't going anywhere for a long time). Don't kneecap yourself because of your current mobo. --khd ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] I/O Currently Suspended Need Help Repairing
Hello, Are you certain that after the outage your disks are indeed accessible? * What does BIOS say? * Are there any errors reported in dmesg output or the /var/adm/messages file? * Does the format command return in a timely manner? ** Can you access and print the disk labels in format command? (Select a disk by number, enter p, enter p again). From the output below it seems that there is some hardware problem, like connectivity (loose SATA cables, etc.) or the disks are fried. And with to disks out of the 3-disk set, the outlook is grim (if they are indeed FUBAR). If it is just a connectivity problem with at least one of the disks, you have a chance of scrub salvaging the data. So more or less start by doing what the status command said, in this case ;) 2011-07-04 23:52, zfsnoob4 wrote: Ok so now I have no idea what to do. The scrub is not working either. The pool is only 3x 1.5TB drives so it should not take so long. Does anyone know what I should do next? pool: tank state: DEGRADED status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://www.sun.com/msg/ZFS-8000-HC scrub: scrub in progress for 39h40m, 0.00% done, 336791319h2m to go config: NAMESTATE READ WRITE CKSUM tankDEGRADED28 0 0 raidz1-0 DEGRADED 18313 6 c6d0ONLINE 0 0 0 c7d0DEGRADED 9 21422 too many errors c7d1DEGRADED1327 0 too many errors errors: 28 data errors, use '-v' for a list Thanks. HTH, //Jim Klimov ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Changed to AHCI, can not access disk???
Ok, so I switch back and then I have my data back? But, it does not work. Because, meanwhile I switched, I tried to zpool import which messed up the drive. Then I switched back, but my data is still not accessible. Earlier, when I switched, I did not do a zpool import and when I switched back, I could access all my data. But now I have done a zpool import while I switched - which messes up things. If I have done zpool import while switched, can I get back my rpool, when I switch back? - Original Message From: Edward Ned Harvey opensolarisisdeadlongliveopensola...@nedharvey.com To: Orvar Korvar knatte_fnatte_tja...@yahoo.com; zfs-discuss@opensolaris.org Sent: Tue, July 5, 2011 3:03:50 PM Subject: RE: [zfs-discuss] Changed to AHCI, can not access disk??? From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Orvar Korvar Here is my problem: I have an 1.5TB disk with OpenSolaris (b134, b151a) using non AHCI. I then changed to AHCI in BIOS, which results in severe problems: I can not boot the system. I suspect the problem is because I changed to AHCI. This is normal, no matter what OS you have. It's the hardware. If you start using a disk in non-AHCI mode, you must always continue to use it in non-AHCI mode. If you switch, it will make the old data inaccessible. Just change it back in BIOS and you'll have your data back. Then backup your data, change to AHCI mode (because it's supposed to perform better) and restore your data. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Changed to AHCI, can not access disk???
2011-07-05 17:11, Fajar A. Nugraha пишет: On Tue, Jul 5, 2011 at 8:03 PM, Edward Ned Harvey opensolarisisdeadlongliveopensola...@nedharvey.com wrote: From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Orvar Korvar Here is my problem: I have an 1.5TB disk with OpenSolaris (b134, b151a) using non AHCI. I then changed to AHCI in BIOS, which results in severe problems: I can not boot the system. I suspect the problem is because I changed to AHCI. This is normal, no matter what OS you have. It's the hardware. If you start using a disk in non-AHCI mode, you must always continue to use it in non-AHCI mode. If you switch, it will make the old data inaccessible. Really? old data inaccessible? Kind of. You can not access it now, right? ;) So the question is, does similar workaround exists for (open)solaris? One of the ways is to have Solaris remap the boot devices. When you change the boot device paths, and this includes switching the access mode in BIOS (the controller number and driver would probably change), Solaris would give you a panic during boot and restart. See this list or the internet for how to boot Solaris with KDB (kernel debugger) and/or verbose mode - this should keep the computer from rebooting and allow you to read the strings on the screen, along the lines: zfs/vfs can't mount the root, panic, tried device paths /pci/something. If that's the case, you can return the BIOS to the mode where Solaris boots okay, and try to fix up the EEPROM - rather its x86 emulation in file /boot/solaris/bootenv.rc - which stores hardware configuration. For an easy solution you can try to comment away the line like (if it is there in the first place): setprop bootpath /pci@7b,0/pci1022,7458@11/pci1000,3060@2/sd@3,0:a Otherwise you may have to discover and insert the correct line by using the livecd/installcd or the failsafe boot mode (Solaris 10/SXCE) while and running devfsadm -Cv in order for it to generate new disk/slice paths. Then you can use ls -la /dev/dsk and see what PCI paths are assigned (possibly this would lead you to /devices/ first, though). There is another possible solution: boot from livecd while in AHCI mode, and import/export the rpool. This may change the device names/links in the ZFS pool header, helping to rename them to the new controller numbers... Good luck! ;) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Changed to AHCI, can not access disk???
On Tue, Jul 5, 2011 at 9:11 AM, Fajar A. Nugraha w...@fajar.net wrote:
On Tue, Jul 5, 2011 at 8:03 PM, Edward Ned Harvey
opensolarisisdeadlongliveopensola...@nedharvey.com wrote:
From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss-
boun...@opensolaris.org] On Behalf Of Orvar Korvar
Here is my problem:
I have an 1.5TB disk with OpenSolaris (b134, b151a) using non AHCI.
I then changed to AHCI in BIOS, which results in severe problems: I can
not
boot the system.
I suspect the problem is because I changed to AHCI.
This is normal, no matter what OS you have. It's the hardware.
If you start using a disk in non-AHCI mode, you must always continue to use
it in non-AHCI mode. If you switch, it will make the old data inaccessible.
Really? old data inaccessible?
While I agree that you should not change the controller mode with
data behind it, I did do that on a Supermicro system running OpenSuSE
and Linux LVM mirrors with no issues. I suspect because Linux both
loads the AHCI drivers in the mini-root (to use a Sun term :-) and
that the LVM layer uses a private region on disk to identify the
device (much like both SLVM and ZFS).
--
{1-2-3-4-5-6-7-}
Paul Kraus
- Senior Systems Architect, Garnet River ( http://www.garnetriver.com/ )
- Sound Coordinator, Schenectady Light Opera Company (
http://www.sloctheater.org/ )
- Technical Advisor, RPI Players
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Finding disks [was: # disks per vdev]
On Tue, Jul 5, 2011 at 12:54 PM, Lanky Doodle lanky_doo...@hotmail.com wrote: OK, I have finally settled on hardware; 2x LSI SAS3081E-R controllers Beware that this controller does not support drives larger than 2TB. -- Trond Michelsen ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Changed to AHCI, can not access disk???
2011-07-05 19:21, Paul Kraus wrote: While I agree that you should not change the controller mode with data behind it, I did do that on a Supermicro system running OpenSuSE and Linux LVM mirrors with no issues. I suspect because Linux both loads the AHCI drivers in the mini-root (to use a Sun term :-) and that the LVM layer uses a private region on disk to identify the device (much like both SLVM and ZFS). Good point about the miniroot. One more possible solution (component) would be this: 1) Set your root disk to AHCI or whatever you need 2) Boot from livecd 3) Import your rpool and mount the root filesystem into /a for example 4) Update device links: # devfsadm -r /a -Cv 5) Update the miniroot # bootadm update-archive -R /a 6) Export the rpool 7) Reboot and hope this helps ;) //Jim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Changed to AHCI, can not access disk???
I have already formatted one disk, so I can not try this anymore. (But, importing the zpool with the name rpool and exporting the rpool again, was successful. I can now use the disk as usual. But this did not work on the other disk, so I formatted it) -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] I/O Currently Suspended Need Help Repairing
Thanks for your help, I did a zpool clear and now this happens: pool: tank state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: scrub in progress for 61h58m, 0.00% done, 526074322h7m to go config: NAMESTATE READ WRITE CKSUM tankONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 c6d0ONLINE 0 0 0 c7d0ONLINE 0 012 15.5K repaired c7d1ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: metadata:0x0 metadata:0x1 metadata:0x3 metadata:0x14 metadata:0x15 metadata:0x16 metadata:0x18 metadata:0x19 metadata:0x1a metadata:0x1b metadata:0x1c metadata:0x1d metadata:0x1e metadata:0x1f tank/server:0x0 tank/media:0x0 So there are two things going on. It says there is still a scrub in progress but its at 0% after 61h. Is there a way to stop it? Should I do that? Also for the error files, how do I delete those files and clear the error? Thanks again for your help. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] I/O Currently Suspended Need Help Repairing
Reading through this page (http://dlc.sun.com/osol/docs/content/ZFSADMIN/gbbwl.html), it seems like all I need to do is 'rm' the file. The problem is finding it in the first place. Near the bottom of this page it says: If the damage is within a file data block, then the file can safely be removed, thereby clearing the error from the system. The first step is to try to locate the file by using the find command and specify the object number that is identified in the zpool status output under DATASET/OBJECT/RANGE output as the inode number to find. For example: # find -inum 6 But the find inum command isn't working. Has anyone run into this? Thanks. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Changed to AHCI, can not access disk???
On 2011-Jul-05 21:03:50 +0800, Edward Ned Harvey opensolarisisdeadlongliveopensola...@nedharvey.com wrote: From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Orvar Korvar ... I suspect the problem is because I changed to AHCI. This is normal, no matter what OS you have. It's the hardware. Switching to AHCI changes the device interface presented to the kernel and you need a different device driver to access the data. As long as your OS supports AHCI (and that is true of any OS that supports ZFS) then you will still be able to access the disks - though the actual path to the disk or disk device name will change. If you start using a disk in non-AHCI mode, you must always continue to use it in non-AHCI mode. If you switch, it will make the old data inaccessible. Only if your OS is broken. The data is equally accessible in either mode. ZFS makes it easier to switch modes because it doesn't care about the actual device name - at worst, you will need an export and import. -- Peter Jeremy pgpHPygB4VeNl.pgp Description: PGP signature ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
