Re: LSI 9240-4i 4K alignment
On 19 Aug, Josh Paetzel wrote: On 08/19/2012 14:04, Steven Hartland wrote: HBA's are the way to go if your using ZFS to manage the disks, you only need RAID if your using a FS which doesn't manage the disk side well such as UFS. Its often quite common for RAID controllers to actually be slower vs RAID controllers as the RAID stack can get in the way. Any idea of what kind of performance penalty I might see by using the RAID firmware in JBOD mode vs flashing the IT firmware? Just to clear up, The 9240 is a sas2008 based card with the megaraid software on top of it. In it's default config from LSI the FreeBSD mfi will recognize it in later versions of FreeBSD (The upcoming 9.1 for sure) Older versions of mfi will not recognize it. The card can be flashed with IT firmware and then becomes a 9211 HBA, but it's a bit more expensive than a 9211 is so that doesn't make sense to do in many cases. The price difference was pretty minor when I looked. Confusingly enough, the 9211 HBA also has some RAID capabilities. For me, the biggest advantage of the 9211 would be that it would have allowed me to use shorter cables. On the dmesg posted the firmware on the card is phase 11. This *must* be in lockstep with the driver version or the card may not play nicely. FreeBSD 8.3 and 9.0 have v13 of the driver, the upcoming 9.1 will have v14. Note that v14 fixes a *ton* of stability bugs, including issues where bad drives would hang the controller or prevent systems from booting. Where do those version numbers come from? The mfi driver in 9.0-RELEASE claims to be version 3.00 and the the driver in 9.1 claims to be version 4.23. This is what shows up in dmesg on my machine: mfi0: Drake Skinny port 0xce00-0xceff mem 0xfcefc000-0xfcef,0xfce8-0xf ceb irq 18 at device 0.0 on pci1 mfi0: Using MSI mfi0: Megaraid SAS driver Ver 4.23 mfi0: 333 (398082533s/0x0020/info) - Shutdown command received from host mfi0: 334 (boot + 3s/0x0020/info) - Firmware initialization started (PCI ID 0073 /1000/9240/1000) mfi0: 335 (boot + 3s/0x0020/info) - Firmware version 2.70.04-0862 mfi0: 336 (boot + 5s/0x0020/info) - Board Revision 04A mfi0: 337 (boot + 3s/0x0020/info) - Firmware initialization started (PCI ID 0073 /1000/9240/1000) mfi0: 338 (boot + 3s/0x0020/info) - Firmware version 2.70.04-0862 mfi0: 339 (boot + 5s/0x0020/info) - Board Revision 04A mfi0: 340 (boot + 3s/0x0020/info) - Firmware initialization started (PCI ID 0073 /1000/9240/1000) mfi0: 341 (boot + 3s/0x0020/info) - Firmware version 2.70.04-0862 mfi0: 342 (boot + 5s/0x0020/info) - Board Revision 04A mfi0: 343 (boot + 3s/0x0020/info) - Firmware initialization started (PCI ID 0073 /1000/9240/1000) mfi0: 344 (boot + 3s/0x0020/info) - Firmware version 2.70.04-0862 mfi0: 345 (boot + 5s/0x0020/info) - Board Revision 04A mfi0: 346 (398759025s/0x0020/info) - Time established as 08/20/12 6:23:45; (25 seconds since power on) mfi0: 347 (398759051s/0x0020/info) - Time established as 08/20/12 6:24:11; (51 seconds since power on) mfi0: 348 (398759078s/0x0020/WARN) - Patrol Read can't be started, as PDs are ei ther not ONLINE, or are in a VD with an active process, or are in an excluded VD % mfiutil show firmware mfi0 Firmware Package Version: 20.5.1-0003 mfi0 Firmware Images: Name VersionDate Time Status BIOS 4.14.00 active PCLI 03.02-001:#%8 Feb 09 2010 13:09:06 active BCON 4.0-22-e_10-RelMar 11 2010 12:38:08 active NVDT 3.04.03-0002 Apr 05 2010 18:50:27 active APP 2.70.04-0862 May 05 2010 18:12:07 active BTBL 2.01.00.00-0019May 14 2009 15:52:08 active The only firmware file on LSI's web site for the 9240-8i is version 20.10.1-107, which appears to be newer than what is on the card if the 20.5.1-0003 is the version number that I should be looking at. Is the BIOS Version 4.14 the v14 version that you mention above? If the FreeBSD mfi driver expects a certain firmware version, shouldn't it complain if it doesn't find it? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: LSI 9240-4i 4K alignment
On 08/20/2012 00:19, Don Lewis wrote: On 19 Aug, Josh Paetzel wrote: On 08/19/2012 14:04, Steven Hartland wrote: HBA's are the way to go if your using ZFS to manage the disks, you only need RAID if your using a FS which doesn't manage the disk side well such as UFS. Its often quite common for RAID controllers to actually be slower vs RAID controllers as the RAID stack can get in the way. Any idea of what kind of performance penalty I might see by using the RAID firmware in JBOD mode vs flashing the IT firmware? I don't have any current numbers, on ZFS v14 14 RAID controllers were actually a bit faster, but that's all changed dramatically. On our high end stuff we can get HBAs to go over 25% faster than high end RAID controllers, like the 9260/9280, but we don't test with RAID controllers anymore at all, so I don't have up to the minute info. ZFS does block checksums, and so do LSI mfi cards, even when in JBOD mode, you also can't bypass the cache on the card without a huge performance hit, so you end up with 256MB or whatever in between your disks and the OS. In addition because the 9240 is based on the 2008 which lacks hardware assist for RAID5/6 those two modes are done in software, so you take another hit there. Advantages: ZFS doesn't work with hot spares as of this moment on FreeBSD, but LSI controllers do, so if your strategy involves hot spares the RAID card is the better choice. LSI controllers can be set to auto-replace, ZFS can't. Enclosure management works better on RAID controllers than through FreeBSD in many cases. Just to clear up, The 9240 is a sas2008 based card with the megaraid software on top of it. In it's default config from LSI the FreeBSD mfi will recognize it in later versions of FreeBSD (The upcoming 9.1 for sure) Older versions of mfi will not recognize it. The card can be flashed with IT firmware and then becomes a 9211 HBA, but it's a bit more expensive than a 9211 is so that doesn't make sense to do in many cases. The price difference was pretty minor when I looked. Confusingly enough, the 9211 HBA also has some RAID capabilities. For me, the biggest advantage of the 9211 would be that it would have allowed me to use shorter cables. On the dmesg posted the firmware on the card is phase 11. This *must* be in lockstep with the driver version or the card may not play nicely. FreeBSD 8.3 and 9.0 have v13 of the driver, the upcoming 9.1 will have v14. Note that v14 fixes a *ton* of stability bugs, including issues where bad drives would hang the controller or prevent systems from booting. Where do those version numbers come from? The mfi driver in 9.0-RELEASE claims to be version 3.00 and the the driver in 9.1 claims to be version 4.23. I was talking about mps, not mfi. The dmesg I was responding to showed an mps. This is what shows up in dmesg on my machine: mfi0: Drake Skinny port 0xce00-0xceff mem 0xfcefc000-0xfcef,0xfce8-0xf ceb irq 18 at device 0.0 on pci1 mfi0: Using MSI mfi0: Megaraid SAS driver Ver 4.23 mfi0: 333 (398082533s/0x0020/info) - Shutdown command received from host mfi0: 334 (boot + 3s/0x0020/info) - Firmware initialization started (PCI ID 0073 /1000/9240/1000) mfi0: 335 (boot + 3s/0x0020/info) - Firmware version 2.70.04-0862 mfi0: 336 (boot + 5s/0x0020/info) - Board Revision 04A mfi0: 337 (boot + 3s/0x0020/info) - Firmware initialization started (PCI ID 0073 /1000/9240/1000) mfi0: 338 (boot + 3s/0x0020/info) - Firmware version 2.70.04-0862 mfi0: 339 (boot + 5s/0x0020/info) - Board Revision 04A mfi0: 340 (boot + 3s/0x0020/info) - Firmware initialization started (PCI ID 0073 /1000/9240/1000) mfi0: 341 (boot + 3s/0x0020/info) - Firmware version 2.70.04-0862 mfi0: 342 (boot + 5s/0x0020/info) - Board Revision 04A mfi0: 343 (boot + 3s/0x0020/info) - Firmware initialization started (PCI ID 0073 /1000/9240/1000) mfi0: 344 (boot + 3s/0x0020/info) - Firmware version 2.70.04-0862 mfi0: 345 (boot + 5s/0x0020/info) - Board Revision 04A mfi0: 346 (398759025s/0x0020/info) - Time established as 08/20/12 6:23:45; (25 seconds since power on) mfi0: 347 (398759051s/0x0020/info) - Time established as 08/20/12 6:24:11; (51 seconds since power on) mfi0: 348 (398759078s/0x0020/WARN) - Patrol Read can't be started, as PDs are ei ther not ONLINE, or are in a VD with an active process, or are in an excluded VD % mfiutil show firmware mfi0 Firmware Package Version: 20.5.1-0003 mfi0 Firmware Images: Name VersionDate Time Status BIOS 4.14.00 active PCLI 03.02-001:#%8 Feb 09 2010 13:09:06 active BCON 4.0-22-e_10-RelMar 11 2010 12:38:08 active NVDT 3.04.03-0002 Apr 05 2010 18:50:27 active APP 2.70.04-0862 May 05 2010 18:12:07 active BTBL 2.01.00.00-0019May 14 2009 15:52:08 active The only firmware file on LSI's web site for the 9240-8i is version
Problem with Linux = 3.3 as NFSv4 server
Hi all, I recently noticed a problem in my network. I use some desktop machines there, two with Linux kernel (Debian and Fedora, both using Kernel 3.5) and a FreeBSD 9.0 (p4) machine. Some days ago, I updated the Debian machine from Kernel 3.2 to Kernel 3.5 (from the experimental branch, 3.5-trunk-amd64; I hope you aren't bothered by the Linux-specific parts). Running 3.2, the FreeBSD machine was able to mount an NFSv4 share on the Debian/k3.2 system properly, with UIDs etc (using nfsuserd on FreeBSD, rpc.idmapd on GNU/Linux). Since I updated to Kernel 3.5, the FreeBSD machine only shows 32767 as UID/GID for all files. `chown` works (even though without any effect, but without error, so nfsuserd works). This behavior occurred also when using Linux Kernel 3.3 or 3.4 on the Debian (server) machine. With the Fedora 17 machine (also Kernel 3.5, and the same users in /etc/passwd, of course), the same operation works without this errors, showing the UIDs and GIDs I want it to. 1. Am I right on this list, or should I ask first on a Linux-oriented list/forum? 2. Has anyone else noticed this or similar behaviour? 3. Any ideas about fixes, workarounds, known bugs? -- norbert ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: LSI 9240-4i 4K alignment
On Sun, Aug 19, 2012 at 7:34 PM, Josh Paetzel j...@tcbug.org wrote: On the dmesg posted the firmware on the card is phase 11. This *must* be in lockstep with the driver version or the card may not play nicely. FreeBSD 8.3 and 9.0 have v13 of the driver, the upcoming 9.1 will have v14. Note that v14 fixes a *ton* of stability bugs, including issues where bad drives would hang the controller or prevent systems from booting. Oof, good to know. I happen to have a server with v9 9211 firmware and v13 mps drivers that's having stability problems under load; it will be interesting to see if updating the firmware solves the problem. Is there someplace I should have been checking to know that this was a requirement? Thanks, Andy ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Problem with Linux = 3.3 as NFSv4 server
Norbert Aschendorff wrote: Hi all, I recently noticed a problem in my network. I use some desktop machines there, two with Linux kernel (Debian and Fedora, both using Kernel 3.5) and a FreeBSD 9.0 (p4) machine. Some days ago, I updated the Debian machine from Kernel 3.2 to Kernel 3.5 (from the experimental branch, 3.5-trunk-amd64; I hope you aren't bothered by the Linux-specific parts). Running 3.2, the FreeBSD machine was able to mount an NFSv4 share on the Debian/k3.2 system properly, with UIDs etc (using nfsuserd on FreeBSD, rpc.idmapd on GNU/Linux). Since I updated to Kernel 3.5, the FreeBSD machine only shows 32767 as UID/GID for all files. `chown` works (even though without any effect, but without error, so nfsuserd works). This behavior occurred also when using Linux Kernel 3.3 or 3.4 on the Debian (server) machine. With the Fedora 17 machine (also Kernel 3.5, and the same users in /etc/passwd, of course), the same operation works without this errors, showing the UIDs and GIDs I want it to. 1. Am I right on this list, or should I ask first on a Linux-oriented list/forum? 2. Has anyone else noticed this or similar behaviour? 3. Any ideas about fixes, workarounds, known bugs? Sounds like rpc.imapd isn't working correctly for that Debian system. To check what's going on, capture some traffic (like an ls -l for a directory) and then look at it in wireshark and see what is going on the wire. The owner and owner_group names in the attributes should look like user@your.dns.domain. Usually the problem is that the domain name isn't set correctly. (Typically, Linux systems default to my.domain or something like that.) rick -- norbert ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
