Re: about thumper aka sun fire x4500
On Tuesday 17 January 2012 22.52, Patrick M. Hausen wrote: Hi, all, Am 17.01.2012 um 18:59 schrieb peter h pe...@hk.ipsec.se: I have been beating on of these a few days, i have udes freebsd 9.0 and 8.2 Both fails when i engage 10 disks, the system craches and messages : Hyper transport sync flood will get into the BIOS errorlog ( but nothing will come to syslog since reboot is immediate) Using a zfs radz of 25 disks and typing zpool scrub will bring the system down in seconds. Anyone using a x4500 that can comfirm that it works ? Or is this box broken ? Well, I hate to write that, but ... does it work with the vendor supported [tm] OS? If yes, you can rule out a hardware defect. I would at least try Solaris for this reason. If no, the HW is broken and there is no need to look for a fault on FreeBSD's side. Kind regards, Patrick Yes, this computer stayus alive and works well with nexenta core ( a clone of sun-os) The conclusion is that something is missing in the dealing with hypertransport. A valid question might be : will other systems using hypertransport work or fail ? Is it a smb-issue ? Or is the problem specific for thumper hardware ? -- Peter Håkanson There's never money to do it right, but always money to do it again ... and again ... and again ... and again. ( Det är billigare att göra rätt. Det är dyrt att laga fel. ) ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: about thumper aka sun fire x4500
On Tuesday 17 January 2012 22.52, Patrick M. Hausen wrote: Hi, all, Am 17.01.2012 um 18:59 schrieb peter h pe...@hk.ipsec.se: I have been beating on of these a few days, i have udes freebsd 9.0 and 8.2 Both fails when i engage 10 disks, the system craches and messages : Hyper transport sync flood will get into the BIOS errorlog ( but nothing will come to syslog since reboot is immediate) Using a zfs radz of 25 disks and typing zpool scrub will bring the system down in seconds. Anyone using a x4500 that can comfirm that it works ? Or is this box broken ? Well, I hate to write that, but ... does it work with the vendor supported [tm] OS? If yes, you can rule out a hardware defect. I would at least try Solaris for this reason. If no, the HW is broken and there is no need to look for a fault on FreeBSD's side. Kind regards, Patrick today i installed nexenta ( 134) , built a simular raidz and it _seems_ to stay up. I'll come back when i have made the same pressure on it. -- Peter Håkanson There's never money to do it right, but always money to do it again ... and again ... and again ... and again. ( Det är billigare att göra rätt. Det är dyrt att laga fel. ) ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: about thumper aka sun fire x4500
On 01/17/12 17:09, Jeremy Chadwick wrote: On Tue, Jan 17, 2012 at 06:59:08PM +0100, peter h wrote: I have been beating on of these a few days, i have udes freebsd 9.0 and 8.2 Both fails when i engage 10 disks, the system craches and messages : Hyper transport sync flood will get into the BIOS errorlog ( but nothing will come to syslog since reboot is immediate) Using a zfs radz of 25 disks and typing zpool scrub will bring the system down in seconds. Anyone using a x4500 that can comfirm that it works ? Or is this box broken ? I've seen what is probably the same base issue but on multiple x4100m2 systems running FreeBSD 7 or 8 a few years ago. For me the instant reboot and HT sync flood error happened when I fetched a ~200mb file via HTTP using an onboard intel nic and wrote it out to a simple zfs mirror on 2 disks. I may have tried the nvidia ethernet ports as an alternative but that driver had its own issues at the time. This was never a problem with FFS instead of ZFS. I could repeat it fairly easily by running fetch in a loop (can't remember if writing the output to disk was necessary to trigger it). The workaround I found that worked for me was to buy a cheap intel PCIE nic and use that instead of the onboard ports. If a zpool scrub triggers it for you, I doubt my workaround will help but I wanted to relate my experience. Given this above diagram, I'm sure you can figure out how flooding might occur. :-) I'm not sure what sync flood means (vs. I/O flooding). As I understand it, a sync flood is a purposeful reaction to an error condition as somewhat of a last ditch effort to regain control over the system (which ends up rebooting). I'm pulling this out of my memory from a few years ago. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: about thumper aka sun fire x4500
On Wednesday 18 January 2012 18.15, Adam McDougall wrote: On 01/17/12 17:09, Jeremy Chadwick wrote: On Tue, Jan 17, 2012 at 06:59:08PM +0100, peter h wrote: I have been beating on of these a few days, i have udes freebsd 9.0 and 8.2 Both fails when i engage 10 disks, the system craches and messages : Hyper transport sync flood will get into the BIOS errorlog ( but nothing will come to syslog since reboot is immediate) Using a zfs radz of 25 disks and typing zpool scrub will bring the system down in seconds. Anyone using a x4500 that can comfirm that it works ? Or is this box broken ? I've seen what is probably the same base issue but on multiple x4100m2 systems running FreeBSD 7 or 8 a few years ago. For me the instant reboot and HT sync flood error happened when I fetched a ~200mb file via HTTP using an onboard intel nic and wrote it out to a simple zfs mirror on 2 disks. I may have tried the nvidia ethernet ports as an alternative but that driver had its own issues at the time. This was never a problem with FFS instead of ZFS. I could repeat it fairly easily by running fetch in a loop (can't remember if writing the output to disk was necessary to trigger it). The workaround I found that worked for me was to buy a cheap intel PCIE nic and use that instead of the onboard ports. If a zpool scrub triggers it for you, I doubt my workaround will help but I wanted to relate my experience. The problem i had was most likley the disc-io itself. It was always there whenever a larger number of discs was in motion.It was never there as violent networking ( i even used myri2000 to increase traffic, never a problem) A scrub on the 20-or-so zpool was all that was needed, andn when rebooting the scrub continued and whoops - a new reboot. Sometimes the bios reported not even 16G mem but 10.5 ( which also freebsd noticed) Right now i am torturing the box with same load ( minus myri2000) and sunk-os, i'll report if it does show simular problems. Given this above diagram, I'm sure you can figure out how flooding might occur. :-) I'm not sure what sync flood means (vs. I/O flooding). As I understand it, a sync flood is a purposeful reaction to an error condition as somewhat of a last ditch effort to regain control over the system (which ends up rebooting). I'm pulling this out of my memory from a few years ago. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org -- Peter Håkanson There's never money to do it right, but always money to do it again ... and again ... and again ... and again. ( Det är billigare att göra rätt. Det är dyrt att laga fel. ) ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: about thumper aka sun fire x4500
Hi Peter On 18.01.2012, at 20:25, peter h wrote: On Wednesday 18 January 2012 18.15, Adam McDougall wrote: On 01/17/12 17:09, Jeremy Chadwick wrote: On Tue, Jan 17, 2012 at 06:59:08PM +0100, peter h wrote: I have been beating on of these a few days, i have udes freebsd 9.0 and 8.2 Both fails when i engage 10 disks, the system craches and messages : Hyper transport sync flood will get into the BIOS errorlog ( but nothing will come to syslog since reboot is immediate) Using a zfs radz of 25 disks and typing zpool scrub will bring the system down in seconds. Anyone using a x4500 that can comfirm that it works ? Or is this box broken ? I've seen what is probably the same base issue but on multiple x4100m2 systems running FreeBSD 7 or 8 a few years ago. For me the instant reboot and HT sync flood error happened when I fetched a ~200mb file via HTTP using an onboard intel nic and wrote it out to a simple zfs mirror on 2 disks. I may have tried the nvidia ethernet ports as an alternative but that driver had its own issues at the time. This was never a problem with FFS instead of ZFS. I could repeat it fairly easily by running fetch in a loop (can't remember if writing the output to disk was necessary to trigger it). The workaround I found that worked for me was to buy a cheap intel PCIE nic and use that instead of the onboard ports. If a zpool scrub triggers it for you, I doubt my workaround will help but I wanted to relate my experience. The problem i had was most likley the disc-io itself. It was always there whenever a larger number of discs was in motion.It was never there as violent networking ( i even used myri2000 to increase traffic, never a problem) A scrub on the 20-or-so zpool was all that was needed, andn when rebooting the scrub continued and whoops - a new reboot. Sometimes the bios reported not even 16G mem but 10.5 ( which also freebsd noticed) Right now i am torturing the box with same load ( minus myri2000) and sunk-os, i'll report if it does show simular problems. Given this above diagram, I'm sure you can figure out how flooding might occur. :-) I'm not sure what sync flood means (vs. I/O flooding). As I understand it, a sync flood is a purposeful reaction to an error condition as somewhat of a last ditch effort to regain control over the system (which ends up rebooting). I'm pulling this out of my memory from a few years ago. As Adam has pointed out, a sync flood is a way to signal an error condition on the hyper transport. As I understand it, it's used as a last resort when less fatal means of error communication are no longer possible because of a problem on the transport or a device attached to it. The transport will not recover from this state until it's reset. On Sun AMD systems a reboot is triggered immediately when a sync flood is detected. The fact that it happened is mentioned during POST, but it should also appear in the machine's error logs (IPMI/iLOM), so if you haven't done this already, it might be worth checking them. Maybe you'll find additional information there. You should be able to disable the automatic reset on sync flood in your BIOS settings. We did this on our Sun X4200M2 machines when we experienced sync flood errors. It allowed the kernel to catch an MCE, panic and print out information about the MCE. This might help you get more information about the cause. Our problems with the X4200M2 have some similarties with your case, though in our case high IO (i.e. zpool scrub) did not reliably (read: within minutes or hours) trigger the MCE/sync flood. If we put load on the zpool _and_ the network (em) we could trigger it easily. An other similarity: an other OS (in our case Linux), did not show the symptoms. Even other FreeBSD branches did not trigger the sync flood. You'll find the thread here: http://lists.freebsd.org/pipermail/freebsd-stable/2010-July/057670.html It's a rather long thread. Short version: If raid controller (mpt) interrupts were routed to the first cpu (cpu0) everything worked, if not, sync flood (or MCE) happened on heavy IO. It happens that Linux and even older and newer FreeBSD versions (7.x, 9.x) assigned different interrupt routes for mpt0 compared to the FreeBSD 8.1 we were testing on. So what seemed like a bug of a specific FreeBSD version, because it didn't happen using other FreeBSD versions and Linux, turned out to be a hardware problem after all. IIRC a change in some hardware clock code caused an additional IRQ to be registered on boot (or one less), which reshuffled interrupt assignments compared to older FreeBSD versions we had used successfully on those machines. So we fixed it by setting a tunable which restored old clock behavior and thus old interrupt assignments. It impossible to tell wether you have the same problem. But if you don't see any problems with other operating systems, maybe it's worth to
Re: about thumper aka sun fire x4500
Hi, all, Am 17.01.2012 um 18:59 schrieb peter h pe...@hk.ipsec.se: I have been beating on of these a few days, i have udes freebsd 9.0 and 8.2 Both fails when i engage 10 disks, the system craches and messages : Hyper transport sync flood will get into the BIOS errorlog ( but nothing will come to syslog since reboot is immediate) Using a zfs radz of 25 disks and typing zpool scrub will bring the system down in seconds. Anyone using a x4500 that can comfirm that it works ? Or is this box broken ? Well, I hate to write that, but ... does it work with the vendor supported [tm] OS? If yes, you can rule out a hardware defect. I would at least try Solaris for this reason. If no, the HW is broken and there is no need to look for a fault on FreeBSD's side. Kind regards, Patrick___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: about thumper aka sun fire x4500
On Tue, Jan 17, 2012 at 06:59:08PM +0100, peter h wrote: I have been beating on of these a few days, i have udes freebsd 9.0 and 8.2 Both fails when i engage 10 disks, the system craches and messages : Hyper transport sync flood will get into the BIOS errorlog ( but nothing will come to syslog since reboot is immediate) Using a zfs radz of 25 disks and typing zpool scrub will bring the system down in seconds. Anyone using a x4500 that can comfirm that it works ? Or is this box broken ? I do not have one of these boxes / am not familiar with them, but HyperTransport is an AMD thing. The concept is that it's a bus that interconnects different pieces of a system to the CPU (and thus the memory bus). ASCII diagram coming up: +---+ | RAM | +--++ | +--++ | CPU (w/ on-die MCH) | +--++ | +--++ +-+ | HyperTransport bridge +-+ PCI Express bus (VGA, etc.) | +--++ +-+ | +--+---+ | Southbridge (SATA, etc.) | +--+ ZFS is memory I/O intensive. Your controller, given that it consists of 25 disks, is probably sitting on the PCI Express bus, and thus is generating an equally high amount of I/O. Given this above diagram, I'm sure you can figure out how flooding might occur. :-) I'm not sure what sync flood means (vs. I/O flooding). Googling turns up *tons* of examples of this on the web, except every time they involve people doing overclocking or having CPU-level problems pertaining to voltage. There may be a BIOS option on your system to help curb this behaviour, or at least try to limit it in some way. I know on our AMD systems at work the number of options in the Memory section of the BIOS is quite large, many of which pertaining to interactivity with HyperTransport. If you want my advice? Bring the issue up to Sun. They will almost certainly be able to assign the case to an engineer, who although may not be familiar with FreeBSD, hopefully WILL be familiar with the bus interconnects described above and might be able to help you out. -- | Jeremy Chadwick j...@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: about thumper aka sun fire x4500
On Tue, 17 Jan 2012 18:59:08 +0100, peter h pe...@hk.ipsec.se wrote: I have been beating on of these a few days, i have udes freebsd 9.0 and 8.2 Both fails when i engage 10 disks, the system craches and messages : Hyper transport sync flood will get into the BIOS errorlog ( but nothing will come to syslog since reboot is immediate) Using a zfs radz of 25 disks and typing zpool scrub will bring the system down in seconds. Anyone using a x4500 that can comfirm that it works ? Or is this box broken ? Does it work if you make 3 raid groups of 8 disks and 1 spare? Ronald. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: about thumper aka sun fire x4500
Am 17.01.2012 um 23:09 schrieb Jeremy Chadwick: On Tue, Jan 17, 2012 at 06:59:08PM +0100, peter h wrote: I have been beating on of these a few days, i have udes freebsd 9.0 and 8.2 Both fails when i engage 10 disks, the system craches and messages : Hyper transport sync flood will get into the BIOS errorlog ( but nothing will come to syslog since reboot is immediate) Using a zfs radz of 25 disks and typing zpool scrub will bring the system down in seconds. Anyone using a x4500 that can comfirm that it works ? Or is this box broken ? I do not have one of these boxes / am not familiar with them, but HyperTransport is an AMD thing. The concept is that it's a bus that interconnects different pieces of a system to the CPU (and thus the memory bus). ASCII diagram coming up: Not exactly: http://www.c0t0d0s0.org/archives/1792-Do-it-yourself-X4500.html At the time, there was no similar board on the market, AFAIK. I haven't looked, but I think it should be easier to get one today... ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: about thumper aka sun fire x4500
On Tuesday 17 January 2012 23.15, Ronald Klop wrote: On Tue, 17 Jan 2012 18:59:08 +0100, peter h pe...@hk.ipsec.se wrote: I have been beating on of these a few days, i have udes freebsd 9.0 and 8.2 Both fails when i engage 10 disks, the system craches and messages : Hyper transport sync flood will get into the BIOS errorlog ( but nothing will come to syslog since reboot is immediate) Using a zfs radz of 25 disks and typing zpool scrub will bring the system down in seconds. Anyone using a x4500 that can comfirm that it works ? Or is this box broken ? Does it work if you make 3 raid groups of 8 disks and 1 spare? No, i did not test this. I did some simple ones ( 5 disks in a raidz ) but what i wanted this box to do is a more powerful work. For smaller stuff i use simple hardware I guess i'll buy some supermicro box instead. Ronald. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org -- Peter Håkanson There's never money to do it right, but always money to do it again ... and again ... and again ... and again. ( Det är billigare att göra rätt. Det är dyrt att laga fel. ) ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: about thumper aka sun fire x4500
The X4500s are oldish systems built around a pair Opteron 290 chips with 16 GB RAM and 6 PCI-X Marvell SATA controllers with 8 ports each supporting 48 drives in the machine. Only the first and 4th drive on the I think 4th controller are bootable. Are you using the latest firmware? If not, you're going to have to pay Oracle for the privilege of updating it as there is no way the machines are still under warranty. I'd find a copy of the OpenSolaris Live CD and see if that boots and supports all your drives. Hope this helps you, or helps someone else on the list with more knowledge of debugging older AMD systems point you in the right direction. Bob Healey Systems Administrator Biocomputation and Bioinformatics Constellation and Molecularium hea...@rpi.edu (518) 276-4407 On 1/17/2012 5:09 PM, Jeremy Chadwick wrote: On Tue, Jan 17, 2012 at 06:59:08PM +0100, peter h wrote: I have been beating on of these a few days, i have udes freebsd 9.0 and 8.2 Both fails when i engage 10 disks, the system craches and messages : Hyper transport sync flood will get into the BIOS errorlog ( but nothing will come to syslog since reboot is immediate) Using a zfs radz of 25 disks and typing zpool scrub will bring the system down in seconds. Anyone using a x4500 that can comfirm that it works ? Or is this box broken ? I do not have one of these boxes / am not familiar with them, but HyperTransport is an AMD thing. The concept is that it's a bus that interconnects different pieces of a system to the CPU (and thus the memory bus). ASCII diagram coming up: +---+ | RAM | +--++ | +--++ | CPU (w/ on-die MCH) | +--++ | +--++ +-+ | HyperTransport bridge +-+ PCI Express bus (VGA, etc.) | +--++ +-+ | +--+---+ | Southbridge (SATA, etc.) | +--+ ZFS is memory I/O intensive. Your controller, given that it consists of 25 disks, is probably sitting on the PCI Express bus, and thus is generating an equally high amount of I/O. Given this above diagram, I'm sure you can figure out how flooding might occur. :-) I'm not sure what sync flood means (vs. I/O flooding). Googling turns up *tons* of examples of this on the web, except every time they involve people doing overclocking or having CPU-level problems pertaining to voltage. There may be a BIOS option on your system to help curb this behaviour, or at least try to limit it in some way. I know on our AMD systems at work the number of options in the Memory section of the BIOS is quite large, many of which pertaining to interactivity with HyperTransport. If you want my advice? Bring the issue up to Sun. They will almost certainly be able to assign the case to an engineer, who although may not be familiar with FreeBSD, hopefully WILL be familiar with the bus interconnects described above and might be able to help you out. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: about thumper aka sun fire x4500
Hi all, Am 18.01.2012 um 00:14 schrieb peter h pe...@hk.ipsec.se: On Tuesday 17 January 2012 23.15, Ronald Klop wrote: On Tue, 17 Jan 2012 18:59:08 +0100, peter h pe...@hk.ipsec.se wrote: I have been beating on of these a few days, i have udes freebsd 9.0 and 8.2 Both fails when i engage 10 disks, the system craches and messages : Hyper transport sync flood will get into the BIOS errorlog ( but nothing will come to syslog since reboot is immediate) Using a zfs radz of 25 disks and typing zpool scrub will bring the system down in seconds. Anyone using a x4500 that can comfirm that it works ? Or is this box broken ? Does it work if you make 3 raid groups of 8 disks and 1 spare? No, i did not test this. I did some simple ones ( 5 disks in a raidz ) but what i wanted this box to do is a more powerful work. For smaller stuff i use simple hardware I guess i'll buy some supermicro box instead. But Ronald is right. I apologize for not reading your initial post thoroughly and jumping on your suspicion that the hardware might be to blame. Did you really create one vdev of 25 disks? This is strongly discouraged by the documentation provided by Sun/Oracle. You should (IIRC) never have more than 9 disk in a single vdev. Of course you can join N vdevs of type, say, raidz2 to a single zpool. See for example http://forums.freebsd.org/archive/index.php/t-4641.html I'm writing from my iPad and cannot quickly find the Link to the Sun documentation. Kind regards, HTH, Patrick___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: about thumper aka sun fire x4500
On Jan 17, 2012, at 2:09 PM, Jeremy Chadwick wrote: I do not have one of these boxes / am not familiar with them, but HyperTransport is an AMD thing. The concept is that it's a bus that interconnects different pieces of a system to the CPU (and thus the memory bus). While that was a nice picture, it's not related to the bus architecture of a Sun 4500. :-) An X or E 4500 is a highly fault-tolerant parallel minicomputer with 8 slots-- one was I/O, and you could put up to 7 CPU boards with dual UltraSPARC processors-- you could hot-plug CPU boards and memory in the event of a failure and keep the rest of the system up. They cost a significant fraction of a million dollars circa y2k. A check of some old docs suggests: Hypertransport Sync Flood occurred on last boot: Uncorrectable ECC error caused the last reboot. Regards, -- -Chuck ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: about thumper aka sun fire x4500
On Tue, Jan 17, 2012 at 03:12:19PM -0800, Chuck Swiger wrote: On Jan 17, 2012, at 2:09 PM, Jeremy Chadwick wrote: I do not have one of these boxes / am not familiar with them, but HyperTransport is an AMD thing. The concept is that it's a bus that interconnects different pieces of a system to the CPU (and thus the memory bus). While that was a nice picture, it's not related to the bus architecture of a Sun 4500. :-) An X or E 4500 is a highly fault-tolerant parallel minicomputer with 8 slots-- one was I/O, and you could put up to 7 CPU boards with dual UltraSPARC processors-- you could hot-plug CPU boards and memory in the event of a failure and keep the rest of the system up. They cost a significant fraction of a million dollars circa y2k. You're thinking E4500, which is as you describe. The X4500 is described here: http://en.wikipedia.org/wiki/Sun_Fire_X4500 Marcus ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
