Re: IPMI SOL serial console wedges
On 2013-11-24, Paul B. Henson hen...@acm.org wrote: Back on topic to my actual problem, it looks like the IPMI SOL com2 is actually using IRQ 10 rather than 5, which both linux and freebsd detect: [2.324044] 00:0e: ttyS2 at I/O 0x3e8 (irq = 10) is a 16550A uart2: 16550 or compatible port 0x3.8-0x3ef irq 10 on acpi0 How come freebsd dynamically detects the correct irq, but openbsd has it hardcoded? linux and freebsd kernels use acpi to configure isa serial ports, openbsd uses static allocations. if you're wondering about the behaviour where you get the kernel messages, and it only stalls later, it just isn't looking at interrupts from the com port until that point.
Re: IPMI SOL serial console wedges
On 2013-11-24, Paul B. Henson hen...@acm.org wrote: erase ^?, werase 1234567890123456 16 chars, because of the 16 byte buffer in the uart. com2 at isa0 port 0x3e8/8 irq 5: ns16550a, 16 byte fifo ^^ when enough chars have been sent to fill the buffer, we wait for an interrupt to say that it's ok to transmit again, but in your case, it's looking for the interrupt on the wrong pin so that interrupt it won't be seen.
Re: IPMI SOL serial console wedges
On Mon, Nov 25, 2013 at 12:09:33PM +, Stuart Henderson wrote: How come freebsd dynamically detects the correct irq, but openbsd has it hardcoded? linux and freebsd kernels use acpi to configure isa serial ports, openbsd uses static allocations. Ah, ok; now that I know what's going on it's easy enough to fix, but it was kinda confusing until I figured out the OS's that were working were using a different interrupt. if you're wondering about the behaviour where you get the kernel messages, and it only stalls later, it just isn't looking at interrupts from the com port until that point. Hmm, I saw kernel messages even after it stalled. All the initial kernel messages were printed, then the first 16 chars from userspace, then nothing else from userspace, but if I did something that resulted in kernel messages (for example, plugging/unplugging the virtual usb cd) those messages still came. when enough chars have been sent to fill the buffer, we wait for an interrupt to say that it's ok to transmit again, but in your case, it's looking for the interrupt on the wrong pin so that interrupt it won't be seen. Just userspace uses the interrupt? The kernel just pushes out characters without waiting for a confirmation the fifo cleared? Thanks much for the explanation...
IPMI SOL serial console wedges
I've got a supermicro X9SCL-F board with ipmi support, and I'm trying to use it for the serial console. It shows up as a third com port. After booting the latest install cd, I run the usual stty com2 115200 and set tty com2, and then boot. The kernel messages show up fine, and then the output just stops: com2 at isa0 port 0x3e8/8 irq 5: ns16550a, 16 byte fifo com2: console [...] root on rd0a swap on rd0b dump on rd0b erase ^?, werase Every time, it wedges up at this spot. The console still works for kernel messages though, if I unplug the ipmi virtual cd, I see that: erase ^?, werasesd2 detached scsibus1 detached umass0 detached I tried installing it via the kvm head and configuring the serial console for the installed OS, same problem, it spews all the kernel messages and then: com2 at isa0 port 0x3e8/8 irq 5: ns16550a, 16 byte fifo com2: console [...] root on sd2a (b158108c15640a28.a) swap on sd2b dump on sd2b Automatic boot i Wedges up. Again, the kernel still works, if I plug in the virtual cd: utomatic boot iumass0 at uhub2 port 1 configuration 1 interface 0 Ours Technology product 0x rev 2.00/2.00 addr 4 umass0: using SCSI over Bulk-Only scsibus3 at umass0: 2 targets, initiator 0 I see the messages. Interesting, the box ends up starting the network and lets me log in via ssh, and when I do, there are *no* getty processes at all. And if I try to reboot from the ssh login, ssh stops responding, but it never reboots and I have to hard reset it. It looks like as soon as userspace touches the console, it freezes up. Nothing from userspace ever gets printed, and I can't type anything on it. But kernel messages still show up. The BIOS has the option to either enable or disable legacy redirection after POST, I've tried it both ways with no difference. I can boot linux and use the ipmi serial console just fine, seeing both the kernel output and userspace output, getting a login prompt, and interacting with no issues, so the underlying SOL is functional. Any ideas what's going on or what I could try to fix or debug it? Thanks much...
Re: IPMI SOL serial console wedges
On Sun, Nov 24, 2013 at 09:54:41AM +0100, Sebastian Benoit wrote: in the bios, you can set the onboard serial ports irq to some higher value. that way, the ipmi console will become com0. (not tried on that board, only on newer supermicros that dont have the serial port on the outside anymore). Hmm, no joy. I tried disabling both the physical serial ports, but the SOL port stayed on the com2 ioport/irq. That completely broke the bootloader, as it showed only com0, and when I told it to use it presumably it tried the standard com0 ioport/irq. I tried setting the bios to use the com1 ioport/irq for com0 and the com2 for com1 in the hopes the SOL port would notice their was a conflict and use something else, but nope, it still didn't work. I don't see anyway to set the SOL port ioport/irq, I fear it's hardcoded. It's weird that the kernel uses com2 as a console fine but then userland borks it. The SOL port works fine as a console when I boot linux on the box, so either there's an openbsd bug with it or linux must be implementing some workaround for a problem. Thanks anyway... Paul B. Henson(hen...@acm.org) on 2013.11.24 00:16:52 -0800: I've got a supermicro X9SCL-F board with ipmi support, and I'm trying to use it for the serial console. It shows up as a third com port. After booting the latest install cd, I run the usual stty com2 115200 and set tty com2, and then boot. The kernel messages show up fine, and then the output just stops: com2 at isa0 port 0x3e8/8 irq 5: ns16550a, 16 byte fifo com2: console [...] root on rd0a swap on rd0b dump on rd0b erase ^?, werase Every time, it wedges up at this spot. The console still works for kernel messages though, if I unplug the ipmi virtual cd, I see that: erase ^?, werasesd2 detached scsibus1 detached umass0 detached I tried installing it via the kvm head and configuring the serial console for the installed OS, same problem, it spews all the kernel messages and then: com2 at isa0 port 0x3e8/8 irq 5: ns16550a, 16 byte fifo com2: console [...] root on sd2a (b158108c15640a28.a) swap on sd2b dump on sd2b Automatic boot i Wedges up. Again, the kernel still works, if I plug in the virtual cd: utomatic boot iumass0 at uhub2 port 1 configuration 1 interface 0 Ours Technology product 0x rev 2.00/2.00 addr 4 umass0: using SCSI over Bulk-Only scsibus3 at umass0: 2 targets, initiator 0 I see the messages. Interesting, the box ends up starting the network and lets me log in via ssh, and when I do, there are *no* getty processes at all. And if I try to reboot from the ssh login, ssh stops responding, but it never reboots and I have to hard reset it. It looks like as soon as userspace touches the console, it freezes up. Nothing from userspace ever gets printed, and I can't type anything on it. But kernel messages still show up. The BIOS has the option to either enable or disable legacy redirection after POST, I've tried it both ways with no difference. I can boot linux and use the ipmi serial console just fine, seeing both the kernel output and userspace output, getting a login prompt, and interacting with no issues, so the underlying SOL is functional. Any ideas what's going on or what I could try to fix or debug it? Thanks much... --
Re: IPMI SOL serial console wedges
On Sun, Nov 24, 2013 at 12:40:31PM -0800, Paul B. Henson wrote: On Sun, Nov 24, 2013 at 09:54:41AM +0100, Sebastian Benoit wrote: in the bios, you can set the onboard serial ports irq to some higher value. that way, the ipmi console will become com0. (not tried on that board, only on newer supermicros that dont have the serial port on the outside anymore). Hmm, no joy. I tried disabling both the physical serial ports, but the SOL port stayed on the com2 ioport/irq. That completely broke the bootloader, as it showed only com0, and when I told it to use it presumably it tried the standard com0 ioport/irq. I tried setting the bios to use the com1 ioport/irq for com0 and the com2 for com1 in the hopes the SOL port would notice their was a conflict and use something else, but nope, it still didn't work. I don't see anyway to set the SOL port ioport/irq, I fear it's hardcoded. It's weird that the kernel uses com2 as a console fine but then userland borks it. The SOL port works fine as a console when I boot linux on the box, so either there's an openbsd bug with it or linux must be implementing some workaround for a problem. Thanks anyway... Supermicro IPMI is crap. Use normal serial console and add a power strip which you can manage via ethernet to poweroff/power cycle the server. jirib
Re: IPMI SOL serial console wedges
On Sun, Nov 24, 2013 at 04:13:27PM -0500, Jiri B wrote: Supermicro IPMI is crap. Use normal serial console and add a power strip which you can manage via ethernet to poweroff/power cycle the server. Well, I can't say it's the greatest implementation ever, but arguably it doesn't seem much worse than on my Sun or IBM servers. The exact same IPMI SOL works fine under linux and illumos, so it doesn't necessarily seem an underlying fault in the implementation. I guess I could try booting freebsd on it and see if that works. If I used a regular serial console, I'd need a terminal server to access it remotely, I'd rather just get this working and avoid the extra parts. I see com2 was actually disabled by default until recently, were there historically problems using com2 under openbsd? Does anybody have a serial console working on a physical com2 port? Maybe it's not the SOL at all but just being com2 that's causing the problem. Unfortunately there doesn't seem to be any way to make the SOL port use a different ioport/irq :(.
Re: IPMI SOL serial console wedges
Supermicro IPMI is crap. Use normal serial console and add a power strip which you can manage via ethernet to poweroff/power cycle the server. Well, I can't say it's the greatest implementation ever, but arguably it doesn't seem much worse than on my Sun or IBM servers. http://news.softpedia.com/news/Rapid7-Researchers-Discover-Vulnerabilities-in-Supermicro-IPMI-Firmware-398010.shtml http://www.darkreading.com/management/new-gaping-security-holes-found-exposing/240157724 [and many many more articles, if you just dug instead of saying arguably] My favorite part is that on some Supermicro machines you have an option to turn it off, but it actually doesn't turn off. Just awesome! We can come up with plausible reasons why that is so, because we read the news. Must be interesting where those machines end up. You just cannot compare this to what Sun did, by (almost always) using a seperate ethernet port. Probably still crap, but at least there was some attempt to provide isolation. At least you can build your own isolation in some way, if you feel compelled to use it. Must be great when people throw a full rack of those machines into a seperate vlan, and one vulnerable machine can screw the others, though. Lessons are just never learned. Is it still arguably fine? The exact same IPMI SOL works fine under linux and illumos, so it doesn't necessarily seem an underlying fault in the implementation. I guess I could try booting freebsd on it and see if that works. works fine If I used a regular serial console, I'd need a terminal server to access it remotely, I'd rather just get this working and avoid the extra parts. avoid the extra parts versus avoid the holes. Similar issues exist on Dell BMC, documented elsewhere, you just need to search. In fact it is unlikely that there are PC server-class machines that have a safe primary ethernet port. It is possible they all have such problems built in (oops). It is really amazing that so many people prefer to remain blissfully unaware. Yummy, ASF driver injection/sniffing, IPMI buffer overflows, BMC cpus without MMUs, SMI, SMIBIOS, shared I2C bus access
Re: IPMI SOL serial console wedges
On 11/24/2013 2:00 PM, Theo de Raadt wrote: Well, I can't say it's the greatest implementation ever, but arguably it doesn't seem much worse than on my Sun or IBM servers. [...] You just cannot compare this to what Sun did, by (almost always) using a seperate ethernet port. Probably still crap, but at least there was some attempt to provide isolation. Actually, my supermicro boxes have a separate dedicated IPMI ethernet port. Only some, not all, supermicro boards use that stupid bridged design that shares a single port for both the primary system ethernet and IPMI ethernet. Is it still arguably fine? What exactly are we arguing about again? My specific statement was not that IPMI was fine, nor even that it wasn't a piece of Swiss cheese; it was that the supermicro implementation is arguably not worse than other manufacturers. Of the articles referenced, the majority of the problems mentioned are in the base IPMI spec, while there were a couple of issues specific to supermicro, for the most part all vendors of IPMI hardware have the same underlying issues, and I'm sure other vendors have their own specific problems as well. If you want to argue that overall IPMI is a sucky specification, and vendors in general have done a crap job of implementing it, you'll have to find somebody else to argue with, because I don't disagree with you. If you want to argue that supermicro in specific has done to a significant extent a crappier job, well, I think you need a bit more evidence. The exact same IPMI SOL works fine under linux and illumos, so it doesn't necessarily seem an underlying fault in the implementation. I guess I could try booting freebsd on it and see if that works. works fine Yes, the IPMI SOL works fine. You can configure it to be a serial console, and the OS boots and functions correctly and successfully. Whether or not IPMI as a platform is a good idea or a secure implementation is really orthogonal to whether or not an OS can use a serial port. Back on topic to my actual problem, it looks like the IPMI SOL com2 is actually using IRQ 10 rather than 5, which both linux and freebsd detect: [2.324044] 00:0e: ttyS2 at I/O 0x3e8 (irq = 10) is a 16550A uart2: 16550 or compatible port 0x3.8-0x3ef irq 10 on acpi0 I assume even if this were a physical serial port it would have the same problem, again nothing to do with IPMI. I'm going to try installing off the vga head again, and then tweaking the kernel to use irq 10 for com2 rather than 5, and hopefully that will let me use the port as the serial console for the installed OS. If that works, presumably the installer could be fixed to work too, but that's probably more trouble than it's worth. How come freebsd dynamically detects the correct irq, but openbsd has it hardcoded? Similar issues exist on Dell BMC, documented elsewhere, you just need to search. Yep, IPMI kinda sucks. Not just on supermicro. In fact it is unlikely that there are PC server-class machines that have a safe primary ethernet port. It is possible they all have such problems built in (oops). Are you talking about systems where the bmc shares a physical port with the OS? That's not what I have. It is really amazing that so many people prefer to remain blissfully unaware. Of what? Potential issues with IPMI? No unawareness here… My boxes all have separate dedicated IPMI ports, all segregated onto a private network that sees no public traffic. Is it perfect? No. From a risk management/cost effectiveness assessment perspective for my specific deployment, is it better than not using it? Yes...
Re: IPMI SOL serial console wedges
On Sun, Nov 24, 2013 at 12:16:52AM -0800, Paul B. Henson wrote: com2 at isa0 port 0x3e8/8 irq 5: ns16550a, 16 byte fifo com2: console [...] root on rd0a swap on rd0b dump on rd0b erase ^?, werase Every time, it wedges up at this spot. The console still works for kernel messages though, if I unplug the ipmi virtual cd, I see that: [...] It looks like as soon as userspace touches the console, it freezes up. Well, it turns out the IPMI SOL com2 uses irq 10 for whatever reason rather than irq 5. Both linux and freebsd successfully detect this: [2.324044] 00:0e: ttyS2 at I/O 0x3e8 (irq = 10) is a 16550A uart2: 16550 or compatible port 0x3.8-0x3ef irq 10 on acpi0 However, openbsd tries to use irq 5, which doesn't turn out so well. I'm guessing the kernel doesn't use the irq, which is why those messages still work, but userspace craps out when it tries to use the wrong irq. I tweaked the kernel with config -e to change the irq for com2 to 10, and it now works fine: com2 at isa0 port 0x3e8/8 irq 10: ns16550a, 16 byte fifo com2: console [...] root on sd2a (b158108c15640a28.a) swap on sd2b dump on sd2b Automatic boot in progress: starting file system checks. setting tty flags pf enabled I suppose the installer kernel could be fixed the same way, but at least for this initial install it's not worth it, I'll just install with the kvm head, fix the installed kernel, and then go serial from there. Is there any particular reason openbsd can't dynamically detect the irq?
Re: IPMI SOL serial console wedges
On Sun, Nov 24, 2013 at 04:10:23PM -0800, Paul B. Henson wrote: I suppose the installer kernel could be fixed the same way, but at least for this initial install it's not worth it, I'll just install with the kvm head, fix the installed kernel, and then go serial from there. Actually, it turned out to be pretty easy to fix the installer. The BIOS redirection works for the initial bootloader, so you can use that to switch the bootloader to use com2, then just boot '-c' to pop up the inkernel editor to fix the irq, and then it all works from there... Sweet..
