Lockups and Semop panics
Hi, I've got a Dell Dimension XPS B866r desktop running as my web/mail server (recently upgraded to 3.9). Occasionally, after a seemingly random amount of time, the machine will do one of two things: it will either lockup/freeze where there are no messages on the console, but you cannot type, ping or access the box in any way (but the reset button) OR, since I have ddb.panic=0, it will reboot with the following message from my serial console: panic: semop - can't undo undos Starting stack trace...panic(fff4,3,e870df10,2,fff4) at panic+0x71panic(d051c1f6,e870dec8,12,d6a4f2d0,3) at panic+0x71 sys_semop(d658b2f0,e870df68,e870df58,0,1998c) at sys_semop+0x492syscall() at syscall+0x2ea --- syscall (number 290) ---0x444e4e9: End of stack trace. syncing disks... 19 19 donerebooting... Since the box rebooted automatically, I don't have ps output, but the box is running: stock GENERIC kernel, mysql-server-5.0.22, stock apache (chrooted) w/ ssl, courier-imap/pop3 w/ssl, imap-proxy, php5-core,curl,gd,imap,mbstring,mcrypt,mysql-5.1.4-hardened (compiled from CVS ports of -current), wordpress-2.0.3, gallery-2.1.1, squirrelmail 1.4.6 and 1.5.1, spamd (openbsd), spamassassin (spamc+spamd+procmail), clamav (clamav-milter + clamd), sendmail (with sasl compiled in), sasl with mysql+crypt patch applied, pure-ftpd (requiring ssl... not really used). When the box comes back up, none of the log files have error messages. Let me know if I can provide any more info. Does anyone have any thoughts? I'm thinking that I'm running too much on my P3-866 w/ 512 MB RAM and should set up another box to offload either mail, apache or mysql. Below is a dmesg. Thanks. OpenBSD 3.9 (GENERIC) #617: Thu Mar 2 02:26:48 MST 2006 [EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC cpu0: Intel Pentium III (GenuineIntel 686-class) 865 MHz cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE real mem = 536190976 (523624K) avail mem = 482246656 (470944K) using 4278 buffers containing 26910720 bytes (26280K) of memory mainbus0 (root) bios0 at mainbus0: AT/286+(00) BIOS, date 03/29/01, BIOS32 rev. 0 @ 0xfda74 apm0 at bios0: Power Management spec V1.2 apm0: AC on, battery charge unknown apm0: flags 30102 dobusy 0 doidle 1 pcibios0 at bios0: rev 2.1 @ 0xf/0x1 pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xf3250/208 (11 entries) pcibios0: PCI Interrupt Router at 000:31:0 (Intel 82371FB ISA rev 0x00) pcibios0: PCI bus #2 is the last bus bios0: ROM list: 0xc/0x8000 0xc8000/0x800 cpu0 at mainbus0 pci0 at mainbus0 bus 0: configuration mode 1 (no bios) pchb0 at pci0 dev 0 function 0 Intel 82820 MCH rev 0x03: rng active, 8Kb/sec ppb0 at pci0 dev 1 function 0 Intel 82820 AGP rev 0x03 pci1 at ppb0 bus 1 ppb1 at pci0 dev 30 function 0 Intel 82801AA Hub-to-PCI rev 0x02 pci2 at ppb1 bus 2 vga1 at pci2 dev 8 function 0 S3 Trio32/64 rev 0x53 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) xl0 at pci2 dev 11 function 0 3Com 3c905C 100Base-TX rev 0x74: irq 10, address 00:50:da:b6:52:f9 bmtphy0 at xl0 phy 24: Broadcom 3C905C internal PHY, rev. 6 ichpcib0 at pci0 dev 31 function 0 Intel 82801AA LPC rev 0x02 pciide0 at pci0 dev 31 function 1 Intel 82801AA IDE rev 0x02: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility wd0 at pciide0 channel 0 drive 0: ST330630A wd0: 16-sector PIO, LBA, 29188MB, 59777640 sectors wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 4 wd1 at pciide0 channel 1 drive 0: ST330013A wd1: 16-sector PIO, LBA, 28629MB, 58633344 sectors atapiscsi0 at pciide0 channel 1 drive 1 scsibus0 at atapiscsi0: 2 targets cd0 at scsibus0 targ 0 lun 0: HITACHI, DVD-ROM GD-5000, 0212 SCSI0 5/cdrom removable wd1(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 4 cd0(pciide0:1:1): using PIO mode 4, DMA mode 2 uhci0 at pci0 dev 31 function 2 Intel 82801AA USB rev 0x02: irq 10 usb0 at uhci0: USB revision 1.0 uhub0 at usb0 uhub0: Intel UHCI root hub, rev 1.00/1.00, addr 1 uhub0: 2 ports with 2 removable, self powered ichiic0 at pci0 dev 31 function 3 Intel 82801AA SMBus rev 0x02: irq 9 iic0 at ichiic0 unknown at iic0 addr 0x18 not configured isa0 at ichpcib0 isadma0 at isa0 pckbc0 at isa0 port 0x60/5 pckbd0 at pckbc0 (kbd slot) pckbc0: using irq 1 for kbd slot wskbd0 at pckbd0: console keyboard, using wsdisplay0 pms0 at pckbc0 (aux slot) pckbc0: using irq 12 for aux slot wsmouse0 at pms0 mux 0 pcppi0 at isa0 port 0x61 midi0 at pcppi0: PC speaker spkr0 at pcppi0 lpt0 at isa0 port 0x378/4 irq 7 npx0 at isa0 port 0xf0/16: using exception 16 pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo pccom0: console fdc0 at isa0 port 0x3f0/6 irq 6 drq 2 fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec biomask ef6d netmask ef6d ttymask ffef pctr: 686-class user-level performance counters enabled mtrr: Pentium Pro MTRR support dkcsum: wd0 matches BIOS drive 0x80 dkcsum: wd1 matches BIOS drive 0x81
Re: Lockups and Semop panics
On Thu, 8 Jun 2006, Michael Erdely wrote: Hi, I've got a Dell Dimension XPS B866r desktop running as my web/mail server (recently upgraded to 3.9). Occasionally, after a seemingly random amount of time, the machine will do one of two things: it will either lockup/freeze where there are no messages on the console, but you cannot type, ping or access the box in any way (but the reset button) OR, since I have ddb.panic=0, it will reboot with the following message from my serial console: panic: semop - can't undo undos Starting stack trace...panic(fff4,3,e870df10,2,fff4) at panic+0x71panic(d051c1f6,e870dec8,12,d6a4f2d0,3) at panic+0x71 sys_semop(d658b2f0,e870df68,e870df58,0,1998c) at sys_semop+0x492syscall() at syscall+0x2ea --- syscall (number 290) ---0x444e4e9: End of stack trace. syncing disks... 19 19 donerebooting... Since the box rebooted automatically, I don't have ps output, but the box is running: stock GENERIC kernel, mysql-server-5.0.22, stock apache (chrooted) w/ ssl, courier-imap/pop3 w/ssl, imap-proxy, php5-core,curl,gd,imap,mbstring,mcrypt,mysql-5.1.4-hardened (compiled from CVS ports of -current), wordpress-2.0.3, gallery-2.1.1, squirrelmail 1.4.6 and 1.5.1, spamd (openbsd), spamassassin (spamc+spamd+procmail), clamav (clamav-milter + clamd), sendmail (with sasl compiled in), sasl with mysql+crypt patch applied, pure-ftpd (requiring ssl... not really used). When the box comes back up, none of the log files have error messages. Let me know if I can provide any more info. Does anyone have any thoughts? I'm thinking that I'm running too much on my P3-866 w/ 512 MB RAM and should set up another box to offload either mail, apache or mysql. your system is running out of semaphore undo data structures. Try increasing at least kern.seminfo.semmnu. Other semaphore related sysctl's might need an increase as well. -Otto Below is a dmesg. Thanks. OpenBSD 3.9 (GENERIC) #617: Thu Mar 2 02:26:48 MST 2006 [EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC cpu0: Intel Pentium III (GenuineIntel 686-class) 865 MHz cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE real mem = 536190976 (523624K) avail mem = 482246656 (470944K) using 4278 buffers containing 26910720 bytes (26280K) of memory mainbus0 (root) bios0 at mainbus0: AT/286+(00) BIOS, date 03/29/01, BIOS32 rev. 0 @ 0xfda74 apm0 at bios0: Power Management spec V1.2 apm0: AC on, battery charge unknown apm0: flags 30102 dobusy 0 doidle 1 pcibios0 at bios0: rev 2.1 @ 0xf/0x1 pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xf3250/208 (11 entries) pcibios0: PCI Interrupt Router at 000:31:0 (Intel 82371FB ISA rev 0x00) pcibios0: PCI bus #2 is the last bus bios0: ROM list: 0xc/0x8000 0xc8000/0x800 cpu0 at mainbus0 pci0 at mainbus0 bus 0: configuration mode 1 (no bios) pchb0 at pci0 dev 0 function 0 Intel 82820 MCH rev 0x03: rng active, 8Kb/sec ppb0 at pci0 dev 1 function 0 Intel 82820 AGP rev 0x03 pci1 at ppb0 bus 1 ppb1 at pci0 dev 30 function 0 Intel 82801AA Hub-to-PCI rev 0x02 pci2 at ppb1 bus 2 vga1 at pci2 dev 8 function 0 S3 Trio32/64 rev 0x53 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) xl0 at pci2 dev 11 function 0 3Com 3c905C 100Base-TX rev 0x74: irq 10, address 00:50:da:b6:52:f9 bmtphy0 at xl0 phy 24: Broadcom 3C905C internal PHY, rev. 6 ichpcib0 at pci0 dev 31 function 0 Intel 82801AA LPC rev 0x02 pciide0 at pci0 dev 31 function 1 Intel 82801AA IDE rev 0x02: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility wd0 at pciide0 channel 0 drive 0: ST330630A wd0: 16-sector PIO, LBA, 29188MB, 59777640 sectors wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 4 wd1 at pciide0 channel 1 drive 0: ST330013A wd1: 16-sector PIO, LBA, 28629MB, 58633344 sectors atapiscsi0 at pciide0 channel 1 drive 1 scsibus0 at atapiscsi0: 2 targets cd0 at scsibus0 targ 0 lun 0: HITACHI, DVD-ROM GD-5000, 0212 SCSI0 5/cdrom removable wd1(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 4 cd0(pciide0:1:1): using PIO mode 4, DMA mode 2 uhci0 at pci0 dev 31 function 2 Intel 82801AA USB rev 0x02: irq 10 usb0 at uhci0: USB revision 1.0 uhub0 at usb0 uhub0: Intel UHCI root hub, rev 1.00/1.00, addr 1 uhub0: 2 ports with 2 removable, self powered ichiic0 at pci0 dev 31 function 3 Intel 82801AA SMBus rev 0x02: irq 9 iic0 at ichiic0 unknown at iic0 addr 0x18 not configured isa0 at ichpcib0 isadma0 at isa0 pckbc0 at isa0 port 0x60/5 pckbd0 at pckbc0 (kbd slot) pckbc0: using irq 1 for kbd slot wskbd0 at pckbd0: console keyboard, using wsdisplay0 pms0 at pckbc0 (aux slot) pckbc0: using irq 12 for aux slot wsmouse0 at pms0 mux 0 pcppi0 at isa0 port 0x61 midi0 at pcppi0: PC speaker spkr0 at pcppi0 lpt0 at isa0 port 0x378/4 irq 7 npx0 at isa0 port 0xf0/16: using exception 16 pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte
Re: Lockups and Semop panics
Otto Moerbeek wrote: On Thu, 8 Jun 2006, Michael Erdely wrote: panic: semop - can't undo undos your system is running out of semaphore undo data structures. Try increasing at least kern.seminfo.semmnu. Other semaphore related sysctl's might need an increase as well. Thanks, Otto. I'll give it a shot (increased semmnu to 40 for starters). -ME -- Support OpenBSD: http://www.openbsd.org/orders.html
Re: Lockups and Semop panics
* Otto Moerbeek [EMAIL PROTECTED] [2006-06-08 10:00]: On Thu, 8 Jun 2006, Michael Erdely wrote: Hi, I've got a Dell Dimension XPS B866r desktop running as my web/mail server (recently upgraded to 3.9). Occasionally, after a seemingly random amount of time, the machine will do one of two things: it will either lockup/freeze where there are no messages on the console, but you cannot type, ping or access the box in any way (but the reset button) OR, since I have ddb.panic=0, it will reboot with the following message from my serial console: panic: semop - can't undo undos Starting stack trace...panic(fff4,3,e870df10,2,fff4) at panic+0x71panic(d051c1f6,e870dec8,12,d6a4f2d0,3) at panic+0x71 sys_semop(d658b2f0,e870df68,e870df58,0,1998c) at sys_semop+0x492syscall() at syscall+0x2ea --- syscall (number 290) ---0x444e4e9: End of stack trace. syncing disks... 19 19 donerebooting... Since the box rebooted automatically, I don't have ps output, but the box is running: stock GENERIC kernel, mysql-server-5.0.22, stock apache (chrooted) w/ ssl, courier-imap/pop3 w/ssl, imap-proxy, php5-core,curl,gd,imap,mbstring,mcrypt,mysql-5.1.4-hardened (compiled from CVS ports of -current), wordpress-2.0.3, gallery-2.1.1, squirrelmail 1.4.6 and 1.5.1, spamd (openbsd), spamassassin (spamc+spamd+procmail), clamav (clamav-milter + clamd), sendmail (with sasl compiled in), sasl with mysql+crypt patch applied, pure-ftpd (requiring ssl... not really used). When the box comes back up, none of the log files have error messages. Let me know if I can provide any more info. Does anyone have any thoughts? I'm thinking that I'm running too much on my your system is running out of semaphore undo data structures. Try increasing at least kern.seminfo.semmnu. Other semaphore related sysctl's might need an increase as well. And mysqueal sucks... Your system may run out of other things as well given the bloatware you're running on it. anything else I have mysqeal on and any significant amount of users I usually have to crank up quite a few of the default limits, both in the kernel and in mysql - I usually build a GENERIC with maxusers=128 to start, and then find the 80 bazillion knobs in mysqeal to turn to tell it to use file descriptors. no really use more than 100 file descriptors. no really really use more than 100 file descriptors If you have a small number of users you may be fine. however seeing what you have there and 512 mb of ram, you may want a bit more if you have significant(1) usage. Did I mention that mysqueal sucks? -Bob 1) Your notion of significant may not be mine - Our similar boxes cluster a load of up to 2500 simultaneous sessions with up to 300 click the thingy at the same time... They've all got 2 gig of ram and there are 7 (I think atm) of them.
Yes, MySQL sucks (was: Re: Lockups and Semop panics)
On Thu, Jun 08, 2006 at 10:19:31AM -0600, Bob Beck wrote: * Otto Moerbeek [EMAIL PROTECTED] [2006-06-08 10:00]: On Thu, 8 Jun 2006, Michael Erdely wrote: Hi, I've got a Dell Dimension XPS B866r desktop running as my web/mail server (recently upgraded to 3.9). Occasionally, after a seemingly random amount of time, the machine will do one of two things: it will either lockup/freeze where there are no messages on the console, but you cannot type, ping or access the box in any way (but the reset button) OR, since I have ddb.panic=0, it will reboot with the following message from my serial console: your system is running out of semaphore undo data structures. Try increasing at least kern.seminfo.semmnu. Other semaphore related sysctl's might need an increase as well. And mysqueal sucks... Your system may run out of other things as well given the bloatware you're running on it. anything else I have mysqeal on and any significant amount of users I usually have to crank up quite a few of the default limits, both in the kernel and in mysql - I usually build a GENERIC with maxusers=128 to start, and then find the 80 bazillion knobs in mysqeal to turn to tell it to use file descriptors. no really use more than 100 file descriptors. no really really use more than 100 file descriptors If you have a small number of users you may be fine. however seeing what you have there and 512 mb of ram, you may want a bit more if you have significant usage. Did I mention that mysqueal sucks? To be fully honest, OpenBSD sucks quite a bit too, in this area (massively threaded servers). There are good reasons for this suckage; but MySQL does not perform quite as well on OpenBSD as it does on almost every other system, so it's not entirely fair to only base one's opinion on that. Of course, I installed it first on Linux, and then found that it was basically a fastish table (not a RDBMS - use PostgreSQL if you want that). With a lot of crud. And, indeed, quite strange configuration options. And which required one to install something like 3.x to build the client part of 4.x, or something similarly inane (supposedly, everyone used binary packages). And which has had quite a few security problems of its own. Finally, it's the wrong solution in almost every case. It's not a database - again, use PostgreSQL. If you only want a table, you could look at SQLite - I never used it, but it's supposedly much, much easier and very fast. Of course, it's even less of an RDBMS than MySQL, but it still works for PHP scripts and it's not like MySQL is much good for anything else. Never mind that PHP scripts are, themselves, better replaced by a proper solution - like FastCGI - written in a proper language - i.e., almost anything but PHP: Perl, Python, Ruby[2], C[1]. Joachim [1] Some of these, especially C, would cost slightly more time, though, which is an issue in some cases. However, I'd assert that anyone not capable of writing in a real language shouldn't be writing internet-exposed programs in the first place. Additionally, I'm not allowed to do this. A good idea, probably - a random PHP script can be understood fairly easily and just plonked down on any commercial hoster; on the other hand, a heavily optimized C program is less likely to be understood... [2] It's been said that Ruby is very slow. I don't know anything about it, really - if this is the case, its place in the list of real languages might need reconsidering.
Re: Yes, MySQL sucks (was: Re: Lockups and Semop panics)
On Thu, Jun 08, 2006 at 10:54:01PM +0200, Joachim Schipper wrote: Did I mention that mysqueal sucks? Finally, it's the wrong solution in almost every case. It's not a database - again, use PostgreSQL. Seconded. MySQL is only an SQLish interface to a file system, it is not a database. Fine for keeping random banner ads in, but that is it. -- Craig Skinner | http://www.kepax.co.uk | [EMAIL PROTECTED]
