Re: Really stuck and help needed of resources depletions on web servers.
Maurice Janssen wrote: Now, why PERL would do this, I have no clue, but it does anyway in the usage done by awstats. So far I reproduce this 5 times, so it's pretty consistent. What may cause this, I do not know more, but look like when PERL needs to process huge amount of data, it end up affecting the server in ways to make it crash/freeze. I don't think that Perl is the problem. Other programs would probably also be able to crash the machine, if the load is high enough. So far 5x it's always happen with PERL kicking in and starting the load. Does it survive 'make build' or orther disk/cpu/mem intensive task? Yes no problem there. I did that test, rebuilt kernel and full userland. No problem. I also have the same problem with the same version on an IBM e326 as well that run PERL a lots and the box freeze and need hard reset. That IBM only run MySQL and nothing else as a cnam server ofr VoIP, but PERL is use for the handling of the connection for VoIP SIP from the outside. Crash three time so far. Different boxes, different applications, but common point is PERL so far. That's really all I have, but both will be wiped out soon and 4.1 put in place and will see. Just find it weird that PREL is the only common point on AMD64 and both are running OpenBSD 3.9 (GENERIC.MP) if that have anything to do with it.
Re: Really stuck and help needed of resources depletions on web servers.
On Monday, May 7, 2007 at 11:27:50 -0400, Daniel Ouellet wrote: >Maurice Janssen wrote: >>On Monday, May 7, 2007 at 03:11:41 -0400, Daniel Ouellet wrote: >>>Every time, I process the logs with webalizer, no problem what so ever. >>>Then a few customers wants the awstats version. So, I process that as >>>well, however it's also processing multiple logs, but when the awstats >>>PERL stuff kicks in, it does get the resources to the roof and badly so, >>>that so far it had the impact of freezing the server as a results of this. >>> >>>Now, why PERL would do this, I have no clue, but it does anyway in the >>>usage done by awstats. >>> >>>So far I reproduce this 5 times, so it's pretty consistent. >>> >>>What may cause this, I do not know more, but look like when PERL needs >>>to process huge amount of data, it end up affecting the server in ways >>>to make it crash/freeze. >> >>I don't think that Perl is the problem. Other programs would probably >>also be able to crash the machine, if the load is high enough. > >So far 5x it's always happen with PERL kicking in and starting the load. Does it survive 'make build' or orther disk/cpu/mem intensive task? >>It could be a bug in the sparc64 port or bad RAM or some other hardware >>related problem. I've seen some strange behaviour [1] with sparc64 as >>well, but I'm not sure wether this is due to a bug. > >This is AMD64 Sorry, for some reason sparc64 was in my mind. Maurice
Re: Really stuck and help needed of resources depletions on web servers.
Maurice Janssen wrote: On Monday, May 7, 2007 at 03:11:41 -0400, Daniel Ouellet wrote: Every time, I process the logs with webalizer, no problem what so ever. Then a few customers wants the awstats version. So, I process that as well, however it's also processing multiple logs, but when the awstats PERL stuff kicks in, it does get the resources to the roof and badly so, that so far it had the impact of freezing the server as a results of this. Now, why PERL would do this, I have no clue, but it does anyway in the usage done by awstats. So far I reproduce this 5 times, so it's pretty consistent. What may cause this, I do not know more, but look like when PERL needs to process huge amount of data, it end up affecting the server in ways to make it crash/freeze. I don't think that Perl is the problem. Other programs would probably also be able to crash the machine, if the load is high enough. So far 5x it's always happen with PERL kicking in and starting the load. It could be a bug in the sparc64 port or bad RAM or some other hardware related problem. I've seen some strange behaviour [1] with sparc64 as well, but I'm not sure wether this is due to a bug. This is AMD64 and the RAM, I already replaced it 4 days ago to be sure with brand new one out of the box that I order last week.
Re: Really stuck and help needed of resources depletions on web servers.
On Monday, May 7, 2007 at 03:11:41 -0400, Daniel Ouellet wrote: >Every time, I process the logs with webalizer, no problem what so ever. >Then a few customers wants the awstats version. So, I process that as >well, however it's also processing multiple logs, but when the awstats >PERL stuff kicks in, it does get the resources to the roof and badly so, >that so far it had the impact of freezing the server as a results of this. > >Now, why PERL would do this, I have no clue, but it does anyway in the >usage done by awstats. > >So far I reproduce this 5 times, so it's pretty consistent. > >What may cause this, I do not know more, but look like when PERL needs >to process huge amount of data, it end up affecting the server in ways >to make it crash/freeze. I don't think that Perl is the problem. Other programs would probably also be able to crash the machine, if the load is high enough. It could be a bug in the sparc64 port or bad RAM or some other hardware related problem. I've seen some strange behaviour [1] with sparc64 as well, but I'm not sure wether this is due to a bug. >No,w that I was able to isolate the cause I will proceed the upgrades to >4.0 as I still don't have my CD for 4.1 yet, so I can't do that. It was You can use the 4.1-release from the FTP-servers (or even 4.1-stable from ftp://ftp.su.se/pub/mirrors/openbsd_stable/ ). Maurice [1] 'make build' fails somewhere in gnu/usr.bin/binutils/ with missing header files when /usr/obj is NFS-mounted. It works fine when /usr/obj is on a local FS.
Re: Really stuck and help needed of resources depletions on web servers.
OK, I have some update on this one. It's not fix, but I was finally able to isolate how that problem is trigger. May be fix now, I don't know, but I am passing the informations in case it's useful and also if someone could tell me if there was a logic behind it and if yes it would not happen in newer release. What happen is so far 5 times I have the server crash, or I should say, freeze and sometime with the display showing > extend_alloc_supregion: can't allocated region Spelling may be wrong a bit as it was from a voice mail I give myself as I didn't have a pen to right down with and there wasn't anyway to access the keyboard, or console. All was frozen and no key was doing anyway. This is on 3.9 and the dmesg was send before on this thread. Now what's going on is the server stop responding, no access, no console, no keyboard, only reset will bring it back. But the broadcom network cards still answer to ping, so it didn't allow my CARP setup to kick in sadly here. Now I did put in place a few trap and logs to try to see what's causing this as I had some ideas before, but wasn't sure to pass that along. However now I am. Each time, all 5, it was cause when I have a script that run well for 4 years, but as traffic grow on this web server so does the logs as well. Every time, I process the logs with webalizer, no problem what so ever. Then a few customers wants the awstats version. So, I process that as well, however it's also processing multiple logs, but when the awstats PERL stuff kicks in, it does get the resources to the roof and badly so, that so far it had the impact of freezing the server as a results of this. Now, why PERL would do this, I have no clue, but it does anyway in the usage done by awstats. So far I reproduce this 5 times, so it's pretty consistent. What may cause this, I do not know more, but look like when PERL needs to process huge amount of data, it end up affecting the server in ways to make it crash/freeze. No,w that I was able to isolate the cause I will proceed the upgrades to 4.0 as I still don't have my CD for 4.1 yet, so I can't do that. It was order a long time ago, but with a book as well that was taken out now. So, I expect that to be the cause of the delay. Anyway, any feedback as to how PERL may cause this and what may be done to avoid this? Hope this is useful to some and if not, then sorry for the noise. Best, Daniel
Really stuck and help needed of resources depletions on web servers.
Hi, I need some help to find out what I can do to address this problem. I did research, but so far can't get where I need to go and I am more stuck then usual and time is the essence this time around. I am adding more server resources tonight, but still need help as it can't keep up with the demand so far. Any help would be more then welcome as I am running like a chicken with his head cutoff now. I am not sure that I am looking at the right thing and run out of ideas, so I am setting more capacity with additional servers, however, that really will take lots of time and I am in a bind. Any clue stick would be very much appreciated right now! I am running 3.9 for now on a web server and today four times it crash and nothing could be done on the keyboard or anything else then reboot the server. Traffic was lots heavier then usual. First look like PF run out of resources, or something, I disable PF to see if that help and I thought it was it, but then it happen again. Crash. I am running 3.9 and the dmesg is below. The only thing I had on the screen was this: 95% sure, as I couldn't do much and no pen, etc. started with a bunch of error message on the screen with: extend_alloc_supregion: can't allocated region extend_alloc_supregion: can't allocated region extend_alloc_supregion: can't allocated region extend_alloc_supregion: can't allocated region extend_alloc_supregion: can't allocated region extend_alloc_supregion: can't allocated region end then I had this one: pf_test: pf_get_frag returned NULL pf_test: pf_get_frag returned NULL pf_test: pf_get_frag returned NULL pf_test: pf_get_frag returned NULL pf_test: pf_get_frag returned NULL pf_test: pf_get_frag returned NULL pf_test: pf_get_frag returned NULL pf_test: pf_get_frag returned NULL Anything I can do to help this situation? I am kind of stuck, so any ideas woud;l be more then welcome. === OpenBSD 3.9 (GENERIC) #462: Thu Mar 2 03:52:16 MST 2006 [EMAIL PROTECTED]:/usr/src/sys/arch/amd64/compile/GENERIC real mem = 1073278976 (1048124K) avail mem = 908550144 (887256K) using 22937 buffers containing 107536384 bytes (105016K) of memory mainbus0 (root) cpu0 at mainbus0: (uniprocessor) cpu0: AMD Opteron(tm) Processor 242, 1593.82 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,NXE,MMXX,LONG,3DNOW2,3DNOW cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 1MB 64b/line 16-way L2 cache cpu0: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative cpu0: DTLB 32 4KB entries fully associative, 8 4MB entries fully associative pci0 at mainbus0 bus 0: configuration mode 1 ppb0 at pci0 dev 1 function 0 "AMD 8131 PCIX" rev 0x12 pci1 at ppb0 bus 3 "AMD 8131 PCIX IOAPIC" rev 0x01 at pci0 dev 1 function 1 not configured ppb1 at pci0 dev 2 function 0 "AMD 8131 PCIX" rev 0x12 pci2 at ppb1 bus 2 bge0 at pci2 dev 3 function 0 "Broadcom BCM5704C" rev 0x02, BCM5704 A2 (0x2002): irq 10, address 00:00:1a:19:55:df brgphy0 at bge0 phy 1: BCM5704 10/100/1000baseT PHY, rev. 0 bge1 at pci2 dev 3 function 1 "Broadcom BCM5704C" rev 0x02, BCM5704 A2 (0x2002): irq 11, address 00:00:1a:19:55:de brgphy1 at bge1 phy 1: BCM5704 10/100/1000baseT PHY, rev. 0 "AMD 8131 PCIX IOAPIC" rev 0x01 at pci0 dev 2 function 1 not configured ppb2 at pci0 dev 6 function 0 "AMD 8111 PCI-PCI" rev 0x07 pci3 at ppb2 bus 1 ohci0 at pci3 dev 0 function 0 "AMD 8111 USB" rev 0x0b: irq 10, version 1.0, legacy support usb0 at ohci0: USB revision 1.0 uhub0 at usb0 uhub0: AMD OHCI root hub, rev 1.00/1.00, addr 1 uhub0: 3 ports with 3 removable, self powered ohci1 at pci3 dev 0 function 1 "AMD 8111 USB" rev 0x0b: irq 10, version 1.0, legacy support usb1 at ohci1: USB revision 1.0 uhub1 at usb1 uhub1: AMD OHCI root hub, rev 1.00/1.00, addr 1 uhub1: 3 ports with 3 removable, self powered vga1 at pci3 dev 4 function 0 "ATI Rage XL" rev 0x27 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) pcib0 at pci0 dev 7 function 0 "AMD AMD8111 LPC" rev 0x05 pciide0 at pci0 dev 7 function 1 "AMD 8111 IDE" rev 0x03: DMA, channel 0 configured to compatibility, channel 1 configured to compatibility wd0 at pciide0 channel 0 drive 0: wd0: 16-sector PIO, LBA, 38166MB, 78165360 sectors wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 6 pciide0: channel 1 disabled (no drives) amdpm0 at pci0 dev 7 function 3 "AMD 8111 Power" rev 0x05: rng active iic0 at amdpm0 pchb0 at pci0 dev 24 function 0 "AMD AMD64 HyperTransport" rev 0x00 pchb1 at pci0 dev 24 function 1 "AMD AMD64 Address Map" rev 0x00 pchb2 at pci0 dev 24 function 2 "AMD AMD64 DRAM Cfg" rev 0x00 pchb3 at pci0 dev 24 function 3 "AMD AMD64 Misc Cfg" rev 0x00 isa0 at pcib0 isadma0 at isa0 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo pckbc0 at isa0 port 0x60/5 pckbd0 at pckbc0 (kbd slot) pckbc0: using irq 1 for kbd slot wskbd0 at pckbd0: console keyboard, using wsdisplay0
