Re: Server crashing, no explanations
Have you tried disabling hyperthreading in the BIOS? I had the same problem with a herd of dell 2850s and the only fix under the conditions you describe seemed to be to disable the hyperthreading. Regards, Sent via my BlackBerry from Vodacom - let your email find you! -Original Message- From: Alan Gilmour [EMAIL PROTECTED] Date: Wed, 21 May 2008 09:46:57 To:Christian Zachariasen [EMAIL PROTECTED] Cc:FreeBSD Mailing Lists freebsd-questions@freebsd.org Subject: Re: Server crashing, no explanations I guess it can, but in the past when hardware has failed for me, I generally got some indicative errors in the logs. I managed to move a lot of the intensive operations across to another server and for the moment,its working much better on the other server which has a newer kernel. Cheers Alan On 21/05/2008, Christian Zachariasen [EMAIL PROTECTED] wrote: On Tue, May 20, 2008 at 8:34 PM, Wojciech Puchar [EMAIL PROTECTED] wrote: Then, if crash dumps are enabled, it could be a HW failure.. no it is not. i have similar problems but not with apache, it is certainly FreeBSD bug that causes it to randomly reboot under certain types of load. i found the way to fix it in my case ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] Yeah, because hardware never fails, right Wojciech? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] -- This message, together with any attachments, is for the confidential and exclusive use of the intended addressee(s). If you receive it in error, please delete the message. All information contained within this e-mail is without prejudice. Do not disclose, copy, circulate or use any information contained herein. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Server crashing, no explanations
On Tue, May 20, 2008 at 8:34 PM, Wojciech Puchar [EMAIL PROTECTED] wrote: Then, if crash dumps are enabled, it could be a HW failure.. no it is not. i have similar problems but not with apache, it is certainly FreeBSD bug that causes it to randomly reboot under certain types of load. i found the way to fix it in my case ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] Yeah, because hardware never fails, right Wojciech? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Server crashing, no explanations
I guess it can, but in the past when hardware has failed for me, I generally got some indicative errors in the logs. I managed to move a lot of the intensive operations across to another server and for the moment,its working much better on the other server which has a newer kernel. Cheers Alan On 21/05/2008, Christian Zachariasen [EMAIL PROTECTED] wrote: On Tue, May 20, 2008 at 8:34 PM, Wojciech Puchar [EMAIL PROTECTED] wrote: Then, if crash dumps are enabled, it could be a HW failure.. no it is not. i have similar problems but not with apache, it is certainly FreeBSD bug that causes it to randomly reboot under certain types of load. i found the way to fix it in my case ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] Yeah, because hardware never fails, right Wojciech? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] -- This message, together with any attachments, is for the confidential and exclusive use of the intended addressee(s). If you receive it in error, please delete the message. All information contained within this e-mail is without prejudice. Do not disclose, copy, circulate or use any information contained herein. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Server crashing, no explanations
I guess it can, but in the past when hardware has failed for me, I generally got some indicative errors in the logs. hardware failures are different. rarely causes reboot, or reboots ramdomly independent of what you do. with bad memory it usually produces sig11 or similar errors much more often than rebooting. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Server crashing, no explanations
___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] Yeah, because hardware never fails, right Wojciech? it does. but this is software failure. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Server crashing, no explanations
On Wed, May 21, 2008 at 6:11 PM, Wojciech Puchar [EMAIL PROTECTED] wrote: I guess it can, but in the past when hardware has failed for me, I generally got some indicative errors in the logs. hardware failures are different. rarely causes reboot, or reboots ramdomly independent of what you do. with bad memory it usually produces sig11 or similar errors much more often than rebooting. heavy load, heats the cpu, cpu reaches upper temp limit set in bios, computer reboots without warning to OS, nothing in logs, nothing recorded in bios, no crashdump cos the os didn't crash. I've seen it happen. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Server crashing, no explanations
On May 20, 2008, at 7:17 AM, Alan Gilmour wrote: Hey all, We have recently been getting a lot of traffic to one of our sites. The CPU is consistently during busy periods using 100% utilisation. When this happens we have approx 150 apache threads, and the loads goes way above 15. However recently the server has been auto-restarting (when under heavy load) with no explanation in any logs. I've checked the console log, messages, db logs e.t.c. but no mention of anything wrong. Brief server summary : FreeBSD 6.3-STABLE #0: CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2800.11-MHz 686-class CPU) Logical CPUs per core: 2 real memory = 17716740096 (16896 MB) avail memory = 16837763072 (16057 MB) FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs We tried installing mbmon and lmmon and healthd, but none seem to work. Anyone got any suggestions for other things we can try to detect why the server is failing? or other ways to check things like CPU temp and memory status? We have experienced this since 6.x began and it's not hardware. It can be reproduced by moving the role to another similar server. When the role is changed and the traffic (not necessarily the load), the problem goes away or rather, will transfer to the new box. Look at the thread named zonealarm issues on Freebsd-Net a couple of months ago. You may find it will apply but there aren't any answers there yet. I gather that people need more data collection. I have never figured out how to get a dump though people have recommended things to try over the last couple of years. I was hoping 7.0 would be the solution but I'm told it's not. Reduce your traffic and the problem will go away. Split the traffic to more than one server is a way to do this. We increased our uptime drastically by doing this but we still get hit hard enough at times to go down. During our low traffic periods of the year, we simply stay up all the time (in the hottest days of summer). By the way, the symptom I see is never immediate reboot, it will hang for reasonable period of time prior to rebooting. As I monitor ours 24/7, I reset power on the box before it reboots to reduce the outage to customers. If I'm not watching it eventually will reboot. Brutal but it works. Realize it's possible you don't have this problem but there are a few of us who do. It has something to do with buffers not being freed up. Cheers Alan ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions- [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Server crashing, no explanations
On May 21, 2008, at 8:05 AM, Chris Pratt wrote: On May 20, 2008, at 7:17 AM, Alan Gilmour wrote: Hey all, We have recently been getting a lot of traffic to one of our sites. The CPU is consistently during busy periods using 100% utilisation. When this happens we have approx 150 apache threads, and the loads goes way above 15. However recently the server has been auto-restarting (when under heavy load) with no explanation in any logs. I've checked the console log, messages, db logs e.t.c. but no mention of anything wrong. Brief server summary : FreeBSD 6.3-STABLE #0: CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2800.11-MHz 686-class CPU) Logical CPUs per core: 2 real memory = 17716740096 (16896 MB) avail memory = 16837763072 (16057 MB) FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs We tried installing mbmon and lmmon and healthd, but none seem to work. Anyone got any suggestions for other things we can try to detect why the server is failing? or other ways to check things like CPU temp and memory status? We have experienced this since 6.x began and it's not hardware. It can be reproduced by moving the role to another similar server. When the role is changed and the traffic (not necessarily the load), the problem goes away or rather, will transfer to the new box. Look at the thread named zonealarm issues on Freebsd-Net a BIG CORRECTION: zonelimit issues (geez, I hadn't touched a windows product in 3 years, no idea where that came from, sorry). couple of months ago. You may find it will apply but there aren't any answers there yet. I gather that people need more data collection. I have never figured out how to get a dump though people have recommended things to try over the last couple of years. I was hoping 7.0 would be the solution but I'm told it's not. Reduce your traffic and the problem will go away. Split the traffic to more than one server is a way to do this. We increased our uptime drastically by doing this but we still get hit hard enough at times to go down. During our low traffic periods of the year, we simply stay up all the time (in the hottest days of summer). By the way, the symptom I see is never immediate reboot, it will hang for reasonable period of time prior to rebooting. As I monitor ours 24/7, I reset power on the box before it reboots to reduce the outage to customers. If I'm not watching it eventually will reboot. Brutal but it works. Realize it's possible you don't have this problem but there are a few of us who do. It has something to do with buffers not being freed up. Cheers Alan ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions- [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions- [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Server crashing, no explanations
Hey all, We have recently been getting a lot of traffic to one of our sites. The CPU is consistently during busy periods using 100% utilisation. When this happens we have approx 150 apache threads, and the loads goes way above 15. However recently the server has been auto-restarting (when under heavy load) with no explanation in any logs. I've checked the console log, messages, db logs e.t.c. but no mention of anything wrong. Brief server summary : FreeBSD 6.3-STABLE #0: CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2800.11-MHz 686-class CPU) Logical CPUs per core: 2 real memory = 17716740096 (16896 MB) avail memory = 16837763072 (16057 MB) FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs We tried installing mbmon and lmmon and healthd, but none seem to work. Anyone got any suggestions for other things we can try to detect why the server is failing? or other ways to check things like CPU temp and memory status? Cheers Alan ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Server crashing, no explanations
Crash dumps should help. Alan Gilmour wrote: Hey all, We have recently been getting a lot of traffic to one of our sites. The CPU is consistently during busy periods using 100% utilisation. When this happens we have approx 150 apache threads, and the loads goes way above 15. However recently the server has been auto-restarting (when under heavy load) with no explanation in any logs. I've checked the console log, messages, db logs e.t.c. but no mention of anything wrong. Brief server summary : FreeBSD 6.3-STABLE #0: CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2800.11-MHz 686-class CPU) Logical CPUs per core: 2 real memory = 17716740096 (16896 MB) avail memory = 16837763072 (16057 MB) FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs We tried installing mbmon and lmmon and healthd, but none seem to work. Anyone got any suggestions for other things we can try to detect why the server is failing? or other ways to check things like CPU temp and memory status? Cheers Alan ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Server crashing, no explanations
Hi Roberto, There's nothing in /var/crash Any other ideas? Cheers Alan On 20/05/2008, Roberto Nunnari [EMAIL PROTECTED] wrote: Crash dumps should help. Alan Gilmour wrote: Hey all, We have recently been getting a lot of traffic to one of our sites. The CPU is consistently during busy periods using 100% utilisation. When this happens we have approx 150 apache threads, and the loads goes way above 15. However recently the server has been auto-restarting (when under heavy load) with no explanation in any logs. I've checked the console log, messages, db logs e.t.c. but no mention of anything wrong. Brief server summary : FreeBSD 6.3-STABLE #0: CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2800.11-MHz 686-class CPU) Logical CPUs per core: 2 real memory = 17716740096 (16896 MB) avail memory = 16837763072 (16057 MB) FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs We tried installing mbmon and lmmon and healthd, but none seem to work. Anyone got any suggestions for other things we can try to detect why the server is failing? or other ways to check things like CPU temp and memory status? Cheers Alan ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] -- This message, together with any attachments, is for the confidential and exclusive use of the intended addressee(s). If you receive it in error, please delete the message. All information contained within this e-mail is without prejudice. Do not disclose, copy, circulate or use any information contained herein. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Server crashing, no explanations
Alan Gilmour wrote: We tried installing mbmon and lmmon and healthd, but none seem to work. How so? Do you have an error message to share with us? Anyone got any suggestions for other things we can try to detect why the server is failing? or other ways to check things like CPU temp and memory status? What is the hardware vendor? Since most of the major players have decent systems management capability and cards for this sort of thing (think RSA for IBM, DRAC for Dell, etc). If you are using RAID verify the disks are OK (both physical and logical). Enable full memory check at POST (not quick) Try diagnostics such as what comes with UBCD for memory disk. http://www.ultimatebootcd.com/ Is this system just like any others at your site or a one-off? -- Some days it's just not worth chewing through the restraints... Mark D. Foster, CISSP [EMAIL PROTECTED] http://mark.foster.cc/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Server crashing, no explanations
Hi Mark, I'm guessing there's not a compatible chip on the motherboard for both of those items. Its a DELL server and unfortunately we only actually have one disk on it at the moment, so there's no RAID (which is also worrying :( ) Full memory checks have been on in the past and never reported any errors at the time. May be worth switching back on I guess. Alan On 20/05/2008, Mark Foster [EMAIL PROTECTED] wrote: Alan Gilmour wrote: We tried installing mbmon and lmmon and healthd, but none seem to work. How so? Do you have an error message to share with us? Anyone got any suggestions for other things we can try to detect why the server is failing? or other ways to check things like CPU temp and memory status? What is the hardware vendor? Since most of the major players have decent systems management capability and cards for this sort of thing (think RSA for IBM, DRAC for Dell, etc). If you are using RAID verify the disks are OK (both physical and logical). Enable full memory check at POST (not quick) Try diagnostics such as what comes with UBCD for memory disk. http://www.ultimatebootcd.com/ Is this system just like any others at your site or a one-off? -- Some days it's just not worth chewing through the restraints... Mark D. Foster, CISSP [EMAIL PROTECTED] http://mark.foster.cc/ -- This message, together with any attachments, is for the confidential and exclusive use of the intended addressee(s). If you receive it in error, please delete the message. All information contained within this e-mail is without prejudice. Do not disclose, copy, circulate or use any information contained herein. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Server crashing, no explanations
At 09:17 AM 5/20/2008, Alan Gilmour wrote: Hey all, We have recently been getting a lot of traffic to one of our sites. The CPU is consistently during busy periods using 100% utilisation. When this happens we have approx 150 apache threads, and the loads goes way above 15. However recently the server has been auto-restarting (when under heavy load) with no explanation in any logs. I've checked the console log, messages, db logs e.t.c. but no mention of anything wrong. Brief server summary : FreeBSD 6.3-STABLE #0: CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2800.11-MHz 686-class CPU) Logical CPUs per core: 2 real memory = 17716740096 (16896 MB) avail memory = 16837763072 (16057 MB) FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs We tried installing mbmon and lmmon and healthd, but none seem to work. Anyone got any suggestions for other things we can try to detect why the server is failing? or other ways to check things like CPU temp and memory status? Cheers Alan Alan, Have you run the complete dell diagnostics? Also I would run the diagnostics for the hard drive from that manufacturer as well. Do you have any issues with power to this server? Does it have a redundant power supply installed? Any issues with heat? -Derek -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Server crashing, no explanations
the server is failing? or other ways to check things like CPU temp and memory status? What is the hardware vendor? Since most of the major players have decent no it is not hardware problem. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Server crashing, no explanations
Then, if crash dumps are enabled, it could be a HW failure.. Once I had a server crashing with no dumps, and I found out it was I power supply problem. Best regards. Robi Alan Gilmour wrote: Hi Roberto, There's nothing in /var/crash Any other ideas? Cheers Alan On 20/05/2008, Roberto Nunnari [EMAIL PROTECTED] wrote: Crash dumps should help. Alan Gilmour wrote: Hey all, We have recently been getting a lot of traffic to one of our sites. The CPU is consistently during busy periods using 100% utilisation. When this happens we have approx 150 apache threads, and the loads goes way above 15. However recently the server has been auto-restarting (when under heavy load) with no explanation in any logs. I've checked the console log, messages, db logs e.t.c. but no mention of anything wrong. Brief server summary : FreeBSD 6.3-STABLE #0: CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2800.11-MHz 686-class CPU) Logical CPUs per core: 2 real memory = 17716740096 (16896 MB) avail memory = 16837763072 (16057 MB) FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs We tried installing mbmon and lmmon and healthd, but none seem to work. Anyone got any suggestions for other things we can try to detect why the server is failing? or other ways to check things like CPU temp and memory status? Cheers Alan ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: Server crashing, no explanations
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Roberto Nunnari Sent: Tuesday, May 20, 2008 10:49 AM To: Alan Gilmour Cc: freebsd-questions@freebsd.org Subject: Re: Server crashing, no explanations Then, if crash dumps are enabled, it could be a HW failure.. Once I had a server crashing with no dumps, and I found out it was I power supply problem. Best regards. Robi Same here: my aging Dell with ECC Rambus (yes, Rambus) memory was rebooting without producing a panic and thus without a chance for a crash-dump, because one memory bank on the system was failing. However, the BIOS on the system was nice enough to tell me about the memory errors and even tell me which module it thought it was. That may be something you should check on your system (meaning: BIOS error log - if you happen to have one of those). Cheers - Tobias ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Server crashing, no explanations
Hi Tobias, Yes the plan is the next time it crashes is to check BIOS, unfortunately I need to get hosts to do this as I can't physically access the machine at this point, but they have some KVM like switch attached to it. Cheers Alan On 20/05/2008, Tobias Hoellrich [EMAIL PROTECTED] wrote: -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Roberto Nunnari Sent: Tuesday, May 20, 2008 10:49 AM To: Alan Gilmour Cc: freebsd-questions@freebsd.org Subject: Re: Server crashing, no explanations Then, if crash dumps are enabled, it could be a HW failure.. Once I had a server crashing with no dumps, and I found out it was I power supply problem. Best regards. Robi Same here: my aging Dell with ECC Rambus (yes, Rambus) memory was rebooting without producing a panic and thus without a chance for a crash-dump, because one memory bank on the system was failing. However, the BIOS on the system was nice enough to tell me about the memory errors and even tell me which module it thought it was. That may be something you should check on your system (meaning: BIOS error log - if you happen to have one of those). Cheers - Tobias ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] -- This message, together with any attachments, is for the confidential and exclusive use of the intended addressee(s). If you receive it in error, please delete the message. All information contained within this e-mail is without prejudice. Do not disclose, copy, circulate or use any information contained herein. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Server crashing, no explanations
Then, if crash dumps are enabled, it could be a HW failure.. no it is not. i have similar problems but not with apache, it is certainly FreeBSD bug that causes it to randomly reboot under certain types of load. i found the way to fix it in my case ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
