Re: [CentOS] Anyway to ensure SSH availability?

2011-07-01 Thread Emmanuel Noobadmin
On 7/1/11, Les Mikesell lesmikes...@gmail.com wrote: The principle is the same but the way to control it would be different. Spamassassin is a perl program that uses a lot of memory and takes a lot of resources to start up. If you run a lot of copies at once, expect the machine to crawl or

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-30 Thread Steve Barnes
Although it would really be interesting to me to see scheduler settings that would indeed allow something of a 'privileged' ssh or an OOB console that would be responsive even under a punishing load with lots of swapping, which is what the OP originally asked about. I'd be interested to

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-30 Thread Steve Barnes
Although it would really be interesting to me to see scheduler settings that would indeed allow something of a 'privileged' ssh or an OOB console that would be responsive even under a punishing load with lots of swapping, which is what the OP originally asked about. I should add, we have

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-30 Thread Alexander Dalloz
Am 30.06.2011 08:36, schrieb Steve Barnes: Although it would really be interesting to me to see scheduler settings that would indeed allow something of a 'privileged' ssh or an OOB console that would be responsive even under a punishing load with lots of swapping, which is what the OP

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-30 Thread Ljubomir Ljubojevic
Robert Heller wrote: If the machine is a public-facing smtp server, I would look first to see if you are getting the problem I was having. Maybe looking at the maillog to see if the volume of incoming mail is just overwhelming the system. In which case you need to do things to keep sendmail

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-30 Thread Ljubomir Ljubojevic
Steve Barnes wrote: I'd be interested to hear thoughts on this. We have a small 1U test server with 2 entry-level SATA drives that was brought to its knees twice this week by an overzealous Java process. Load averages were up around 60+ and as a result, SSH access would timeout. I don't

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-30 Thread rainer
Steve Barnes wrote: [...] Or maybe having that core root tree on separate HDD and separate HDD controller. Unfortunately, all this does not matter at all. The problem is: sshd is swapped out and the system needs to swap-out something else first, before it can take sshd back in.

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-30 Thread Eero Volotinen
2011/6/30 rai...@ultra-secure.de: Steve Barnes wrote: [...] Or maybe having that core root tree on separate HDD and separate HDD controller. Unfortunately, all this does not matter at all. The problem is: sshd is swapped out and the system needs to swap-out something else first,

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-30 Thread Emmanuel Noobadmin
On 6/30/11, Devin Reade g...@gno.org wrote: I don't recall you mentioning which VM solution you're using. KVM :) Some problematic areas that I've seen when using VMs: + memory ballooning sometimes causes problems (I've not actually seen it, but I've seen various warnings about having it

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-30 Thread Emmanuel Noobadmin
On 6/30/11, rai...@ultra-secure.de rai...@ultra-secure.de wrote: Unfortunately, all this does not matter at all. The problem is: sshd is swapped out and the system needs to swap-out something else first, before it can take sshd back in. There appears to be some functions available to programs

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-30 Thread Emmanuel Noobadmin
On 6/30/11, Simon Matter simon.mat...@invoca.ch wrote: Hm, I thought the problem was I/O, not memory? If memory is not the problem then it has nothing to do with swapping (more correctly paging). After looking through the various replies here and rechecking whatever logs I managed to get, it

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-30 Thread Les Mikesell
On 6/30/11 6:11 AM, Emmanuel Noobadmin wrote: On 6/30/11, Simon Mattersimon.mat...@invoca.ch wrote: Hm, I thought the problem was I/O, not memory? If memory is not the problem then it has nothing to do with swapping (more correctly paging). After looking through the various replies here and

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-30 Thread Steven Tardy
On 06/29/11 14:50, Emmanuel Noobadmin wrote: I was having problems with the same server locking up to the point I can't even get in via SSH. investigate instead of band-aiding... 1) syslog to a remote host. remote syslogging rarely stops when the system is disk/iowait bound. 2) log

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-30 Thread John Hodrien
On Thu, 30 Jun 2011, rai...@ultra-secure.de wrote: Steve Barnes wrote: [...] Or maybe having that core root tree on separate HDD and separate HDD controller. Unfortunately, all this does not matter at all. The problem is: sshd is swapped out and the system needs to swap-out something

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-30 Thread Giovanni Tirloni
On Thu, Jun 30, 2011 at 4:38 AM, Alexander Dalloz ad+li...@uni-x.orgwrote: Am 30.06.2011 08:36, schrieb Steve Barnes: Although it would really be interesting to me to see scheduler settings that would indeed allow something of a 'privileged' ssh or an OOB console that would be responsive

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-30 Thread Emmanuel Noobadmin
On 6/30/11, Les Mikesell lesmikes...@gmail.com wrote: Again, fixable by not sharing the disk the guest uses with the disk the host needs to load programs from... The disk head is always going to be in the wrong place. Well, let's just say my original recommendation specifications for this

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-30 Thread Les Mikesell
On 6/30/2011 12:39 PM, Emmanuel Noobadmin wrote: But, odds are that the source of the problem is starting too many mail delivery programs, especially if they, or the user's local procmail, starts a spamassassin instance per message. Look at the mail logs for a problem time to see if you had

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-30 Thread Les Mikesell
On 6/30/2011 12:39 PM, Emmanuel Noobadmin wrote: Right now it doesn't look like an mail run, more like a httpd run because it's starting to look like a large number of httpd threads was spawned just before that. Oh, one other thing... Do the web programs using mysql for anything? I've seen

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-30 Thread Robert Heller
At Fri, 1 Jul 2011 01:39:19 +0800 CentOS mailing list centos@centos.org wrote: On 6/30/11, Les Mikesell lesmikes...@gmail.com wrote: Again, fixable by not sharing the disk the guest uses with the disk the host needs to load programs from... The disk head is always going to be in the

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-30 Thread John Hinton
On 6/30/2011 4:53 PM, Robert Heller wrote: Right now it doesn't look like an mail run, more like a httpd run because it's starting to look like a large number of httpd threads was spawned just before that. OK, there are probably settings for Apache to run fewer threads. Probably better have

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Rainer Duffner
Am 29.06.2011 um 21:50 schrieb Emmanuel Noobadmin: Since I'm not the only person who face problems trying to remotely access a locked up server, surely somebody must had come up with a solution that didn't involve somebody/something hitting the power button? Yes, it's called out of band

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Max Pyziur
Am 29.06.2011 um 21:50 schrieb Emmanuel Noobadmin: Since I'm not the only person who face problems trying to remotely access a locked up server, surely somebody must had come up with a solution that didn't involve somebody/something hitting the power button? Yes, it's called out of

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Emmanuel Noobadmin
On 6/30/11, Rainer Duffner rai...@ultra-secure.de wrote: Yes, it's called out of band management. Have dial-in access to IPMI/iLO interfaces or just an APC remote controlled power-switch to power-off the server. I don't want to reboot the server everytime something like that happens. I'll

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Rainer Duffner
Am 29.06.2011 um 22:08 schrieb Max Pyziur: Am 29.06.2011 um 21:50 schrieb Emmanuel Noobadmin: Since I'm not the only person who face problems trying to remotely access a locked up server, surely somebody must had come up with a solution that didn't involve somebody/something hitting the

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Rainer Duffner
Am 29.06.2011 um 22:15 schrieb Emmanuel Noobadmin: On 6/30/11, Rainer Duffner rai...@ultra-secure.de wrote: Yes, it's called out of band management. Have dial-in access to IPMI/iLO interfaces or just an APC remote controlled power-switch to power-off the server. I don't want to reboot the

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Giovanni Tirloni
On Wed, Jun 29, 2011 at 4:50 PM, Emmanuel Noobadmin centos.ad...@gmail.comwrote: I was having problems with the same server locking up to the point I can't even get in via SSH. I've already used HTB/TC to reserve bandwidth for my SSH port but the problem now isn't an attack on the bandwidth.

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Emmanuel Noobadmin
On 6/30/11, Rainer Duffner rai...@ultra-secure.de wrote: If it's a server that actually deserves that name, it should have IPMI on board. Problem is some of us work for budget constraints customers and define server by purpose and not specifications. So very often they buy servers based on

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Brian Mathis
On Wed, Jun 29, 2011 at 4:15 PM, Emmanuel Noobadmin centos.ad...@gmail.com wrote: On 6/30/11, Rainer Duffner rai...@ultra-secure.de wrote: Yes, it's called out of band management. Have dial-in access to IPMI/iLO interfaces or just an APC remote controlled power-switch to power-off the server.

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Les Mikesell
On 6/29/2011 3:26 PM, Emmanuel Noobadmin wrote: On 6/30/11, Rainer Duffnerrai...@ultra-secure.de wrote: If it's a server that actually deserves that name, it should have IPMI on board. Problem is some of us work for budget constraints customers and define server by purpose and not

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Robert Heller
At Wed, 29 Jun 2011 16:08:02 -0400 (EDT) CentOS mailing list centos@centos.org wrote: Am 29.06.2011 um 21:50 schrieb Emmanuel Noobadmin: Since I'm not the only person who face problems trying to remotely access a locked up server, surely somebody must had come up with a solution

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Rainer Duffner
Am 29.06.2011 um 22:26 schrieb Emmanuel Noobadmin: On 6/30/11, Rainer Duffner rai...@ultra-secure.de wrote: If it's a server that actually deserves that name, it should have IPMI on board. Problem is some of us work for budget constraints customers and define server by purpose and not

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Keith Keller
On Thu, Jun 30, 2011 at 03:50:30AM +0800, Emmanuel Noobadmin wrote: I was having problems with the same server locking up to the point I can't even get in via SSH. I've already used HTB/TC to reserve bandwidth for my SSH port but the problem now isn't an attack on the bandwidth. So I'm trying

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Les Mikesell
On 6/29/2011 3:43 PM, Rainer Duffner wrote: Virtualization is an option, but the trouble is: if the server is I/O- constrained anyway, virtualization won't help. Everything will just be even slower. That's sort-of true, but you don't have to manage the host through the same interface the

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Paul Heinlein
On Wed, 29 Jun 2011, Keith Keller wrote: In addition to the suggestions already made, one possibility is to attach a serial console or IP KVM. Logging in may still be awful, but at least you won't have to go through sshd. I've been able to log in through a serial getty when sshd was not

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Ljubomir Ljubojevic
Les Mikesell wrote: You can buy add-on PCI-cards for OOB-management, though. Thanks for the information, although unless they are really cheap... The seriously on-the-cheap approach is to run a few virtual servers on hardware slightly better than one of the individual servers would need.

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Emmanuel Noobadmin
On 6/30/11, Les Mikesell lesmikes...@gmail.com wrote: The seriously on-the-cheap approach is to run a few virtual servers on hardware slightly better than one of the individual servers would need. Actually THAT is the fundamental problem ;) The physical server is frankly much more powerful than

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Emmanuel Noobadmin
On 6/30/11, Paul Heinlein heinl...@madboa.com wrote: I actually relied for a while on the last choice. I had a remotely accessible root shell that never logged out. When things got sluggish, I was able to /bin/kill to my heart's content. It wasn't a pretty solution, but it kept me running

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Giovanni Tirloni
On Wed, Jun 29, 2011 at 5:57 PM, Emmanuel Noobadmin centos.ad...@gmail.comwrote: On 6/30/11, Giovanni Tirloni gtirl...@sysdroid.com wrote: I would approach this issue from another perspective: who's locking up the server (as in eating all resources) and how to stop/constrain it. You can

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Emmanuel Noobadmin
On 6/30/11, Robert Heller hel...@deepsoft.com wrote: If the problem is excessive load because Sendmail / Mimedefang / spamd / etc. is too busy handling tons of mail/spam being dumped on your server, you might want to look at these sendmail options: Mail was my first suspect because I had

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Les Mikesell
On 6/29/2011 4:04 PM, Emmanuel Noobadmin wrote: On 6/30/11, Les Mikeselllesmikes...@gmail.com wrote: The seriously on-the-cheap approach is to run a few virtual servers on hardware slightly better than one of the individual servers would need. Actually THAT is the fundamental problem ;) The

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Lamar Owen
On Wednesday, June 29, 2011 04:43:09 PM Rainer Duffner wrote: Virtualization is an option, but the trouble is: if the server is I/O- constrained anyway, virtualization won't help. Everything will just be even slower. That depends. More expensive servers that would be suitable for

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Rainer Duffner
Am 29.06.2011 um 23:17 schrieb Lamar Owen: On Wednesday, June 29, 2011 04:43:09 PM Rainer Duffner wrote: Virtualization is an option, but the trouble is: if the server is I/ O- constrained anyway, virtualization won't help. Everything will just be even slower. That depends. More

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread m . roth
Les Mikesell wrote: On 6/29/2011 4:04 PM, Emmanuel Noobadmin wrote: On 6/30/11, Les Mikeselllesmikes...@gmail.com wrote: The seriously on-the-cheap approach is to run a few virtual servers on hardware slightly better than one of the individual servers would need. Actually THAT is the

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Les Mikesell
On 6/29/2011 4:12 PM, Emmanuel Noobadmin wrote: On 6/30/11, Robert Hellerhel...@deepsoft.com wrote: If the problem is excessive load because Sendmail / Mimedefang / spamd / etc. is too busy handling tons of mail/spam being dumped on your server, you might want to look at these sendmail

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Paul Heinlein
On Thu, 30 Jun 2011, Emmanuel Noobadmin wrote: On 6/30/11, Paul Heinlein heinl...@madboa.com wrote: I actually relied for a while on the last choice. I had a remotely accessible root shell that never logged out. When things got sluggish, I was able to /bin/kill to my heart's content. It

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Emmanuel Noobadmin
On 6/30/11, Giovanni Tirloni gtirl...@sysdroid.com wrote: Linux includes I/O in how it calculates the load average so you're not measuring CPU alone. On the host, it's expected, I've got two qemu-kvm process loading up 100% cpu. Within the guest VM, top looks like this, high load but low cpu %.

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Emmanuel Noobadmin
On 6/30/11, m.r...@5-cent.us m.r...@5-cent.us wrote: Here's another one, that I got from another admin talking to VMware: watch out just how many virtual CPUs you assign to each VM. If you've assigned 4, it is actually going to sit there waiting until it gets 4 virtual CPUs. As of '09, VMware

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Les Mikesell
On 6/29/2011 4:22 PM, m.r...@5-cent.us wrote: Here's another one, that I got from another admin talking to VMware: watch out just how many virtual CPUs you assign to each VM. If you've assigned 4, it is actually going to sit there waiting until it gets 4 virtual CPUs. As of '09, VMware was

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Brian Mathis
On Wed, Jun 29, 2011 at 5:22 PM, m.r...@5-cent.us wrote: Les Mikesell wrote: On 6/29/2011 4:04 PM, Emmanuel Noobadmin wrote: On 6/30/11, Les Mikeselllesmikes...@gmail.com  wrote: The seriously on-the-cheap approach is to run a few virtual servers on hardware slightly better than one of the

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Emmanuel Noobadmin
On 6/30/11, Les Mikesell lesmikes...@gmail.com wrote: OK, but without knowing the cause, you already know the cure. Make the virtual servers not share physical disks - they will always want a single head to be in different places at the same time. Same old problem: budget :D Also, I expect

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread m . roth
Brian Mathis wrote: On Wed, Jun 29, 2011 at 5:22 PM, m.r...@5-cent.us wrote: snip Here's another one, that I got from another admin talking to VMware: watch out just how many virtual CPUs you assign to each VM. If you've assigned 4, it is actually going to sit there waiting until it gets 4

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Devin Reade
--On Thursday, June 30, 2011 04:15:07 AM +0800 Emmanuel Noobadmin centos.ad...@gmail.com wrote: Would ILO work on a server that's unresponsive due to heavy load? ILO or any other OOB solution gives you the functionality of sitting at the console. So if the problem is one that would cause the

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Les Mikesell
On 6/29/2011 4:47 PM, Emmanuel Noobadmin wrote: OK, but without knowing the cause, you already know the cure. Make the virtual servers not share physical disks - they will always want a single head to be in different places at the same time. Same old problem: budget :D If an extra SATA

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Robert Heller
At Thu, 30 Jun 2011 05:12:12 +0800 CentOS mailing list centos@centos.org wrote: On 6/30/11, Robert Heller hel...@deepsoft.com wrote: If the problem is excessive load because Sendmail / Mimedefang / spamd / etc. is too busy handling tons of mail/spam being dumped on your server, you

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Robert Heller
At Thu, 30 Jun 2011 05:31:05 +0800 CentOS mailing list centos@centos.org wrote: On 6/30/11, Giovanni Tirloni gtirl...@sysdroid.com wrote: Linux includes I/O in how it calculates the load average so you're not measuring CPU alone. On the host, it's expected, I've got two qemu-kvm

Re: [CentOS] Anyway to ensure SSH availability?

2011-06-29 Thread Lamar Owen
On Wednesday, June 29, 2011 05:20:26 PM Rainer Duffner wrote: Am 29.06.2011 um 23:17 schrieb Lamar Owen: More expensive servers that would be suitable for virtualization host use also tend to have better I/O subsystems and faster disks. Relative to a 'cheap' system with much poorer