> I stopped logging so much in haproxy, but I get the same thing if I
> grep the nginx logs on this server: haproxy's mongrel backend checks
> start failing. I've noticed it only happens when using httpchk (or at
> least it happens much, much more quickly).
>
> Here's an iostat I ran -- the first two are during the grep on the
> nginx logs; the last one is after I finished:
The iostat looks ok.
Cut-n-past the following (or run from a script) so we can get a better idea
of the box's general load and to see if they turn up anything:
cat /proc/interrupts
free
netstat --inet -n | awk '{ print $6 }' | sort | uniq -c
ulimit -a
vmstat 1 10 & ( sleep 5 ; grep whatever /var/log/messages >/dev/null )
cat /proc/interrupts
echo lsof count `lsof | wc -l`
What type of disk subsystem do you have? Given how it chokes when doing a
grep, it almost sounds like you might have a faulty driver. You do realize
8 cores is overkill for this, unless you are running other stuff on the box.
The two checks on the interrupts is to see if something (especially disk
I/O) is generating too many as we need to look at the difference.
