At the bottom of this message is an extract from the monitoring system for the
LUV server. Load average spikes to over 20, but at the time of monitoring
there was only 1 D state process and nothing was using much CPU time or much
RAM. At the same time other VMs didn't report high load so it wasn't an issue
of the disk capacity of the hardware being saturated (which would be difficult
for a RAID array of SSDs on a mostly quiet server).
09:55:01 CPU %user %nice %system %iowait %steal %idle
12:35:02 all 4.28 0.00 1.15 17.87 1.40 75.29
12:45:01 all 4.82 0.00 1.54 17.73 1.57 74.35
12:55:01 all 5.00 0.00 1.20 18.15 0.91 74.75
13:05:01 all 5.60 0.00 1.33 17.79 1.00 74.28
13:15:01 all 14.11 0.00 2.70 16.95 1.56 64.69
13:25:01 all 4.21 0.00 1.52 20.83 0.89 72.56
Average: all 5.15 0.00 1.31 19.99 1.09 72.45
Above is part of the sar output. Note that at around the time of the high
load average there was a higher than usual amount of user CPU time.
The iowait while not correlated with this issue was higher than I expected, I
ran "iotop -o -d5 -b -P" which indicated that writes from mysqld was the main
disk access. I ran "fatrace -f W" which indicated that mysqld was writing to
deleted files in /tmp.
| 68991 | luv_drupal | localhost | luv_drupal | Query | 0 | Creating sort
index | SELECT v.vid, v.*, n.type FROM vocabulary v LEFT JOIN
vocabulary_node_types n ON v.vid = n.vid WHERE |
The only time I caught an access with the "show processlist;" SQL command was
the above, might "Creating sort index" mean writing to deleted files in /tmp?
ALERT itmustbe/loadavg: 21.27 6.84 3.01 >= 7 5 4 (Wed May 13 13:06:39)
Summary output : 21.27 6.84 3.01 >= 7 5 4
Detailed text (if any) follows:
-------------------------------
Here are D state processes:
USER PID VSZ RSS TTY COMMAND
root 157 0B 0B [jbd2/vda-8]
Here are processes with the top CPU percentages:
USER PID CPU TTY COMMAND
mon 1630 7.0 /usr/bin/perl /usr/lib/mon/mon-local.d/
loadavg.monitor 7 5 4
mon 1629 6.0 /usr/bin/perl /usr/lib/mon/mon.d/msql-
mysql.monitor --mode m
www-data 1445 5.1 /usr/sbin/apache2 -k start
Here are processes with the top RAM use:
USER PID VIRT RES TTY COMMAND
clamav 335 1.12GB 874MB /usr/sbin/clamd --foreground=true
mysql 1392 1.7GB 613MB /usr/sbin/mysqld --basedir=/usr --
datadir=/var/lib/mysql --p
spamassassin 20347 101MB 89.3MB spamd child
spamassassin 29351 97MB 84.9MB spamd child
root 1047 94.5MB 84.1MB /usr/bin/perl -T -w /usr/sbin/spamd -d
--pidfile=/var/run/sp
Swap Used: 7.8MB / 256MB
--
My Main Blog http://etbe.coker.com.au/
My Documents Blog http://doc.coker.com.au/
_______________________________________________
luv-main mailing list
luv-main@luv.asn.au
https://lists.luv.asn.au/cgi-bin/mailman/listinfo/luv-main
