No takers for this one? If I missed to provide any important information please let me know. The issue happens regularly on several hardware nodes so if I missed anything I can check it next time it happens.
On Wed, Jul 4, 2012 at 4:16 PM, Rene C. <[email protected]> wrote: > Today I again had a VE that went up to a relative high load for no > apparent reason. > > Below are the details for the hardware node, followed by the high-load > container. > > I realize it's not the latest kernel, but a reboot takes half an hour > (from first VE goes down to last VE is back up, assuming everything goes > well and no FSCK is forced) so we only reboot into new kernels when there > is a really serious reason for it or the server crashes - but I don't see > anything in the kernel updates since our current kernel that would address > this issue anyway. > > Why does the load in this container suddenly go up like that? Websites > hosted by the container becomes very sluggish, so it is a real problem. > > It isn't just a problem with this container - or even this hardware node > for that reason, I occasionally see it with containers on other hardware > nodes as well. One idea I brought up before was that perhaps it's the file > system journal, as suggested in http://wiki.openvz.org/Ploop/Why - but I > think that would affect all containers on that file system, not just a > single container? > > --- HARDWARE NODE --- > > # uname -a > Linux server15.hardwarenode.com 2.6.32-042stab049.6 #1 SMP Mon Feb 6 > 19:17:43 MSK 2012 x86_64 x86_64 x86_64 GNU/Linux > > # rpm -q sl-release > sl-release-6.1-2.x86_64 > > # top -cbn1 | head -17 > top - 21:00:02 up 123 days, 15:31, 1 user, load average: 0.97, 2.70, 2.37 > Tasks: 886 total, 6 running, 880 sleeping, 0 stopped, 0 zombie > Cpu(s): 8.4%us, 1.7%sy, 0.0%ni, 86.3%id, 3.5%wa, 0.0%hi, 0.1%si, > 0.0%st > Mem: 16420716k total, 15566264k used, 854452k free, 1477372k buffers > Swap: 16777184k total, 623672k used, 16153512k free, 4578176k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 94153 27 20 0 164m 41m 3392 S 150.9 0.3 50575:37 > /usr/libexec/mys > 9178 27 20 0 159m 29m 3000 S 72.6 0.2 1284:50 > /usr/libexec/mysq > 567031 apache 20 0 40296 15m 3588 S 17.2 0.1 0:00.09 > /usr/sbin/httpd > 567382 root 20 0 15672 1820 864 R 5.7 0.0 0:00.04 top -cbn1 > 38 root 20 0 0 0 0 S 1.9 0.0 2:55.25 [events/3] > 41 root 20 0 0 0 0 S 1.9 0.0 0:29.00 [events/6] > 566362 apache 20 0 43240 19m 4448 R 1.9 0.1 0:01.04 > /usr/sbin/httpd > 566857 apache 20 0 55248 11m 3456 R 1.9 0.1 0:00.05 > /usr/sbin/httpd > 566918 apache 20 0 42596 17m 3704 S 1.9 0.1 0:00.15 > /usr/sbin/httpd > 567033 apache 20 0 39784 14m 3468 S 1.9 0.1 0:00.01 > /usr/sbin/httpd > > # vzlist -o ctid,laverage > CTID LAVERAGE > 1501 0.00/0.05/0.02 > 1502 0.00/0.00/0.00 > 1503 0.08/0.03/0.01 > 1504 0.00/0.00/0.00 > 1505 8.29/6.04/3.67 > 1506 27.11/16.97/7.89 > 1507 0.00/0.00/0.00 > 1508 0.19/0.06/0.01 > 1509 0.07/0.03/0.00 > 1510 0.02/0.02/0.00 > 1512 0.00/0.00/0.00 > 1514 0.00/0.00/0.00 > > # iostat -xN > Linux 2.6.32-042stab049.6 (server15.hardwarenode.com) 07/03/12 > _x86_64_ (8 CPU) > > avg-cpu: %user %nice %system %iowait %steal %idle > 8.41 0.04 1.75 3.51 0.00 86.28 > > Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz > avgqu-sz await svctm %util > sdd 0.76 56.58 0.59 0.59 20.27 457.28 402.66 > 0.25 211.66 4.03 0.48 > sdc 1.72 27.94 17.20 16.16 887.30 336.18 36.68 > 0.02 12.71 5.23 17.45 > sdb 1.65 27.79 19.48 12.95 975.43 318.64 39.91 > 0.09 15.22 3.77 12.23 > sda 0.01 0.16 0.10 0.24 1.95 2.79 13.79 > 0.00 7.06 4.16 0.14 > vg01-swap 0.00 0.00 0.00 0.00 0.00 0.00 8.00 > 0.00 3.68 2.22 0.00 > vg01-root 0.00 0.00 0.11 0.35 1.94 2.78 10.30 > 0.02 38.30 3.12 0.14 > vg04-swap 0.00 0.00 1.30 0.22 10.41 1.80 8.00 > 0.01 9.28 1.44 0.22 > vg04-vz 0.00 0.00 0.05 56.94 9.86 455.49 8.17 > 0.01 0.18 0.05 0.27 > vg03-swap 0.00 0.00 0.00 0.00 0.00 0.00 8.00 > 0.00 6.72 1.10 0.00 > vg03-vz 0.00 0.00 18.98 42.41 887.30 336.18 19.93 > 0.39 6.33 2.84 17.45 > vg02-swap 0.00 0.00 0.00 0.00 0.00 0.00 8.00 > 0.00 7.03 0.89 0.00 > vg02-vz 0.00 0.00 21.19 39.91 975.43 318.64 21.18 > 0.15 8.99 2.00 12.23 > vg01-vz 0.00 0.00 0.00 0.00 0.00 0.00 7.98 > 0.00 17.73 17.73 0.00 > > --- CONTAINER --- > > # top -cbn1 | head -100 > top - 21:00:04 up 123 days, 15:25, 0 users, load average: 27.11, 16.97, > 7.89 > Tasks: 86 total, 2 running, 84 sleeping, 0 stopped, 0 zombie > Cpu(s): 1.4%us, 0.2%sy, 0.0%ni, 98.1%id, 0.1%wa, 0.0%hi, 0.0%si, > 0.2%st > Mem: 655360k total, 316328k used, 339032k free, 0k buffers > Swap: 1310720k total, 68380k used, 1242340k free, 58268k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 916 mysql 20 0 159m 29m 3000 S 79.3 4.6 1284:51 > /usr/libexec/mysqld > 1 root 20 0 2156 92 64 S 0.0 0.0 0:36.50 init [3] > 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 > [kthreadd/1506] > 3 root 20 0 0 0 0 S 0.0 0.0 0:00.00 [khelper/1506] > 97 root 16 -4 2244 8 4 S 0.0 0.0 0:00.00 /sbin/udevd -d > 634 root 20 0 1812 212 136 S 0.0 0.0 2:39.88 syslogd -m 0 > 667 root 20 0 7180 268 168 S 0.0 0.0 1:01.55 /usr/sbin/sshd > 676 root 20 0 2832 392 304 S 0.0 0.1 0:15.13 xinetd > -stayalive - > 690 root 20 0 6040 124 72 S 0.0 0.0 0:02.45 > /usr/lib/courier-im > 693 root 20 0 4872 252 200 S 0.0 0.0 0:01.94 > /usr/sbin/courierlo > 701 root 20 0 6040 124 72 S 0.0 0.0 0:06.34 > /usr/lib/courier-im > 703 root 20 0 4872 256 200 S 0.0 0.0 0:03.09 > /usr/sbin/courierlo > 709 root 20 0 6040 128 72 S 0.0 0.0 0:18.15 > /usr/lib/courier-im > 711 root 20 0 4872 256 200 S 0.0 0.0 0:09.15 > /usr/sbin/courierlo > 718 root 20 0 6040 124 72 S 0.0 0.0 0:05.68 > /usr/lib/courier-im > 720 root 20 0 4872 252 200 S 0.0 0.0 0:02.54 > /usr/sbin/courierlo > 730 qmails 20 0 1796 224 144 S 0.0 0.0 1:27.21 qmail-send > 732 qmaill 20 0 1752 244 192 S 0.0 0.0 0:22.64 splogger qmail > 733 root 20 0 1780 140 64 S 0.0 0.0 0:07.85 qmail-lspawn > | /usr > 734 qmailr 20 0 1776 148 76 S 0.0 0.0 0:14.07 qmail-rspawn > 735 qmailq 20 0 1748 104 68 S 0.0 0.0 0:14.01 qmail-clean > 781 root 20 0 51880 4364 196 S 0.0 0.7 1:35.02 > /usr/sbin/httpd > 828 named 20 0 44104 5708 1112 S 0.0 0.9 10:10.53 > /usr/sbin/named -u > 866 root 20 0 3708 8 4 S 0.0 0.0 0:00.00 /bin/sh > /usr/bin/my > 981 root 20 0 33912 3756 916 S 0.0 0.6 10:55.30 > /usr/bin/spamd --us > 1107 xfs 20 0 3392 72 40 S 0.0 0.0 0:00.09 xfs -droppriv > -daem > 1115 root 20 0 5672 8 4 S 0.0 0.0 0:00.00 > /usr/sbin/saslauthd > 1116 root 20 0 5672 8 4 S 0.0 0.0 0:00.00 > /usr/sbin/saslauthd > 1122 root 20 0 22992 1868 1084 S 0.0 0.3 2:09.79 > /usr/bin/sw-engine > 1123 root 20 0 27328 1508 1160 S 0.0 0.2 6:06.30 > /usr/local/psa/admi > 7251 root 20 0 4488 192 136 S 0.0 0.0 0:22.85 crond > 9463 apache 20 0 59184 14m 4356 S 0.0 2.3 0:05.10 > /usr/sbin/httpd > 10512 apache 20 0 42316 2504 84 S 0.0 0.4 0:00.91 > /usr/sbin/httpd > 12090 apache 20 0 56964 14m 4492 S 0.0 2.2 0:04.48 > /usr/sbin/httpd > 12682 apache 20 0 61060 17m 4516 S 0.0 2.7 0:02.45 > /usr/sbin/httpd > 13870 sw-cp-se 20 0 7852 1932 16 S 0.0 0.3 1:19.03 > /usr/sbin/sw-cp-ser > 17443 apache 20 0 62416 17m 4436 S 0.0 2.7 0:05.27 > /usr/sbin/httpd > 17461 apache 20 0 52788 10m 4480 S 0.0 1.6 0:02.24 > /usr/sbin/httpd > 20430 apache 20 0 62164 17m 4356 S 0.0 2.7 0:04.25 > /usr/sbin/httpd > 23539 popuser 20 0 37612 25m 2328 S 0.0 3.9 0:01.50 spamd child > 23924 apache 20 0 58004 15m 5536 S 0.0 2.4 0:01.56 > /usr/sbin/httpd > 26361 apache 20 0 54496 11m 3864 S 0.0 1.8 0:01.35 > /usr/sbin/httpd > 26366 apache 20 0 52944 9.8m 3892 S 0.0 1.5 0:01.45 > /usr/sbin/httpd > 26964 apache 20 0 59184 14m 4316 S 0.0 2.3 0:07.26 > /usr/sbin/httpd > 27096 apache 20 0 53728 10m 3868 S 0.0 1.6 0:00.33 > /usr/sbin/httpd > 27102 apache 20 0 54736 11m 3780 S 0.0 1.8 0:00.15 > /usr/sbin/httpd > 27103 apache 20 0 54480 11m 3784 S 0.0 1.7 0:00.11 > /usr/sbin/httpd > 27115 apache 20 0 57064 12m 3816 S 0.0 2.0 0:00.32 > /usr/sbin/httpd > 27118 apache 20 0 53728 10m 3884 S 0.0 1.6 0:01.21 > /usr/sbin/httpd > 27120 apache 20 0 52184 8376 3120 S 0.0 1.3 0:00.00 > /usr/sbin/httpd > 27129 apache 20 0 52168 8072 2960 S 0.0 1.2 0:00.00 > /usr/sbin/httpd > 27139 apache 20 0 53304 9840 3744 S 0.0 1.5 0:01.08 > /usr/sbin/httpd > 27140 apache 20 0 53000 9.8m 3832 S 0.0 1.5 0:00.66 > /usr/sbin/httpd > 27144 apache 20 0 52168 8072 2960 S 0.0 1.2 0:00.00 > /usr/sbin/httpd > 27147 apache 20 0 53252 12m 5536 S 0.0 1.9 0:00.50 > /usr/sbin/httpd > 27149 apache 20 0 52980 9924 3740 S 0.0 1.5 0:00.17 > /usr/sbin/httpd > 27153 apache 20 0 53728 10m 3836 S 0.0 1.6 0:00.49 > /usr/sbin/httpd > 27164 apache 20 0 55224 11m 3812 S 0.0 1.9 0:00.47 > /usr/sbin/httpd > 27171 apache 20 0 52916 9776 3708 S 0.0 1.5 0:00.16 > /usr/sbin/httpd > 27172 apache 20 0 52916 9452 3436 S 0.0 1.4 0:00.17 > /usr/sbin/httpd > 27173 apache 20 0 55340 11m 3720 S 0.0 1.8 0:00.08 > /usr/sbin/httpd > 27179 apache 20 0 52020 7764 2716 S 0.0 1.2 0:00.00 > /usr/sbin/httpd > 27182 apache 20 0 52020 7764 2716 S 0.0 1.2 0:00.00 > /usr/sbin/httpd > 27185 apache 20 0 55224 11m 3824 S 0.0 1.9 0:00.30 > /usr/sbin/httpd > 27186 apache 20 0 53788 10m 3840 S 0.0 1.7 0:00.11 > /usr/sbin/httpd > 27187 apache 20 0 52916 9448 3436 S 0.0 1.4 0:00.08 > /usr/sbin/httpd > 27188 apache 20 0 54628 10m 3504 S 0.0 1.7 0:00.05 > /usr/sbin/httpd > 27196 apache 20 0 53728 10m 3572 S 0.0 1.6 0:00.36 > /usr/sbin/httpd > 27200 apache 20 0 54628 11m 3796 S 0.0 1.7 0:00.05 > /usr/sbin/httpd > 27202 apache 20 0 54480 11m 3796 S 0.0 1.7 0:00.10 > /usr/sbin/httpd > 27204 apache 20 0 53992 10m 3544 S 0.0 1.6 0:00.09 > /usr/sbin/httpd > 27207 apache 20 0 52168 8084 2960 S 0.0 1.2 0:00.00 > /usr/sbin/httpd > 27213 apache 20 0 52020 6464 1788 S 0.0 1.0 0:00.00 > /usr/sbin/httpd > 27214 apache 20 0 54216 10m 3516 S 0.0 1.6 0:00.05 > /usr/sbin/httpd > 27215 apache 20 0 52020 6456 1788 S 0.0 1.0 0:00.00 > /usr/sbin/httpd > 27216 apache 20 0 52020 7860 2804 S 0.0 1.2 0:00.00 > /usr/sbin/httpd > 27218 root 20 0 9400 1900 1408 S 0.0 0.3 0:00.00 crond > 27219 root 20 0 2492 956 848 S 0.0 0.1 0:00.00 /bin/sh -c > /usr/loc > 27220 root 20 0 2496 1052 920 S 0.0 0.2 0:00.00 /bin/sh > /usr/local/ > 27233 root 20 0 2540 1016 892 S 0.0 0.2 0:00.00 /bin/bash -c > top -c > 27234 root 20 0 2284 952 724 R 0.0 0.1 0:00.00 top -cbn1 > 27235 root 20 0 1756 420 352 S 0.0 0.1 0:00.00 head -100 > 27247 root 20 0 2496 452 320 S 0.0 0.1 0:00.00 /bin/sh > /usr/local/ > 27248 root 20 0 8280 1504 1120 R 0.0 0.2 0:00.00 > /usr/bin/mysql -uad > 27249 root 20 0 1800 448 376 S 0.0 0.1 0:00.00 sed -e 1d > 27250 root 20 0 2240 640 540 S 0.0 0.1 0:00.00 awk > {printf("%s", $ > > # netstat -ptan | grep ESTABLISHED > tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:77.87.207.166:21863 > ESTABLISHED 23924/httpd > tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:95.165.204.26:62259 > ESTABLISHED 27144/httpd > tcp 0 0 ::ffff:xx.xx.xx.xx:80 > ::ffff:193.151.105.100:4059ESTABLISHED 27200/httpd > tcp 0 0 ::ffff:xx.xx.xx.xx:80 > ::ffff:109.169.207.68:50087ESTABLISHED 27185/httpd > tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:31.131.70.135:57017 > ESTABLISHED 27179/httpd > tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:95.165.204.26:62220 > ESTABLISHED 27103/httpd > tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:188.134.61.1:60732 > ESTABLISHED 27215/httpd > tcp 0 0 ::ffff:xx.xx.xx.xx:80 > ::ffff:193.151.105.100:4112ESTABLISHED 26964/httpd > tcp 0 0 ::ffff:xx.xx.xx.xx:80 > ::ffff:109.169.207.68:50043ESTABLISHED 27164/httpd > tcp 0 0 ::ffff:xx.xx.xx.xx:80 ::ffff:31.131.70.135:56976 > ESTABLISHED 27153/httpd > > # cat /proc/user_beancounters > Version: 2.5 > uid resource held maxheld > barrier limit failcnt > 1506: kmemsize 27735306 179081216 > 304087040 335544320 0 > lockedpages 0 0 > 81920 81920 0 > privvmpages 393683 430195 > 9223372036854775807 9223372036854775807 0 > shmpages 823 21639 > 9223372036854775807 9223372036854775807 0 > dummy 0 0 > 0 0 0 > numproc 128 204 > 9223372036854775807 9223372036854775807 0 > physpages 79702 163840 > 0 163840 0 > vmguarpages 0 0 > 0 9223372036854775807 0 > oomguarpages 74734 75707 > 0 9223372036854775807 0 > numtcpsock 59 153 > 9223372036854775807 9223372036854775807 0 > numflock 46 62 > 9223372036854775807 9223372036854775807 0 > numpty 0 1 > 9223372036854775807 9223372036854775807 0 > numsiginfo 0 33 > 9223372036854775807 9223372036854775807 0 > tcpsndbuf 1037680 11426176 > 9223372036854775807 9223372036854775807 0 > tcprcvbuf 966656 2867584 > 9223372036854775807 9223372036854775807 0 > othersockbuf 53824 838688 > 9223372036854775807 9223372036854775807 0 > dgramrcvbuf 0 502224 > 9223372036854775807 9223372036854775807 0 > numothersock 114 273 > 9223372036854775807 9223372036854775807 0 > dcachesize 10070617 167772160 > 150994944 167772160 0 > numfile 1634 1865 > 9223372036854775807 9223372036854775807 0 > dummy 0 0 > 0 0 0 > dummy 0 0 > 0 0 0 > dummy 0 0 > 0 0 0 > numiptent 20 20 > 9223372036854775807 9223372036854775807 0 >
_______________________________________________ Users mailing list [email protected] https://openvz.org/mailman/listinfo/users
