Hello Cloudstack community,
During last week on CloudStack 4.2 I have seen following issue:
CloudStack web interface is not responding in browser. So, I looked to
server side for determining what's going on.
# cat /etc/redhat-release
CentOS release 6.4 (Final)
top - 03:09:43 up 8 days, 14:59, 4 users, load average: 0.99, 0.97, 0.99
Tasks: 234 total, 1 running, 233 sleeping, 0 stopped, 0 zombie
Cpu0 :*100.0%*us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi,
0.0%si, 0.0%st
Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu2 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu4 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu5 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu6 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu7 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
*Mem: 16333844k total, 9968012k used, 6365832k free, 169624k buffers*
Swap: 2097144k total, 976k used, 2096168k free, 5900544k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
*30869* cloud 20 0 7859m *2.2g* 23m S *100.4* 14.0 430:30.61
java
36 root 20 0 0 0 0 S 1.0 0.0 0:48.13
events/1
3833 root 20 0 15160 1436 1000 R 1.0 0.0 0:00.53
top
1 root 20 0 19228 1464 1256 S 0.0 0.0 0:02.48 init
*java* process use 100.4% CPU and 2.2G of memory.
lsof -p 30869 | sort -nk7
COMMAND PID USER FD TYPE DEVICE *SIZE/OFF* NODE NAME
java 30869 cloud 284r REG 8,3 7707959 1841612
/usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/lib/aws-java-sdk-1.3.22.jar
java 30869 cloud mem REG 8,3 7707959 1841612
/usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/lib/aws-java-sdk-1.3.22.jar
java 30869 cloud mem REG 8,3 10252136 1706534
/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/amd64/server/libjvm.so
java 30869 cloud 3r REG 8,3 60484682 1706608
/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/rt.jar
java 30869 cloud mem REG 8,3 60484682 1706608
/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/rt.jar
java 30869 cloud mem REG 8,3 99158576 1183591
/usr/lib/locale/locale-archive
java 30869 cloud 271w REG 8,3 *395727946* 265565
/var/log/cloudstack/management/access_log.2013-11-13.txt
java 30869 cloud 462w REG 8,3 637460040 265558
/var/log/cloudstack/management/apilog.log.2013-11-13
java 30869 cloud 467w REG 8,3 *1144568381* 265563
/var/log/cloudstack/management/management-server.log.2013-11-13
As you can see above *java* process use a lot of memory for log files.
# ls -lha /var/log/cloudstack/management/management-server.log*
-rw-rw-r--. 1 cloud cloud 179K Nov 14 02:59
/var/log/cloudstack/management/management-server.log
-rw-rw-r--. 1 cloud cloud 1.1M Oct 25 00:00
/var/log/cloudstack/management/management-server.log.2013-10-24.gz
-rw-rw-r--. 1 cloud cloud 1.4M Oct 26 00:00
/var/log/cloudstack/management/management-server.log.2013-10-25.gz
-rw-rw-r--. 1 cloud cloud 1.2M Oct 27 00:00
/var/log/cloudstack/management/management-server.log.2013-10-26.gz
-rw-rw-r--. 1 cloud cloud 1.2M Oct 28 00:00
/var/log/cloudstack/management/management-server.log.2013-10-27.gz
-rw-rw-r--. 1 cloud cloud 1.6M Oct 29 00:00
/var/log/cloudstack/management/management-server.log.2013-10-28.gz
-rw-rw-r--. 1 cloud cloud 2.0M Oct 30 00:00
/var/log/cloudstack/management/management-server.log.2013-10-29.gz
-rw-rw-r--. 1 cloud cloud 2.2M Oct 31 00:00
/var/log/cloudstack/management/management-server.log.2013-10-30.gz
-rw-rw-r--. 1 cloud cloud 2.0M Nov 1 00:00
/var/log/cloudstack/management/management-server.log.2013-10-31.gz
-rw-rw-r--. 1 cloud cloud 2.7M Nov 2 00:00
/var/log/cloudstack/management/management-server.log.2013-11-01.gz
-rw-rw-r--. 1 cloud cloud 3.0M Nov 3 00:00
/var/log/cloudstack/management/management-server.log.2013-11-02.gz
-rw-rw-r--. 1 cloud cloud 1.2M Nov 4 00:00
/var/log/cloudstack/management/management-server.log.2013-11-03.gz
-rw-rw-r--. 1 cloud cloud 34M Nov 5 00:01
/var/log/cloudstack/management/management-server.log.2013-11-04.gz
-rw-rw-r--. 1 cloud cloud 35M Nov 6 00:00
/var/log/cloudstack/management/management-server.log.2013-11-05.gz
-rw-rw-r--. 1 cloud cloud 15M Nov 7 00:00
/var/log/cloudstack/management/management-server.log.2013-11-06.gz
-rw-rw-r--. 1 cloud cloud 30M Nov 8 00:00
/var/log/cloudstack/management/management-server.log.2013-11-07.gz
-rw-rw-r--. 1 cloud cloud 1.6M Nov 9 00:00
/var/log/cloudstack/management/management-server.log.2013-11-08.gz
-rw-rw-r--. 1 cloud cloud 1.3M Nov 10 00:00
/var/log/cloudstack/management/management-server.log.2013-11-09.gz
-rw-rw-r--. 1 cloud cloud 18M Nov 11 00:00
/var/log/cloudstack/management/management-server.log.2013-11-10.gz
-rw-rw-r--. 1 cloud cloud 12M Nov 12 00:00
/var/log/cloudstack/management/management-server.log.2013-11-11.gz
-rw-rw-r--. 1 cloud cloud 6.4M Nov 13 00:00
/var/log/cloudstack/management/management-server.log.2013-11-12.gz
*-rw-rw-r--. 1 cloud cloud 1.1G Nov 13 23:45
/var/log/cloudstack/management/management-server.log.2013-11-13*
# ls -lah /var/log/cloudstack/management/apilog*
-rw-rw-r--. 1 cloud cloud 336 Nov 14 00:27
/var/log/cloudstack/management/apilog.log
-rw-rw-r--. 1 cloud cloud 4.6K Oct 24 04:32
/var/log/cloudstack/management/apilog.log.2013-10-18.gz
-rw-rw-r--. 1 cloud cloud 340K Oct 25 03:03
/var/log/cloudstack/management/apilog.log.2013-10-24.gz
-rw-rw-r--. 1 cloud cloud 82K Oct 28 03:04
/var/log/cloudstack/management/apilog.log.2013-10-25.gz
-rw-rw-r--. 1 cloud cloud 708K Oct 29 00:09
/var/log/cloudstack/management/apilog.log.2013-10-28.gz
-rw-rw-r--. 1 cloud cloud 1.1M Oct 30 00:09
/var/log/cloudstack/management/apilog.log.2013-10-29.gz
-rw-rw-r--. 1 cloud cloud 1.5M Oct 31 00:09
/var/log/cloudstack/management/apilog.log.2013-10-30.gz
-rw-rw-r--. 1 cloud cloud 1.7M Nov 1 00:04
/var/log/cloudstack/management/apilog.log.2013-10-31.gz
-rw-rw-r--. 1 cloud cloud 1.7M Nov 2 00:00
/var/log/cloudstack/management/apilog.log.2013-11-01.gz
-rw-rw-r--. 1 cloud cloud 2.6M Nov 3 00:11
/var/log/cloudstack/management/apilog.log.2013-11-02.gz
-rw-rw-r--. 1 cloud cloud 60K Nov 4 02:24
/var/log/cloudstack/management/apilog.log.2013-11-03.gz
-rw-rw-r--. 1 cloud cloud 21M Nov 5 00:00
/var/log/cloudstack/management/apilog.log.2013-11-04.gz
-rw-rw-r--. 1 cloud cloud 23M Nov 6 02:12
/var/log/cloudstack/management/apilog.log.2013-11-05.gz
-rw-rw-r--. 1 cloud cloud 11M Nov 7 02:34
/var/log/cloudstack/management/apilog.log.2013-11-06.gz
-rw-rw-r--. 1 cloud cloud 20M Nov 8 02:54
/var/log/cloudstack/management/apilog.log.2013-11-07.gz
-rw-rw-r--. 1 cloud cloud 1.9M Nov 9 09:07
/var/log/cloudstack/management/apilog.log.2013-11-08.gz
-rw-rw-r--. 1 cloud cloud 268K Nov 10 05:11
/var/log/cloudstack/management/apilog.log.2013-11-09.gz
-rw-rw-r--. 1 cloud cloud 12M Nov 11 00:00
/var/log/cloudstack/management/apilog.log.2013-11-10.gz
-rw-rw-r--. 1 cloud cloud 7.2M Nov 12 00:00
/var/log/cloudstack/management/apilog.log.2013-11-11.gz
-rw-rw-r--. 1 cloud cloud 3.7M Nov 13 03:13
/var/log/cloudstack/management/apilog.log.2013-11-12.gz
*-rw-rw-r--. 1 cloud cloud 608M Nov 13 22:11
/var/log/cloudstack/management/apilog.log.2013-11-13*
For solving this issue I have to:
# kill -9 30869
# /etc/init.d/cloudstack-management restart
Stopping cloudstack-management: [ OK ]
Starting cloudstack-management: [ OK ]
And after that action web interface is working fine again.
Could you please tell me how to resolve or avoid this issue.
Thank you!
