Keiichi Yusa created CLOUDSTACK-7951:
----------------------------------------
Summary: cloudstack-agent jsvc gets too large virtual memory space.
Key: CLOUDSTACK-7951
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7951
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: KVM
Affects Versions: 4.3.0
Environment: - Builds up with 155 physical machines and a cloudstack
server.
- Each physical machine equips these.
+ Xeon E5-2680 (10 cores, 20 threads)
+ 128GB memory (and is not set swap space).
+ 600GB Solid State Drive.
- All physical machines are networked by InfiniBand and 10GBase-T.
- Using CentOS6.6 (64bit)
Reporter: Keiichi Yusa
Priority: Critical
cloudstack-agent jsvc gets too large virtual memory space on huge
memory equipped machine.
We have 155 physical machines that each is equipped 128GB RAM and
is installed CentOS6.6 (64bit). On these physical machines, I use
KVM as hypervisor.
I create Compute Offering with 120GB RAM and I deploy VM with this
Compute Offering on this environment.
Now, I have a problem that qemu-kvm process often fails deploying VM
in the above conditions. I confirm that the following message is
outputted in /var/log/libvirt/qemu/i-X-X-VM.log.
{noformat}
"Cannot set up guest memory 'pc.ram': Cannot allocate memory"
{noformat}
I find that cloudstack-agent jsvc gets virtual memory about 35GB.
{noformat}
$ top -a
(snip)
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5787 root 20 0 34.9g 667m 18m S 0.0 0.5 8:45.20 jsvc
(snip)
{noformat}
I suspect that this failure is caused by what cloudstack-agent jsvc
gets too large virtual memory.
I try to patch /etc/init.d/cloudstack-agent as follows:
{noformat}
@@ -66,7 +66,7 @@
start() {
echo -n $"Starting $PROGNAME: "
if hostname --fqdn >/dev/null 2>&1 ; then
- $JSVC -cp "$CLASSPATH" -pidfile "$PIDFILE" \
+ $JSVC -Xms1024m -Xmx2048m -cp "$CLASSPATH" -pidfile "$PIDFILE" \
-errfile $LOGDIR/cloudstack-agent.err -outfile
$LOGDIR/cloudstack-agent.out $CLASS
RETVAL=$?
echo
{noformat}
Then, I restart cloudstack-agent.
As a result, cloudstack agent jsvc reduces virtual memory about 6GB.
Also, qemu-kvm process does not fail deploying VM for now.
{noformat}
$ top -a
(snip)
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
141405 root 20 0 6290m 393m 18m S 0.0 0.3 1:11.44 jsvc
(snip)
{noformat}
If that helps, our CloudStack environment is as follows:
{noformat}
- Builds up with 155 physical machines and a cloudstack server.
- Each physical machine equips these.
+ Xeon E5-2680 (10 cores, 20 threads)
+ 128GB memory (and is not set swap space).
+ 600GB Solid State Drive.
- All physical machines are networked by InfiniBand and 10GBase-T.
- Using CentOS6.6 (64bit)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
