[
https://issues.apache.org/jira/browse/CLOUDSTACK-7951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14221941#comment-14221941
]
Rohit Yadav commented on CLOUDSTACK-7951:
-----------------------------------------
[~edison]
There is a (potential) issue with the mix; the Xms value is 1GB that means the
initial heap size for JVM is that value (this fails for people using DevCloud
etc.); and the maximum heap size Xmx is set to 2GB so mgmt server may never
use/exploit more memory even if the server has some. Additional memory is
useful for a lot of dynamically resizable internal data-structures such as
temporary data structures, maps and lists so should we impose such limits?
> cloudstack-agent jsvc gets too large virtual memory space.
> ----------------------------------------------------------
>
> Key: CLOUDSTACK-7951
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7951
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM
> Affects Versions: 4.3.0
> Environment: - Builds up with 155 physical machines and a cloudstack
> server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads).
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
> Reporter: Keiichi Yusa
> Priority: Critical
> Fix For: Future, 4.3.0, 4.4.0, 4.5.0, 4.6.0
>
>
> cloudstack-agent jsvc gets too large virtual memory space on huge
> memory equipped machine.
> We have 155 physical machines that each is equipped 128GB RAM and
> is installed CentOS6.6 (64bit). On these physical machines, I use
> KVM as hypervisor.
> I create Compute Offering with 120GB RAM and I deploy VM with this
> Compute Offering on this environment.
> Now, I have a problem that qemu-kvm process often fails deploying VM
> in the above conditions. I confirm that the following message is
> outputted in /var/log/libvirt/qemu/i-X-X-VM.log.
> {noformat}
> "Cannot set up guest memory 'pc.ram': Cannot allocate memory"
> {noformat}
> I find that cloudstack-agent jsvc gets virtual memory about 35GB.
> {noformat}
> $ top -a
> (snip)
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 5787 root 20 0 34.9g 667m 18m S 0.0 0.5 8:45.20 jsvc
> (snip)
> {noformat}
>
> I suspect that this failure is caused by what cloudstack-agent jsvc
> gets too large virtual memory.
> I try to patch /etc/init.d/cloudstack-agent as follows:
> {noformat}
> @@ -66,7 +66,7 @@
> start() {
> echo -n $"Starting $PROGNAME: "
> if hostname --fqdn >/dev/null 2>&1 ; then
> - $JSVC -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> + $JSVC -Xms1024m -Xmx2048m -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> -errfile $LOGDIR/cloudstack-agent.err -outfile
> $LOGDIR/cloudstack-agent.out $CLASS
> RETVAL=$?
> echo
> {noformat}
> Then, I restart cloudstack-agent.
> As a result, cloudstack agent jsvc reduces virtual memory about 6GB.
> Also, qemu-kvm process does not fail deploying VM for now.
> {noformat}
> $ top -a
> (snip)
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 141405 root 20 0 6290m 393m 18m S 0.0 0.3 1:11.44 jsvc
> (snip)
> {noformat}
> If that helps, our CloudStack environment is as follows:
> {noformat}
> - Builds up with 155 physical machines and a cloudstack server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads)
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
