[
https://issues.apache.org/jira/browse/CLOUDSTACK-9375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271301#comment-15271301
]
Carles Figuerola commented on CLOUDSTACK-9375:
----------------------------------------------
Thanks for looking into this.
I don't think it's a resource limit as the first thing we tried when it crashed
is allocating more heap on the process (coredumps are 12G now, the one I
analyzed was 6.7G)
{code}
sh-4.1$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 257427
max locked memory (kbytes, -l) 44040192
max memory size (kbytes, -m) unlimited
open files (-n) 4096
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
{code}
As for the threads, it's currently sitting at 236 (it was at 276 but I
restarted it to have a clean slate). I'm going to send this metric and the
memory usage by the process to graphite to monitor them.
> Cloudstack Management Memory Leak on org.apache.coyote.RequestInfo
> ------------------------------------------------------------------
>
> Key: CLOUDSTACK-9375
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9375
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server
> Affects Versions: 4.5.2
> Environment: CentOS release 6.7 (Final)
> Reporter: Carles Figuerola
>
> After deploying ACS (cloudstack-management-4.5.2-1.el6.x86_64), using a 4
> management cluster, the management pid keeps crashing with out of memory
> errors. Eclipse Memory Analyzer points to this class
> (org.apache.coyote.RequestInfo) as the culpable one:
> {code}
> 66,923 instances of "org.apache.coyote.RequestInfo", loaded by
> "org.apache.catalina.loader.StandardClassLoader @ 0x600026b98" occupy
> 6,227,824,472 (94.91%) bytes. These instances are referenced from one
> instance of "java.lang.Object[]", loaded by "<system class loader>"
> {code}
> Strangely, this is not affecting a smaller installation (2 management, 90
> hosts, ~2000 VMs) that is running ACS 4.5.2 over CentOS 6.7 or a clone of the
> broken one (4 management, 300 hosts, ~4000 VMs) running ACS 4.5.2 over Ubuntu
> 12.04.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
