No we didn't, it wouldn't matter because the memory would still fill up, the problem is it opens a thread and it fails to close it so whatever you will increase soon or later the memory will fill up (if I understand right)
The error in catalina is as follows: SEVERE: The web application [/client] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1bd66d2d]) and a value of type [com.cloud.api.SerializationContext] (value [com.cloud.api.SerializationContext@2f6baed9]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak. If someone could help with this error generated in the catalina log, that would be much appreicated. Kind Regards Amin -----Original Message----- From: Michael Phillips [mailto:mphilli7...@hotmail.com] Sent: Thursday, 3 April 2014 9:34 AM To: dev@cloudstack.apache.org Subject: RE: Interesting 4.2.1. Issue... A few other articles also mention setting the initial heap size "-Xms" to the same value as the heap size, to go ahead and fully commit the heap. Have you tried that? One other thing I am curious of is have you fiddled with the Perm Size "XX:Permsize" setting? > Subject: RE: Interesting 4.2.1. Issue... > From: a...@opencloud.net.au > To: dev@cloudstack.apache.org > Date: Thu, 3 Apr 2014 09:26:31 +0800 > > Believe or not!! We have tried setting them in both formats and still > the catalina log produces java heap space error > > Kind Regards > Amin > > > > -----Original Message----- > From: Michael Phillips [mailto:mphilli7...@hotmail.com] > Sent: Thursday, 3 April 2014 9:24 AM > To: dev@cloudstack.apache.org > Subject: RE: Interesting 4.2.1. Issue... > > This may be a silly question, but in all of the docs I am reading online in > regards to increasing the java heap size, they are specifying it as > -Xmx"size""MB" example -Xmx2048m vs -Xmx2g Is it possible that it's not > reading the 2g variable as 2GB? > > > Subject: RE: Interesting 4.2.1. Issue... > > From: a...@opencloud.net.au > > To: dev@cloudstack.apache.org > > Date: Thu, 3 Apr 2014 09:06:17 +0800 > > > > It is 6.0.35 and it still produces this error, even after increasing the > > Xmx to 4g, we have installed tomcat 6.0.33 and each time we install the > > cloudstack it does not sense the installed 6.0.33 and attempts to install > > 6.0.35 as it is dependent on it. Silly solution is that we scheduled a > > daily restart @ 2PM through a cron job!!!! But first you have to "killall > > jvsc" then restart the management server. > > > > What we are considering is to migrate the management server to CentOS 6.5 > > it comes with tomcat 6.0.24 and mysql 5.1, we attempted to restore the dump > > on a pilot environment and it worked. > > > > If someone else has a better solution than this would you please share it? > > > > Kind Regards > > Amin > > > > -----Original Message----- > > From: Michael Phillips [mailto:mphilli7...@hotmail.com] > > Sent: Thursday, 3 April 2014 5:31 AM > > To: dev@cloudstack.apache.org > > Subject: RE: Interesting 4.2.1. Issue... > > > > So I just checked my tomcat version and we are running 6.0.35 which must be > > the default that comes with Ubuntu 12.04 out of the box. Our memory > > settings are as follows: > > JAVA_OPTS="-Djava.awt.headless=true > > -Dcom.sun.management.jmxremote.port=45219 > > -Dcom.sun.management.jmxremote.authenticate=false > > -Dcom.sun.management.jmxremote.ssl=false -Xmx4g > > -XX:+HeapDumpOnOutOfMemoryError > > -XX:HeapDumpPath=/var/log/cloudstack/management/ -XX:PermSize=512M > > -XX:MaxPermSize=800m" > > So what version of tomcat are you running 6.0.35 or 6.0.33? > > > > > Subject: RE: Interesting 4.2.1. Issue... > > > From: a...@opencloud.net.au > > > To: dev@cloudstack.apache.org > > > Date: Tue, 1 Apr 2014 11:23:57 +0800 > > > > > > We have the same issue, after an upgrade from 2.2.14 to 4.2.1, and > > > during this upgrade we had upgrade from Ubuntu 10 LTS to Ubuntu 12 > > > LTS, it seems it related to tomcat 6.0.35, because it is > > > recommended to have tomcat 6.0.33 which doesn't come by default > > > with Ubuntu > > > 12.04.4 > > > > > > Kind Regards > > > Amin > > > > > > > > > > > > -----Original Message----- > > > From: Marcus [mailto:shadow...@gmail.com] > > > Sent: Tuesday, 1 April 2014 6:22 AM > > > To: dev@cloudstack.apache.org > > > Subject: Re: Interesting 4.2.1. Issue... > > > > > > I'm running 3 mgmt servers on 4.2.1, haven't seen any issues like > > > that. You can send along your memory settings... here's what I'm > > > running: > > > > > > JAVA_OPTS="-Djava.awt.headless=true > > > -Dcom.sun.management.jmxremote.port=45219 > > > -Dcom.sun.management.jmxremote.authenticate=false > > > -Dcom.sun.management.jmxremote.ssl=false -Xmx2g > > > -XX:+HeapDumpOnOutOfMemoryError > > > -XX:HeapDumpPath=/var/log/cloudstack/management/ -XX:PermSize=512M > > > -XX:MaxPermSize=800m > > > > > > On Mon, Mar 31, 2014 at 9:33 AM, Michael Phillips > > > <mphilli7...@hotmail.com> wrote: > > > > So I have a redundant pair of management servers running on 4.2.1. At > > > > least once a day one of the management servers crashes and the log gets > > > > filled with the following messages: > > > > java.lang.OutOfMemoryError: Java heap > > > > spac0java.lang.ArrayIndexOutOfBoundsExceptioneCaused by: > > > > java.io.EOFException: SSL peer shut down incorrectlyCaused by: > > > > javax.net.ssl.SSLHandshakeException: Remote host closed > > > > connection during handshake and there are a few others. When the > > > > one management server tanks, although the other management > > > > server is up and active, it won't actually process any UI > > > > commands until the crashed server is restarted. Example of > > > > won't process any UI command is create a new instance, create a > > > > new account, etc. After doing some searching I have found that > > > > others have noticed java heap errors in 4.2.1, and the suggested > > > > fix is to increase the heap size. I am planning on increasing it > > > > from 2g to 4g, however if the problem is something like a memory > > > > leak, then increasing the heap size will just delay the > > > > inevitable. Has anyone else fixed this issue by increasing the > > > > heap size? Or what is the recommended value? **updated...as > > > > expected I increased the heap size from 2g to 4g, and it just > > > > too > > > k longer for the problem to reoccur... > > > > In my honest opinion of bigger concern is the fact that when one > > > > management server crashes the other stops functioning as well. So this > > > > begs the question of why even bother with a redundant pair of > > > > servers..Anybody else experience this issue? I would love to hear any > > > > dev guys opinion on this.... > > > > > > > > > > > >
