So I manually downloaded tomcat 6.0.33 herehttps://cwiki.apache.org/confluence/display/CLOUDSTACK/Setting+up+CloudStack+Development+Environment+on+Linux Then did the following1. extracted 6.0.33 to /usr/share/tomcat6.0.33. 2. Changed symlink of /usr/share/cloudstack-managemet/bin to /usr/share/tomcat6.0.33/bin3. Changed symlink of /usr/share/cloudstack-management/lib to /usr/share/tomcat6.0.33/lib4. Verified cloudstack was running tomcat 6.0.33 by creating a tomcat_version.jsp file in /usr/share/cloudstack-management/webapps/client code for tomcat_version.jsp can be found herehttp://stackoverflow.com/questions/14925073/how-to-find-out-running-tomcat-version I'll definitely let you know how it goes...
> Subject: RE: Interesting 4.2.1. Issue... > From: a...@opencloud.net.au > To: dev@cloudstack.apache.org > Date: Fri, 4 Apr 2014 10:43:35 +0800 > > I tried but I failed to do so, each time cloudstack attempts to install to go > fetches the 6.0.35 from the repo, maybe you have installed it after > installing the cloudstack, if you managed to have a running cloudstack > version above the 6.0.33 feedback with the results. > > Kind Regards > Amin > > -----Original Message----- > From: Michael Phillips [mailto:mphilli7...@hotmail.com] > Sent: Friday, 4 April 2014 10:41 AM > To: dev@cloudstack.apache.org > Subject: RE: Interesting 4.2.1. Issue... > > So did you try changing your version of tomcat? > > > Subject: RE: Interesting 4.2.1. Issue... > > From: a...@opencloud.net.au > > To: dev@cloudstack.apache.org > > Date: Fri, 4 Apr 2014 10:35:42 +0800 > > > > cd /usr/share/tomcat6/bin/ > > ./version.sh > > > > The output should be 6.0.33 instead of 6.0.35 > > > > Using CATALINA_BASE: /usr/share/tomcat6 > > Using CATALINA_HOME: /usr/share/tomcat6 > > Using CATALINA_TMPDIR: /usr/share/tomcat6/temp > > Using JRE_HOME: /usr > > Using CLASSPATH: /usr/share/tomcat6/bin/bootstrap.jar > > Server version: Apache Tomcat/6.0.35 > > Server built: > > Server number: 6.0.35.0 > > OS Name: Linux > > OS Version: 3.11.0-18-generic > > Architecture: amd64 > > JVM Version: 1.6.0_30-b30 > > JVM Vendor: Sun Microsystems Inc. > > > > > > Kind Regards > > Amin > > > > > > > > -----Original Message----- > > From: Michael Phillips [mailto:mphilli7...@hotmail.com] > > Sent: Friday, 4 April 2014 10:31 AM > > To: dev@cloudstack.apache.org > > Subject: RE: Interesting 4.2.1. Issue... > > > > I've put tomcat 6.0.33 on our mgmt servers. I'm going to monitor it for the > > next few days to see if we get the error again. > > Do you know any way way to verify the version of tomcat that's running? > > > > > Subject: RE: Interesting 4.2.1. Issue... > > > From: a...@opencloud.net.au > > > To: dev@cloudstack.apache.org > > > Date: Thu, 3 Apr 2014 10:35:29 +0800 > > > > > > No we didn't, it wouldn't matter because the memory would still fill > > > up, the problem is it opens a thread and it fails to close it so > > > whatever you will increase soon or later the memory will fill up (if > > > I understand right) > > > > > > The error in catalina is as follows: > > > > > > SEVERE: The web application [/client] created a ThreadLocal with key of > > > type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1bd66d2d]) and > > > a value of type [com.cloud.api.SerializationContext] (value > > > [com.cloud.api.SerializationContext@2f6baed9]) but failed to remove it > > > when the web application was stopped. This is very likely to create a > > > memory leak. > > > > > > If someone could help with this error generated in the catalina log, that > > > would be much appreicated. > > > > > > Kind Regards > > > Amin > > > > > > > > > > > > > > > -----Original Message----- > > > From: Michael Phillips [mailto:mphilli7...@hotmail.com] > > > Sent: Thursday, 3 April 2014 9:34 AM > > > To: dev@cloudstack.apache.org > > > Subject: RE: Interesting 4.2.1. Issue... > > > > > > A few other articles also mention setting the initial heap size "-Xms" to > > > the same value as the heap size, to go ahead and fully commit the heap. > > > Have you tried that? > > > One other thing I am curious of is have you fiddled with the Perm Size > > > "XX:Permsize" setting? > > > > > > > Subject: RE: Interesting 4.2.1. Issue... > > > > From: a...@opencloud.net.au > > > > To: dev@cloudstack.apache.org > > > > Date: Thu, 3 Apr 2014 09:26:31 +0800 > > > > > > > > Believe or not!! We have tried setting them in both formats and > > > > still the catalina log produces java heap space error > > > > > > > > Kind Regards > > > > Amin > > > > > > > > > > > > > > > > -----Original Message----- > > > > From: Michael Phillips [mailto:mphilli7...@hotmail.com] > > > > Sent: Thursday, 3 April 2014 9:24 AM > > > > To: dev@cloudstack.apache.org > > > > Subject: RE: Interesting 4.2.1. Issue... > > > > > > > > This may be a silly question, but in all of the docs I am reading > > > > online in regards to increasing the java heap size, they are specifying > > > > it as -Xmx"size""MB" example -Xmx2048m vs -Xmx2g Is it possible that > > > > it's not reading the 2g variable as 2GB? > > > > > > > > > Subject: RE: Interesting 4.2.1. Issue... > > > > > From: a...@opencloud.net.au > > > > > To: dev@cloudstack.apache.org > > > > > Date: Thu, 3 Apr 2014 09:06:17 +0800 > > > > > > > > > > It is 6.0.35 and it still produces this error, even after increasing > > > > > the Xmx to 4g, we have installed tomcat 6.0.33 and each time we > > > > > install the cloudstack it does not sense the installed 6.0.33 and > > > > > attempts to install 6.0.35 as it is dependent on it. Silly solution > > > > > is that we scheduled a daily restart @ 2PM through a cron job!!!! But > > > > > first you have to "killall jvsc" then restart the management server. > > > > > > > > > > What we are considering is to migrate the management server to CentOS > > > > > 6.5 it comes with tomcat 6.0.24 and mysql 5.1, we attempted to > > > > > restore the dump on a pilot environment and it worked. > > > > > > > > > > If someone else has a better solution than this would you please > > > > > share it? > > > > > > > > > > Kind Regards > > > > > Amin > > > > > > > > > > -----Original Message----- > > > > > From: Michael Phillips [mailto:mphilli7...@hotmail.com] > > > > > Sent: Thursday, 3 April 2014 5:31 AM > > > > > To: dev@cloudstack.apache.org > > > > > Subject: RE: Interesting 4.2.1. Issue... > > > > > > > > > > So I just checked my tomcat version and we are running 6.0.35 which > > > > > must be the default that comes with Ubuntu 12.04 out of the box. Our > > > > > memory settings are as follows: > > > > > JAVA_OPTS="-Djava.awt.headless=true > > > > > -Dcom.sun.management.jmxremote.port=45219 > > > > > -Dcom.sun.management.jmxremote.authenticate=false > > > > > -Dcom.sun.management.jmxremote.ssl=false -Xmx4g > > > > > -XX:+HeapDumpOnOutOfMemoryError > > > > > -XX:HeapDumpPath=/var/log/cloudstack/management/ -XX:PermSize=512M > > > > > -XX:MaxPermSize=800m" > > > > > So what version of tomcat are you running 6.0.35 or 6.0.33? > > > > > > > > > > > Subject: RE: Interesting 4.2.1. Issue... > > > > > > From: a...@opencloud.net.au > > > > > > To: dev@cloudstack.apache.org > > > > > > Date: Tue, 1 Apr 2014 11:23:57 +0800 > > > > > > > > > > > > We have the same issue, after an upgrade from 2.2.14 to 4.2.1, > > > > > > and during this upgrade we had upgrade from Ubuntu 10 LTS to > > > > > > Ubuntu 12 LTS, it seems it related to tomcat 6.0.35, because > > > > > > it is recommended to have tomcat 6.0.33 which doesn't come by > > > > > > default with Ubuntu > > > > > > 12.04.4 > > > > > > > > > > > > Kind Regards > > > > > > Amin > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > From: Marcus [mailto:shadow...@gmail.com] > > > > > > Sent: Tuesday, 1 April 2014 6:22 AM > > > > > > To: dev@cloudstack.apache.org > > > > > > Subject: Re: Interesting 4.2.1. Issue... > > > > > > > > > > > > I'm running 3 mgmt servers on 4.2.1, haven't seen any issues > > > > > > like that. You can send along your memory settings... here's > > > > > > what I'm > > > > > > running: > > > > > > > > > > > > JAVA_OPTS="-Djava.awt.headless=true > > > > > > -Dcom.sun.management.jmxremote.port=45219 > > > > > > -Dcom.sun.management.jmxremote.authenticate=false > > > > > > -Dcom.sun.management.jmxremote.ssl=false -Xmx2g > > > > > > -XX:+HeapDumpOnOutOfMemoryError > > > > > > -XX:HeapDumpPath=/var/log/cloudstack/management/ > > > > > > -XX:PermSize=512M -XX:MaxPermSize=800m > > > > > > > > > > > > On Mon, Mar 31, 2014 at 9:33 AM, Michael Phillips > > > > > > <mphilli7...@hotmail.com> wrote: > > > > > > > So I have a redundant pair of management servers running on > > > > > > > 4.2.1. At least once a day one of the management servers crashes > > > > > > > and the log gets filled with the following messages: > > > > > > > java.lang.OutOfMemoryError: Java heap > > > > > > > spac0java.lang.ArrayIndexOutOfBoundsExceptioneCaused by: > > > > > > > java.io.EOFException: SSL peer shut down incorrectlyCaused by: > > > > > > > javax.net.ssl.SSLHandshakeException: Remote host closed > > > > > > > connection during handshake and there are a few others. When > > > > > > > the one management server tanks, although the other > > > > > > > management server is up and active, it won't actually > > > > > > > process any UI commands until the crashed server is > > > > > > > restarted. Example of won't process any UI command is > > > > > > > create a new instance, create a new account, etc. After > > > > > > > doing some searching I have found that others have noticed > > > > > > > java heap errors in 4.2.1, and the suggested fix is to > > > > > > > increase the heap size. I am planning on increasing it from > > > > > > > 2g to 4g, however if the problem is something like a memory > > > > > > > leak, then increasing the heap size will just delay the > > > > > > > inevitable. Has anyone else fixed this issue by increasing > > > > > > > the heap size? Or what is the recommended value? > > > > > > > **updated...as expected I increased the heap size from 2g to > > > > > > > 4g, and it just too > > > > > > k longer for the problem to reoccur... > > > > > > > In my honest opinion of bigger concern is the fact that when one > > > > > > > management server crashes the other stops functioning as well. So > > > > > > > this begs the question of why even bother with a redundant pair > > > > > > > of servers..Anybody else experience this issue? I would love to > > > > > > > hear any dev guys opinion on this.... > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
