Hi, I increased the Heap & PermGen memory as per trail mail but the performance seemed to deteriorate. On the day I increased the memory in the morning, the system hung 3 times in the day whereas the maximum times it has got hang before is 2 times in a day. I changed the memory settings for Heap & PermGen back to before when it hung again the next day, I reverted to the old memory values.
The catalina.out showed the entries like below when I monitored it twice after the system hung with the new memory values: -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [ 22-May-2014 03:15:40 ] [ TRACE ] [http-30080-56] [ CHECK SESSION ] : Inside doInit Method May 22, 2014 3:16:14 PM org.zkoss.zk.ui.impl.UiEngineImpl handleError:1351 SEVERE: >>org.zkoss.zk.ui.UiException: org.jboss.remoting.InvocationFailureException: Socket timed out. Waited 300000 milliseconds for response while calling on InvokerLocator [socket:// bssossapp01.snl.com:4173/?invokerDestructionDelay=5000&timeout=300000]; nested exception is: java.net.SocketTimeoutException: Read timed out >>org.jboss.remoting.InvocationFailureException: Socket timed out. Waited 300000 milliseconds for response while calling on InvokerLocator [socket:// bssossapp01.snl.com:4173/?invokerDestructionDelay=5000&timeout=300000]; nested exception is: java.net.SocketTimeoutException: Read timed out >>java.net.SocketTimeoutException: Read timed out >> at java.net.SocketInputStream.socketRead0(Native Method) -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- I would be eager to respond in case of any clarification. Requesting a reply to resolve this as the attempt to resolve this by increasing Heap & PermGen memory has failed. I have worked on this issue a lot and have been trying to interact with the experts on the forum. Regards On Wed, May 21, 2014 at 6:01 PM, Randhir Singh <randhir.si...@sterlite.com>wrote: > Hi, > > There is a correction as JAVA_OPTS variable is defined in catalina.sh > under $CATALINA_HOME/bin and we use $CATALINA_HOME/bin/startup.sh to start > tomcat. $CATALINA_HOME/bin/startup.sh has the code like below , > > > ---------------------------------------------------------------------------------------------------------------------------------------------- > PRGDIR=`dirname "$PRG"` > EXECUTABLE=catalina.sh > > # Check that target executable exists > if $os400; then > # -x will Only work on the os400 if the files are: > # 1. owned by the user > # 2. owned by the PRIMARY group of the user > # this will not work if the user belongs in secondary groups > eval > else > if [ ! -x "$PRGDIR"/"$EXECUTABLE" ]; then > echo "Cannot find $PRGDIR/$EXECUTABLE" > echo "The file is absent or does not have execute permission" > echo "This file is needed to run this program" > exit 1 > fi > fi > > ---------------------------------------------------------------------------------------------------------------------------------------------- > > I had changed catalina.sh in our development environment like a week back > and want to implement it in the production environment but I got this > doubt. I feel catalina.sh is invoked by startup.sh but am not sure. I have > already taken downtime for production tomorrow but I got this doubt whether > changing the > > JAVA_OPTS="$JAVA_OPTS -Xms1303m -Xmx1303m -XX:MaxPermSize=256m > -Dcwom.bl.ip=127.0.0.1 -Dcwom.bl.port=1399" to > > JAVA_OPTS="$JAVA_OPTS -Xms2048m -Xmx2048m > -XX:MaxPermSize=512m -Dcwom.bl.ip=127.0.0.1 -Dcwom.bl.port=1399" > > in catalina.sh would have the affect of increasing the heap & Permgen > memory in Tomcat. > > I hope my query is clear as am quite anxious as I have a downtime > scheduled tomorrow. > > Requesting a reply on priority. > > > <randhir.si...@sterlite.com> wrote: > >> Thanks a lot for your detailed reply. >> >> The JAVA_OPTS variable in $CATALINA_HOME/bin/startup.sh has the value as: >> >> JAVA_OPTS="$JAVA_OPTS -Xms1303m -Xmx1303m -XX:MaxPermSize=256m >> -Dcwom.bl.ip=127.0.0.1 >> -Dcwom.bl.port=1399" >> >> I checked for free space on the VM instance with Red Hat Linux as the OS >> on >> which the Tomcat is hosted with a total memory of 19 GB, I found that >> there >> is around 8 GB of free space in the server. >> >> As per your inputs, I am planning to increase the memory allocation for >> Heap >> size and PermGen size to almost double as below: >> >> JAVA_OPTS="$JAVA_OPTS -Xms2048m -Xmx2048m -XX:MaxPermSize=512m >> -Dcwom.bl.ip=127.0.0.1 >> -Dcwom.bl.port=1399" >> >> I hope, this should avoid the hang-up issues where Tomcat does not seem to >> respond. >> >> I plan to implement this in the UAT environment at first and see how it >> goes. >> >> Regards >> >> -----Original Message----- >> From: Jeffrey Janner [mailto:jeffrey.jan...@polydyne.com] >> Sent: Monday, April 28, 2014 7:39 PM >> To: 'Randhir Singh' >> Subject: RE: How to monitor performance of tomcat >> >> > -----Original Message----- >> > From: Randhir Singh [mailto:randhir.si...@sterlite.com] >> > Sent: Monday, April 28, 2014 5:47 AM >> > To: Jeffrey Janner; Tomcat Users List >> > Subject: RE: How to monitor performance of tomcat >> > >> > Thanks for your valuable inputs. I am a bit of a novice to this. When >> > this problem happens, it is quite a loss of face in front of the users >> > who are widely spread geographically. I checked the catalina.out logs >> > on the 2 days that the problem happened on 23rd & 25th April, the >> > findings are as below: >> > >> > 23rd April '14: >> > >> > The catalina.out log showed the message like, >> > ---------------------------------------------------------------------- >> > - >> > ---------------------------------------------------------------------- >> > - >> > --------- >> > Caused by: org.jboss.remoting.InvocationFailureException: Socket timed >> > out. >> > Waited 300000 milliseconds for response while calling on >> > InvokerLocator [socket://<hostname of our >> > machine>:4173/?invokerDestructionDelay=5000&timeout=300000]; nested >> > exception is: >> > java.net.SocketTimeoutException: Read timed out >> > ---------------------------------------------------------------------- >> > - >> > ---------------------------------------------------------------------- >> > - >> > --------- >> > >> > 3 times among other exceptions in a span of around 2 minutes. >> > >> > 25th April '14: >> > >> > The errors captured in the logs were like, >> > ---------------------------------------------------------------------- >> > - >> > ---------------------------------------------------------------------- >> > - >> > --------- >> > java.lang.OutOfMemoryError: GC overhead limit exceeded >> >> This error specifically means that the JVM is basically spending all of >> its >> time doing garbage collection and not really getting any significant >> amount >> of memory back for its efforts. The end user experiences this usually as >> a >> hung system. It might eventually come back or it might end up with a >> different, more severe, OOM error. >> There are lots of various causes for this: too small memory allocation; >> too >> much load; an activity generating a lot of objects, most of which persist >> for a long time; memory leaks; or a combination of the above. >> How to address it? >> a) Profile what parts of the app are being used with this occurs and have >> the dev team look at what could be causing the problem. There might be a >> better way to perform the action requested. >> b) Increase the available memory (double it or more). This is only a >> band-aid and you may still have the problem, but less often. Have the dev >> team look for memory leaks. >> c) Anyone else with ideas? >> Jeff >> >> > ---------------------------------------------------------------------- >> > - >> > ---------------------------------------------------------------------- >> > - >> > --------- >> > >> > 4 times among other exceptions in a span of around 2 minutes. >> > >> > Requesting a reply on this. >> > >> > Regards >> >> __________________________________________________________________________ >> Confidentiality Notice: This Transmission (including any attachments) may >> contain information that is privileged, confidential, and exempt from >> disclosure under applicable law. If the reader of this message is not the >> intended recipient you are hereby notified that any dissemination, >> distribution, or copying of this communication is strictly prohibited. >> >> If you have received this transmission in error, please immediately reply >> to >> the sender or telephone (512) 343-9100 and delete this transmission from >> your system. >> > > -- *STL Disclaimer:* The content of this message may be legally privileged and confidential and are for the use of the intended recipient(s) only. It should not be read, copied and used by anyone other than the intended recipient(s). If you have received this message in error, please immediately notify the sender, preserve its confidentiality and delete it. Before opening any attachments please check them for viruses and defects. No employee or agent is authorised to conclude any binding agreement on behalf of Sterlite Technologies Limited with another party by email without express written confirmation by authorised person. Visit us at www.sterlitetechnologies.com Please consider environment before printing this email !
