Re: java.lang.OutOfMemoryError: Java heap space

[Joe Hansen]

 Yup, it looks like you have more of your application objects taking up
 memory than the tickets you were worried about.

So True! We are right now designing and developing our next generation
of websites and this experience will surely come handy. I will now
think twice before I store anything in the session and will make sure
I remove no longer used objects from the HTTP Session.

 You can couple session-object expiration with lazy instantiation/fetch for a 
 solution
 that can keep your sessions lightweight yet allow insanely long session
 timeouts.

Chris, can you please elaborate what you mean by coupling
session-object expiration with lazy fetch?

 Is it an absolute requirement that these objects be stored in the
 session at all? If so, then maybe the caching strategy can be tweaked a
 bit to allow both requirements to peacefully coexist.

The code is 3 years old and it does not use a caching strategy at all.
However, I am planning to use ehcache when we developer our future
websites.

 There are even techniques that will allow your session objects to expire
 when memory gets tight (which is super cool IMO).

I did not know that something like that existed. Thanks for letting me
know, Chris!

Thanks for telling me about SoftReference, Charles. Looks like
SoftReference existed since JDK 1.2 and I never knew about it! Is the
use of SoftReference popular?

Thanks guys!
Joe

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: java.lang.OutOfMemoryError: Java heap space

[Caldarale, Charles R]

 From: Joe Hansen [mailto:joe.hansen...@gmail.com]
 Subject: Re: java.lang.OutOfMemoryError: Java heap space
 
 Is the use of SoftReference popular?

You'll find it as the underlying mechanism of many cache managers, since it 
automatically adapts reasonably well to whatever the execution environment 
happens to be.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: java.lang.OutOfMemoryError: Java heap space

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Joe,

On 10/8/2009 12:12 PM, Joe Hansen wrote:
 I will now think twice before I store anything in the session and
 will make sure I remove no longer used objects from the HTTP
 Session.

It's always a good goal to limit your session size, but it's not always
foolproof because users don't always do the things you expect. For
instance, if you have a process your user goes through, and you store a
big object (or set of objects) in the session during that process, and
the user does not finish the process, you still need to clean-up after that.

 You can couple session-object expiration with lazy 
 instantiation/fetch for a solution that can keep your sessions
 lightweight yet allow insanely long session timeouts.
 
 Chris, can you please elaborate what you mean by coupling
 session-object expiration with lazy fetch?

So, lazy instantiation (or initialization) is a design pattern where you
only create objects when you are actually going to use them, as opposed
to creating everything you /might/ need just in case it's necessary.
Lazy fetch is just a term I used to describe grabbing your Foto
objects only when actually needed (rather than speculatively loading them).

This concept is illustrated easily in a few lines of code:

// This would be speculative instantiation
Vector v = new Vector(1000); // Just in case

...

if(something) {
   // use the Vector object
}

// This would be lazy instantiation
Vector v = null;

if(something) {
  if(null == v)
v = new Vector();

  // use the Vector object
}

It sounds simple, and it is. The same concept can be applied when
fetching large objects from the database and possibly storing them in
the session. You can use the session as a poor-mans cache something like
this:

// You probably already do this kind of thing in your code:
Foto foto = (Foto)session.getAttribute(myFotoObject);

// But now, you can plan for the case where the object either
// was never there, or has disappeared because it has been
// removed to save memory:

if(null == foto) {
  foto = fotoManager.fetch(whatever);

  session.setAttribute(myFotoObject, foto);
}

Using this technique, you can get the benefit of session caching that
is tolerant of disappearing objects due to the lazy-fetching technique
I describe. See below for how to hook it up to cache-flushing schemes.

 Is it an absolute requirement that these objects be stored in the
 session at all? If so, then maybe the caching strategy can be tweaked a
 bit to allow both requirements to peacefully coexist.
 
 The code is 3 years old and it does not use a caching strategy at all.
 However, I am planning to use ehcache when we developer our future
 websites.

3 years is not a ripe old age for code. Our best code is the stuff that
has lasted for 3 or more years without having to be rewritten. ;)

If ehcache is your thing, go for it. There are other possibilities, too.

 There are even techniques that will allow your session objects to expire
 when memory gets tight (which is super cool IMO).

One way to retrofit your web application is to use SoftReferences to
store objects within your session attributes. You can either re-write
all your code to deal with SoftReferences (see examples below), or you
can get tricky and write a simple wrapper around your request/session
objects to do the magic for you.

I'll show you how such tricks could be implemented.

Let's say that you always use the prefix foto for all Foto objects in
the session. You could use this technique with all session objects, but
it really only makes sense with the big ones. Here's the plan:

1. Alter your code that deals with Foto objects stored in the session to
tolerate the objects apparently disappearing from the session without
notice. That is, always check for null and re-fetch from the database or
wherever you get your Foto data.

2. SoftReference objects will be used to store your actual Foto objects
in the session.

3. Write a Filter which wraps the HttpServletRequest object to return a
wrapped HttpSession object which will handle the SoftReferences
mentioned in #2 (got all that?)

It's easier than it sounds. Let's assume #1 is already done. Here's how
to write the filter (which implements both #2 and #2 above):

public class SoftReferenceFilter
  implements Filter
{
  public void doFilter(ServletRequest request, ServletResponse response,
   FilterChain chain)
throws ServletException, IOException
  {
if(request instanceof HttpServletRequest)
  request
= new SoftReferenceRequestWrapper((HttpServletRequest)request);

chain.doFilter(request, response);
  }

  public static class SoftReferenceRequestWrapper
extends HttpServletRequestWrapper
  {
public SoftReferenceRequestWrapper(HttpServletRequest request)
{
  super(request);
}

// Override
public HttpSession getSession()
{
  HttpSession session = super.getSession();

  if(null != session)

Re: java.lang.OutOfMemoryError: Java heap space

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

All,

A correction to my code:

On 10/8/2009 4:22 PM, Christopher Schultz wrote:
 public class SoftReferenceFilter
   implements Filter
 {

This class needs two additional methods:

  public void init(FilterChain chain) { }
  public void destroy() { }

 // Here's where the magic happens
 public Object getAttribute(String key)
 {
   Object o = super.getAttribute(key); // Get the stored object

This should be:

Object o = _base.getAttribute(key);

 public Object setAttribute(String key, Object value)
 {
   // Wrap the value in a SoftReference if appropriate
   if(key.startsWith(foto)  null != value)
 value = new SoftReference(value);
 
   super.setAttribute(key, value);

This should be:

_base.setAttribute(key, value);

Not too bad for code typed directly into an email message.

- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkrOTIwACgkQ9CaO5/Lv0PAgoACgoAZOVSzVhSOwD893/qC/YtZ/
57MAnApZ064qpwt15QzMuEswaU3KFxMY
=nno+
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: java.lang.OutOfMemoryError: Java heap space

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

All,

Another correction to the code:

On 10/8/2009 4:22 PM, Christopher Schultz wrote:
   public static class SoftReferenceSessionWrapper
 implements HttpSession
   {

Remember to implement getValue and putValue to call getAttribute and
setAttribute. ;)

- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkrOTdUACgkQ9CaO5/Lv0PCIDwCgt+kyda0PTi+LoqQvQhGb2yHC
HKMAoL/uMesdmV1Ubrco5Hyx8F9x0IKF
=53/Y
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: java.lang.OutOfMemoryError: Java heap space

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Joe,

On 10/6/2009 3:44 PM, Joe Hansen wrote:
 I did not check the CPU usage when the issue happened.  I just checked
 the web application logs and found that first there was an
 OutOfMemoryError, followed by MySQL errors when
 TWFotoSetListDAO.getAllFotosets tried to get a database connection.

It's possible that, if you really are grabbing photos (like, actual
images) from a database and fitting them into memory, you might just be
busting your heap yourself (rather than CAS, etc.). Are there large
objects that you store in memory and/or sessions?

 I am *guessing* :
 I changed the session timeout to 4 hours for the Jasig CAS
 Single-Signon web application. This stores a ton of tickets in memory.
 During a 4 hour period when there are more http requests, the 512MB of
 Java Heap space is not sufficient to store all these tickets. During
 that time OutOfMemoryError happens and all hell breaks loose
 subsequently.

Wow, those tickets seem very heavyweight. Try this when the server is
quiet and then, later, when it's getting stormed:

$ jmap -histo [jvm pid]  heap.log

Then, go into the heap log and see what's taking up so much of your
memory. For instance, if I run this against my own JVM running in
development, here are the top 10 objects in memory:

SizeCount   Class description
- ---
1078520098394   char[]
5548488 40283   * ConstMethodKlass
2911352 7238byte[]
2465208 102717  java.lang.String
2259496 40283   * MethodKlass
2212736 54518   * SymbolKlass
2171032 3767* ConstantPoolKlass
1912032 7936int[]
1761184 38585   java.lang.Object[]
1500112 3767* InstanceKlassKlass

These are the objects taking up most memory. If you look at the count
column, you can see how many of the objects are in memory. I suggest
looking at both the size and the count columns, because your
understanding of the size of an object and the JVM's notion may
differ: most heap space is taken up by byte and char arrays and things
like that, but then encapsulated into larger objects that, themselves,
take up very little space.

You may find that you have thousands of CAS tickets in memory and that
they are very heavy. You may also find that the tickets themselves
aren't the problem, but that a 4-hour session timeout leads to sessions
with huge amounts of cached information in them. It's possible that your
webapp is simply being careless with memory if you want a 4-hour session
timeout.

You might want to invest in a real memory profiler to see what is
happening. These products can give you a better picture of large
objects that you can look at to make decisions as to where you will
apply your efforts.

- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkrM2wYACgkQ9CaO5/Lv0PCjTwCfS/bwZmXLT+BUgYxJVyqIv6tT
AuYAn1Jupcl6eB240BHH3pvtKJNGTglq
=XDLj
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: java.lang.OutOfMemoryError: Java heap space

[Joe Hansen]

WoW! jmap was a great command to learn Chris! Thank you so much!

I generated a histogram of the Objects in the java heap (while things
were stable) and here's what I got:
Object Histogram:

SizeCount   Class description
---
149989048   3496198 char[]
847084323529518 java.lang.String
58308016560654  tw.beans.TWFotoBean
35620304179282  byte[]
2328687230767   int[]
18220184142094  * ConstMethodKlass
14932056622169  java.sql.Date
10235120142094  * MethodKlass
1016478490757   tw.beans.TWFotoSetBean
8878248 134561  java.lang.Object[]
7949144 16190   * ConstantPoolKlass
6147768 16190   * InstanceKlassKlass
4590176 13997   * ConstantPoolCacheKlass
...
...

As you've pointed out, most of the data is taken by character arrays,
Strings, integer arrays, byte arrays, Foto and FotoSet beans! There
were just 67 Ticket entries (related to CAS) and the size of the
Ticket objects wasn't huge either.

Seeing the histogram results was an eye opener. I am wondering if
Rainer was right all along. Is it just the application data (FotoBean
and FotoSetBean) that's inundating the java heap?

I had changed the session-timeout in server.xml from 30 minutes to 240
minutes. That means an HTTP Session would now last 8 times longer.
Since the Garbage Collector runs only when the session expires, the
application's memory needs are probably increased 8 fold (Am I
right?). And I am storing the FotoBeans and FotoSetBeans in each
user's HTTP Session. As I mentioned before, previously the java heap
was set to the default, 64MB, and it used to be sufficient. From the
time I changed the session-timeout, even 512MB (= 64 X 8) is
insufficient.

Should I just decrease the session-timeout to 2 hours and see if 512MB
is sufficient? Any other thoughts/ideas guys?

Thanks again Chris,
Joe

Here are all the Ticket entries in the heap:
96  4   org.jasig.cas.util.DefaultUniqueTicketIdGenerator
72  3   org.jasig.cas.util.DefaultUniqueTicketIdGenerator
72  3   org.jasig.cas.util.DefaultUniqueTicketIdGenerator
72  3   org.jasig.cas.util.DefaultUniqueTicketIdGenerator
72  3   org.jasig.cas.util.DefaultUniqueTicketIdGenerator
72  3   org.jasig.cas.util.DefaultUniqueTicketIdGenerator
24  1   org.jasig.cas.web.flow.SendTicketGrantingTicketAction
24  1   org.jasig.cas.web.flow.SendTicketGrantingTicketAction
24  1   org.jasig.cas.web.flow.SendTicketGrantingTicketAction
24  1   org.jasig.cas.web.flow.SendTicketGrantingTicketAction
24  1   
org.acegisecurity.providers.cas.ticketvalidator.CasProxyTicketValidator
24  1   
org.acegisecurity.providers.cas.ticketvalidator.CasProxyTicketValidator
24  1   
org.acegisecurity.providers.cas.ticketvalidator.CasProxyTicketValidator
24  1   
org.acegisecurity.providers.cas.ticketvalidator.CasProxyTicketValidator
24  1   org.jasig.cas.web.flow.SendTicketGrantingTicketAction
24  1   
org.acegisecurity.providers.cas.ticketvalidator.CasProxyTicketValidator
24  1   org.jasig.cas.web.flow.SendTicketGrantingTicketAction
24  1   
org.acegisecurity.providers.cas.ticketvalidator.CasProxyTicketValidator
16  1   org.jasig.cas.util.SamlCompliantUniqueTicketIdGenerator
16  1   org.jasig.cas.ticket.registry.DefaultTicketRegistry
16  1   
org.jasig.cas.ticket.registry.support.DefaultTicketRegistryCleaner
16  1   org.jasig.cas.util.SamlCompliantUniqueTicketIdGenerator
16  1   org.jasig.cas.ticket.registry.DefaultTicketRegistry
16  1   org.acegisecurity.providers.cas.proxy.RejectProxyTickets
16  1   org.jasig.cas.web.flow.GenerateServiceTicketAction
16  1   org.jasig.cas.ticket.registry.DefaultTicketRegistry
16  1   
org.jasig.cas.ticket.registry.support.DefaultTicketRegistryCleaner
16  1   org.acegisecurity.providers.cas.proxy.RejectProxyTickets
16  1   
org.jasig.cas.ticket.registry.support.DefaultTicketRegistryCleaner
16  1   org.jasig.cas.web.flow.GenerateServiceTicketAction
16  1   org.jasig.cas.util.SamlCompliantUniqueTicketIdGenerator
16  1   org.acegisecurity.providers.cas.cache.EhCacheBasedTicketCache
16  1   org.acegisecurity.providers.cas.cache.EhCacheBasedTicketCache
16  1   
org.jasig.cas.ticket.registry.support.DefaultTicketRegistryCleaner
16  1   org.jasig.cas.util.SamlCompliantUniqueTicketIdGenerator
16  1   org.jasig.cas.web.flow.GenerateServiceTicketAction
16  1   org.acegisecurity.providers.cas.proxy.RejectProxyTickets
16  1   
org.jasig.cas.ticket.registry.support.DefaultTicketRegistryCleaner
16  1   org.jasig.cas.web.flow.GenerateServiceTicketAction
16  1   org.acegisecurity.providers.cas.proxy.RejectProxyTickets
16  1   org.jasig.cas.ticket.registry.DefaultTicketRegistry
16  

Re: java.lang.OutOfMemoryError: Java heap space

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Joe,

On 10/7/2009 8:40 PM, Joe Hansen wrote:
 WoW! jmap was a great command to learn Chris! Thank you so much!

Yeah, it's great as a poor-man's memory profiler. On the other hand, I
haven't seen a big-time memory profiler offer such a simple view of
everything since Borland's OptimizeIt.

 As you've pointed out, most of the data is taken by character arrays,
 Strings, integer arrays, byte arrays, Foto and FotoSet beans! There
 were just 67 Ticket entries (related to CAS) and the size of the
 Ticket objects wasn't huge either.

Yup, it looks like you have more of your application objects taking up
memory than the tickets you were worried about.

 I had changed the session-timeout in server.xml from 30 minutes to 240
 minutes. That means an HTTP Session would now last 8 times longer.
 Since the Garbage Collector runs only when the session expires, the
 application's memory needs are probably increased 8 fold (Am I
 right?).

Er, the GC runs all the time, but the objects in the session are not
cleaned-out until the session expires. You might want to consider
cleaning your sessions a bit more regularly. You can couple
session-object expiration with lazy instantiation/fetch for a solution
that can keep your sessions lightweight yet allow insanely long session
timeouts. It will require you to do some additional work, but improved
stability is probably worth it.

 Should I just decrease the session-timeout to 2 hours and see if 512MB
 is sufficient? Any other thoughts/ideas guys?

I think your session timeout should be dictated by your business
requirements, not technical limitations (since this is a relatively low
technical hurdle).

Is it an absolute requirement that these objects be stored in the
session at all? If so, then maybe the caching strategy can be tweaked a
bit to allow both requirements to peacefully coexist.

There are even techniques that will allow your session objects to expire
when memory gets tight (which is super cool IMO).

- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkrNQGUACgkQ9CaO5/Lv0PD1XwCePdhJbbdSakmn2x9uopHrN7Na
6R8AoMDfMpMioh/J2D+UfpJTUxFiHJ3G
=avdi
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: java.lang.OutOfMemoryError: Java heap space

[Caldarale, Charles R]

 From: Christopher Schultz [mailto:ch...@christopherschultz.net]
 Subject: Re: java.lang.OutOfMemoryError: Java heap space
 
 There are even techniques that will allow your session objects to
 expire when memory gets tight (which is super cool IMO).

To put a name on the above technique, read up on using SoftReference objects to 
track entries in your cache(s).  These will allow GC to discard objects not in 
immediate use when the heap is full.
http://www.ibm.com/developerworks/java/library/j-jtp01246.html
http://java.sun.com/javase/6/docs/api/java/lang/ref/SoftReference.html

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.

Re: java.lang.OutOfMemoryError: Java heap space

[Rainer Jung]

On 05.10.2009 22:04, André Warnier wrote:
 André Warnier wrote:
 ...
 and still wants to add something :
 
 - a new KeepAlive connection is made from the browser to Apache (httpd).
 - then a request comes in on that connection, and it happens to be one
 that gets forwarded to Tomcat.  So a mod_jk connection is made to
 Tomcat, Tomcat allocates a thread for it.
 - I would imagine that mod_jk must pass on to Tomcat the fact that this
 is a KeepAlive connection, so that Tomcat would know that its thread for
 that connection, should also wait for subsequent requests.
 - so now the webapp/thread generates the response to the first request,
 and waits on the connection for more requests.
 - however, the browser does send more requests to Apache, but these are
 not ones that get forwarded to Tomcat (for example, they are for items
 that Apache serves locally) ...
 
 So now I wonder about how Apache + mod_jk + Tomcat react in such a
 situation.  Do the mod_jk connection +Tomcat thread keep waiting anyway,
 and how long ?

No in general. mod_jk only kicks in after a new request arrived over
such a connection. mod_jk is agnotic about whether the rquests came over
the same connection or not, even whether they belong to the same user or
not.

But: in case you are using a prefork MPM, each connection gets assigned
to one process having one thread. This process has a connection pool of
size 1 to each backend and the backend ties a dedicated thread to that
connection (unless using tcnative). So in this case, it is a yes.
Between two Keep Alive requests, there's no way of letting the Apache
process handle other requests and thus the backend connection and
backend thread are blocked waiting for the next request. That's why e.g.
NTLM has a chance to work via mod_jk in combination with prefork.

So here's a use case for the worker MPM, because then one client
connection still blocks one Apache thread. But multiple connections are
concurrently handled by the same Apache process, so mod_jk can
efficiently reuse its backend connections for requests from other
connections handled by the same process. The total used connection pool
size might then be much smaller than with the prefork MPM.

Regards,

Rainer

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: java.lang.OutOfMemoryError: Java heap space

[Rainer Jung]

On 05.10.2009 18:58, Joe Hansen wrote:
 Thank you so much for your tips, Rainer!
 
 The websites went down yet again. Increasing the java heap size took
 care of the OutOfMemoryError, but the number of httpd processes keep
 increasing until the websites crash. I haven't added any new code in
 the past few months, hence I am surprised why the requests are getting
 stuck. Here's a link to the tomcat thread dumps:
 http://pastebin.com/m17eea139

Just tried to look at it, but pastebin replies with:


Down for maintenance - 6th Oct 2009

Pastebin.com is getting an unprecedented amount of traffic due to a news
story in which some leaked Hotmail passwords have been pasted on this site

...


Let's see, when they will be up again. They're running Apache 1.3.33 ...

 Please let me know if you cannot view it and I will email the relevant
 portion of the catalina.out file to you. Is there an easy way to find
 out what code is causing the requests to get stuck?
 
 Thank you!
 
 Joe
 
 
 On Sun, Oct 4, 2009 at 2:36 PM, Rainer Jung rainer.j...@kippdata.de wrote:
 Hi Joe,

 On 04.10.2009 21:45, Joe Hansen wrote:
 Rainer, Thank you so much for your kind reply!

 I have increased the java heap size to 512MB (-Xms512m -Xmx512m). I am
 hoping that would fix the issue. I had configured our webserver to use
 Jasig's Central Authentication System (CAS). Recently I increased the
 session timeout from 30 minutes to 4 hours. I am guessing that must
 have had an impact on the number of tickets that the CAS could store
 in the Java's memory space.

 I did run the kill -QUIT command against the tomcat process. It did
 generate a huge output in the catalina.out file. I am unable to
 decipher it. I do not want to post it to the mailing list because its
 very long. Would you be able to please tell me what should I be
 looking for within this long thread dump?

 Can you put it somewhere on the web, so we can look at it, or are you
 afraid there is something private in there? You could use pastebin or
 something similar in case you do not have a public web server yourself.

 If you don't want to post in public, you can also mail it to me, I will
 post the result, in case I find something relevant.

 Regards,

 Rainer

 On Sat, Oct 3, 2009 at 12:24 PM, Rainer Jung rainer.j...@kippdata.de 
 wrote:
 On 03.10.2009 20:07, Joe Hansen wrote:
 Hey All,

 I get this error (java.lang.OutOfMemoryError: Java heap space) after
 my Apache 2.0/Tomcat 5.5/mod_jk installation has been up and running
 for a few hours. This problem started just since two days. Never had
 this issue before!

 I have also noticed that as soon as I startup the server, 9 httpd
 processes start. Number of httpd processes keep on increasing until I
 get the OutOfMemoryError.
 $ps -aef | grep httpd
 root 31984 1  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31987 31984  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31988 31984  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31989 31984  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31990 31984  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31991 31984  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31992 31984  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31993 31984  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31994 31984  0 11:23 ?00:00:00 /usr/sbin/httpd

 Sounds like requests get stuck or responses are only returned very slowly.

 I would take thread dumps during the time requests pile up (e.g. httpd
 process count increases). Thread dumps are generated by kil -QUIT
 against the Tomcat process. Result is written to catalina.out. Always
 take afew thread dumps shortly after each other, e.g. 3 dumps each 3
 seconds apart from the previous one, so that you can find out, if a
 status in a dump is pure coincidence or lasts for somewhat longer.

 $ps -aef | grep tomcat
 root 31949 1 43 11:23 pts/000:00:58 /usr/java/jdk/bin/java 
 -Djava.u
 il.logging.manager=org.apache.juli.ClassLoaderLogManager 
 -Djava.util.logging.co
 fig.file=/usr/lib/apache-tomcat/conf/logging.properties 
 -Djava.endorsed.dirs=/u
 r/lib/apache-tomcat/common/endorsed -classpath 
 :/usr/lib/apache-tomcat/bin/boot
 trap.jar:/usr/lib/apache-tomcat/bin/commons-logging-api.jar 
 -Dcatalina.base=/us
 /lib/apache-tomcat -Dcatalina.home=/usr/lib/apache-tomcat 
 -Djava.io.tmpdir=/usr
 lib/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start

 There is no Java memory configuration included above (i.e. al defaults).
 It might well be, that you have to explicitely set heap size, perm size
 and if you like also eden and semi spaces.

 Can someone on this list please help me resolve this issue.

 Thanks you,
 Joe

 Regards,

 Rainer

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: java.lang.OutOfMemoryError: Java heap space

[Rainer Jung]

On 05.10.2009 18:58, Joe Hansen wrote:
 Thank you so much for your tips, Rainer!
 
 The websites went down yet again. Increasing the java heap size took
 care of the OutOfMemoryError, but the number of httpd processes keep
 increasing until the websites crash. I haven't added any new code in
 the past few months, hence I am surprised why the requests are getting
 stuck. Here's a link to the tomcat thread dumps:
 http://pastebin.com/m17eea139
 
 Please let me know if you cannot view it and I will email the relevant
 portion of the catalina.out file to you. Is there an easy way to find
 out what code is causing the requests to get stuck?

The dump file contains three thread dumps.

The things all dumps have in common:

- 60 threads for the quartz scheduler, all idle
- 13 threads in the AJP connection pool, connected to Apache, but idle
waiting for the next request to be send (the same threads in all three
dumps)
- 6 store plus 6 expiry threads of the EHCache, seems idle
- 1 AJP + 1 HTTP(S) thread (port 8443) waiting to accept the next new
connection to come in
- 2 AJP + 3 HTTP(S) threads (port 8443) sitting idle the pool, waiting
for work
- a couple of other normal threads not directly related to request handling

So the time you took the three dumps, this Tomcat was completely idle
and did not have a single request to handle.

If you are completely sure, you took the dumps while there was a storm
of requests and your system couldn't cope the load, something has
prevented the requests to ever reach Tomcat.

I don't have your Tomcat version at hand at the moment, but for some
time very special OutOfMemory errors (could not create native thread)
lead to a situation, where Tomcat simply wouldn't accept any new
connections. Although you report OutOfMemory errors, I'm not directly
suggesting that that is your problem here. There might still be a
relation though.

Are you sure, that you took the dumps for the right Tomcat at the right
time?

Regards,

Rainer

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: java.lang.OutOfMemoryError: Java heap space

[Joe Hansen]

Rainer,

Thanks for looking at those long thread dumps for me!!

I am sorry. I did NOT take these dumps at the right time (i.e. when
Tomcat was inundated with requests and couldn't cope with the load).
After I increased the heap size to 512MB (from 64MB default), I am not
getting the OutOfMemoryError(s) anymore. After I set KeepAlive On
(Thanks Andre!), the number httpd processes isn't increasing either.
The number of httpd processes increased from 8 to 21 and stayed there
for more than 16 hours now. If the number of httpd processes gets out
of control again, I will definitely take thread dumps once again.

However, I doubt the issue is fixed because I should have seen this
issue long time ago (since I haven't changed any code nor the traffic
to our websites increased in a long while).

Should I just wait and see or are there any tests that I can do?

Your contribution to this forum is amazing, Rainer. I am grateful to
you and Andre for your efforts. Thank you!

Regards,
Joe



On Tue, Oct 6, 2009 at 7:25 AM, Rainer Jung rainer.j...@kippdata.de wrote:
 On 05.10.2009 18:58, Joe Hansen wrote:
 Thank you so much for your tips, Rainer!

 The websites went down yet again. Increasing the java heap size took
 care of the OutOfMemoryError, but the number of httpd processes keep
 increasing until the websites crash. I haven't added any new code in
 the past few months, hence I am surprised why the requests are getting
 stuck. Here's a link to the tomcat thread dumps:
 http://pastebin.com/m17eea139

 Please let me know if you cannot view it and I will email the relevant
 portion of the catalina.out file to you. Is there an easy way to find
 out what code is causing the requests to get stuck?

 The dump file contains three thread dumps.

 The things all dumps have in common:

 - 60 threads for the quartz scheduler, all idle
 - 13 threads in the AJP connection pool, connected to Apache, but idle
 waiting for the next request to be send (the same threads in all three
 dumps)
 - 6 store plus 6 expiry threads of the EHCache, seems idle
 - 1 AJP + 1 HTTP(S) thread (port 8443) waiting to accept the next new
 connection to come in
 - 2 AJP + 3 HTTP(S) threads (port 8443) sitting idle the pool, waiting
 for work
 - a couple of other normal threads not directly related to request handling

 So the time you took the three dumps, this Tomcat was completely idle
 and did not have a single request to handle.

 If you are completely sure, you took the dumps while there was a storm
 of requests and your system couldn't cope the load, something has
 prevented the requests to ever reach Tomcat.

 I don't have your Tomcat version at hand at the moment, but for some
 time very special OutOfMemory errors (could not create native thread)
 lead to a situation, where Tomcat simply wouldn't accept any new
 connections. Although you report OutOfMemory errors, I'm not directly
 suggesting that that is your problem here. There might still be a
 relation though.

 Are you sure, that you took the dumps for the right Tomcat at the right
 time?

 Regards,

 Rainer

 -
 To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: java.lang.OutOfMemoryError: Java heap space

[Joe Hansen]

Rainer,

I spoke to soon! As I suspected, the problem isn't fixed yet and the
websites crashed again. This time I took three thread dumps at the
time tomcat was down. Here they are: http://pastebin.com/m2a7e1198

I will learn from your previous analysis of the thread dumps and I try
to understand what's happening.

Thanks,
Joe

On Tue, Oct 6, 2009 at 10:23 AM, Joe Hansen joe.hansen...@gmail.com wrote:
 Rainer,

 Thanks for looking at those long thread dumps for me!!

 I am sorry. I did NOT take these dumps at the right time (i.e. when
 Tomcat was inundated with requests and couldn't cope with the load).
 After I increased the heap size to 512MB (from 64MB default), I am not
 getting the OutOfMemoryError(s) anymore. After I set KeepAlive On
 (Thanks Andre!), the number httpd processes isn't increasing either.
 The number of httpd processes increased from 8 to 21 and stayed there
 for more than 16 hours now. If the number of httpd processes gets out
 of control again, I will definitely take thread dumps once again.

 However, I doubt the issue is fixed because I should have seen this
 issue long time ago (since I haven't changed any code nor the traffic
 to our websites increased in a long while).

 Should I just wait and see or are there any tests that I can do?

 Your contribution to this forum is amazing, Rainer. I am grateful to
 you and Andre for your efforts. Thank you!

 Regards,
 Joe



 On Tue, Oct 6, 2009 at 7:25 AM, Rainer Jung rainer.j...@kippdata.de wrote:
 On 05.10.2009 18:58, Joe Hansen wrote:
 Thank you so much for your tips, Rainer!

 The websites went down yet again. Increasing the java heap size took
 care of the OutOfMemoryError, but the number of httpd processes keep
 increasing until the websites crash. I haven't added any new code in
 the past few months, hence I am surprised why the requests are getting
 stuck. Here's a link to the tomcat thread dumps:
 http://pastebin.com/m17eea139

 Please let me know if you cannot view it and I will email the relevant
 portion of the catalina.out file to you. Is there an easy way to find
 out what code is causing the requests to get stuck?

 The dump file contains three thread dumps.

 The things all dumps have in common:

 - 60 threads for the quartz scheduler, all idle
 - 13 threads in the AJP connection pool, connected to Apache, but idle
 waiting for the next request to be send (the same threads in all three
 dumps)
 - 6 store plus 6 expiry threads of the EHCache, seems idle
 - 1 AJP + 1 HTTP(S) thread (port 8443) waiting to accept the next new
 connection to come in
 - 2 AJP + 3 HTTP(S) threads (port 8443) sitting idle the pool, waiting
 for work
 - a couple of other normal threads not directly related to request handling

 So the time you took the three dumps, this Tomcat was completely idle
 and did not have a single request to handle.

 If you are completely sure, you took the dumps while there was a storm
 of requests and your system couldn't cope the load, something has
 prevented the requests to ever reach Tomcat.

 I don't have your Tomcat version at hand at the moment, but for some
 time very special OutOfMemory errors (could not create native thread)
 lead to a situation, where Tomcat simply wouldn't accept any new
 connections. Although you report OutOfMemory errors, I'm not directly
 suggesting that that is your problem here. There might still be a
 relation though.

 Are you sure, that you took the dumps for the right Tomcat at the right
 time?

 Regards,

 Rainer

 -
 To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: java.lang.OutOfMemoryError: Java heap space

[Joe Hansen]

There were 29 httpd processes running when the websites crashed.

One thing that was clear from the thread dumps was there were lots of
entries for org.jasig.cas.ticket.registry.support.DefaultTicketRegistryCleaner:
Starting cleaning of expired tickets from ticket registry at ...

Does that indicate that there is something wrong with my CAS configuration?

Here's the thread count from the first thread dump:
DefaultQuartzScheduler_QuartzSchedulerThread
4 sleeping + 2 waiting on condition = 6 Total

DefaultQuartzScheduler_Worker
60 waiting = 60 Total

Store ticketCache Expiry Thread (ehcache)
6 waiting on condition = 6 Total

Store ticketCache Spool Thread (ehcache)
6 waiting on condition = 6 Total

TP-ProcessorXY daemon runnable
20 runnable (mod_jk?) + 3 waiting = 23 Total

Java2D Disposer daemon
1 waiting = 1 Total

http-8443-Monitor
1 waiting = 1 Total

http-8443-Processor
3 waiting + 1 runnable = 4 Total

ContainerBackgroundProcessor
1 waiting on condition = 1 Total

Low Memory Detector
1 runnable = 1 Total

CompilerThread
2 waiting on condition = 2 Total

AdapterThread
1 waiting on condition = 1 Total

Signal Dispatcher
1 runnable = 1 Total

Finalizer
1 waiting = 1 Total

Reference Handler
1 waiting = 1 Total

org.apache.catalina.startup.Bootstrap.main
1 runnable = 1 Total

VM Thread
1 runnable = 1 Total

GC task thread
1 runnable = 1 Total

VM Periodic Task Thread
1 waiting on condition = 1 Total


Thanks,
Joe

On Tue, Oct 6, 2009 at 11:44 AM, Joe Hansen joe.hansen...@gmail.com wrote:
 Rainer,

 I spoke to soon! As I suspected, the problem isn't fixed yet and the
 websites crashed again. This time I took three thread dumps at the
 time tomcat was down. Here they are: http://pastebin.com/m2a7e1198

 I will learn from your previous analysis of the thread dumps and I try
 to understand what's happening.

 Thanks,
 Joe

 On Tue, Oct 6, 2009 at 10:23 AM, Joe Hansen joe.hansen...@gmail.com wrote:
 Rainer,

 Thanks for looking at those long thread dumps for me!!

 I am sorry. I did NOT take these dumps at the right time (i.e. when
 Tomcat was inundated with requests and couldn't cope with the load).
 After I increased the heap size to 512MB (from 64MB default), I am not
 getting the OutOfMemoryError(s) anymore. After I set KeepAlive On
 (Thanks Andre!), the number httpd processes isn't increasing either.
 The number of httpd processes increased from 8 to 21 and stayed there
 for more than 16 hours now. If the number of httpd processes gets out
 of control again, I will definitely take thread dumps once again.

 However, I doubt the issue is fixed because I should have seen this
 issue long time ago (since I haven't changed any code nor the traffic
 to our websites increased in a long while).

 Should I just wait and see or are there any tests that I can do?

 Your contribution to this forum is amazing, Rainer. I am grateful to
 you and Andre for your efforts. Thank you!

 Regards,
 Joe



 On Tue, Oct 6, 2009 at 7:25 AM, Rainer Jung rainer.j...@kippdata.de wrote:
 On 05.10.2009 18:58, Joe Hansen wrote:
 Thank you so much for your tips, Rainer!

 The websites went down yet again. Increasing the java heap size took
 care of the OutOfMemoryError, but the number of httpd processes keep
 increasing until the websites crash. I haven't added any new code in
 the past few months, hence I am surprised why the requests are getting
 stuck. Here's a link to the tomcat thread dumps:
 http://pastebin.com/m17eea139

 Please let me know if you cannot view it and I will email the relevant
 portion of the catalina.out file to you. Is there an easy way to find
 out what code is causing the requests to get stuck?

 The dump file contains three thread dumps.

 The things all dumps have in common:

 - 60 threads for the quartz scheduler, all idle
 - 13 threads in the AJP connection pool, connected to Apache, but idle
 waiting for the next request to be send (the same threads in all three
 dumps)
 - 6 store plus 6 expiry threads of the EHCache, seems idle
 - 1 AJP + 1 HTTP(S) thread (port 8443) waiting to accept the next new
 connection to come in
 - 2 AJP + 3 HTTP(S) threads (port 8443) sitting idle the pool, waiting
 for work
 - a couple of other normal threads not directly related to request handling

 So the time you took the three dumps, this Tomcat was completely idle
 and did not have a single request to handle.

 If you are completely sure, you took the dumps while there was a storm
 of requests and your system couldn't cope the load, something has
 prevented the requests to ever reach Tomcat.

 I don't have your Tomcat version at hand at the moment, but for some
 time very special OutOfMemory errors (could not create native thread)
 lead to a situation, where Tomcat simply wouldn't accept any new
 connections. Although you report OutOfMemory errors, I'm not directly
 suggesting that that is your problem here. There might still be a
 relation though.

 Are you sure, that you took the dumps for the right Tomcat at the right

Re: java.lang.OutOfMemoryError: Java heap space

[Rainer Jung]

On 06.10.2009 19:44, Joe Hansen wrote:
I will only comment the threads for the AJP pool. Everything else seems
not relevant:

Dump 1:

20 threads connected to Apache, waiting for the next request
3  threads idle in the pool
1  thread waiting for the next connection
0  threads working on requests

Dump 2:

12 threads connected to Apache, waiting for the next request
5  threads idle in the pool
1  thread waiting for the next connection
14 threads working on requests

Dump 3:

11 threads connected to Apache, waiting for the next request
5  threads idle in the pool
1  thread waiting for the next connection
15 threads working on requests

Dump 4:

11 threads connected to Apache, waiting for the next request
5  threads idle in the pool
1  thread waiting for the next connection
15 threads working on requests

Those busy threads in dump 2, 3 and 4 are the same except for one, which
started only in dump 3.

Most of them are busy working on data. They are *not* waiting for some
other system, like database or similar.

Out of the 14+15+15 thread stacks, that are interesting, 42 work on some
DAOs.

Those are the DAO methods they work on:

  18 tw.beans.TWFotoSetDAO.getFotosetFotos
  11 tw.beans.TWEmailUpdateListDAO.getEmailUpdateList
   3 tw.beans.TWFotoSetListDAO.getAllFotosets
   3 tw.beans.TWFotoSetDAO.getFotosetFotosQuery2
   3 tw.beans.TWFotoSetDAO.getFotosetFotosQuery
   3 tw.beans.TWEmailUpdateDAO.getEmailUpdate
   1 tw.beans.TWEmailUpdateListDAO.getEmailUpdateListQuery

It seems your application is CPU heavy. Either the data objects handled
are to heavy weight (maybe some user having huge Fotoset or Email list)
or the request rate is simply to large. Is the CPU saturated during the
problems?

I would activate a good access log and try to find out from that and
your webapp logs what maybe special about these web requests or users.

Regards,

Rainer

 I will learn from your previous analysis of the thread dumps and I try
 to understand what's happening.
 
 Thanks,
 Joe
 
 On Tue, Oct 6, 2009 at 10:23 AM, Joe Hansen joe.hansen...@gmail.com wrote:
 Rainer,

 Thanks for looking at those long thread dumps for me!!

 I am sorry. I did NOT take these dumps at the right time (i.e. when
 Tomcat was inundated with requests and couldn't cope with the load).
 After I increased the heap size to 512MB (from 64MB default), I am not
 getting the OutOfMemoryError(s) anymore. After I set KeepAlive On
 (Thanks Andre!), the number httpd processes isn't increasing either.
 The number of httpd processes increased from 8 to 21 and stayed there
 for more than 16 hours now. If the number of httpd processes gets out
 of control again, I will definitely take thread dumps once again.

 However, I doubt the issue is fixed because I should have seen this
 issue long time ago (since I haven't changed any code nor the traffic
 to our websites increased in a long while).

 Should I just wait and see or are there any tests that I can do?

 Your contribution to this forum is amazing, Rainer. I am grateful to
 you and Andre for your efforts. Thank you!

 Regards,
 Joe



 On Tue, Oct 6, 2009 at 7:25 AM, Rainer Jung rainer.j...@kippdata.de wrote:
 On 05.10.2009 18:58, Joe Hansen wrote:
 Thank you so much for your tips, Rainer!

 The websites went down yet again. Increasing the java heap size took
 care of the OutOfMemoryError, but the number of httpd processes keep
 increasing until the websites crash. I haven't added any new code in
 the past few months, hence I am surprised why the requests are getting
 stuck. Here's a link to the tomcat thread dumps:
 http://pastebin.com/m17eea139

 Please let me know if you cannot view it and I will email the relevant
 portion of the catalina.out file to you. Is there an easy way to find
 out what code is causing the requests to get stuck?

 The dump file contains three thread dumps.

 The things all dumps have in common:

 - 60 threads for the quartz scheduler, all idle
 - 13 threads in the AJP connection pool, connected to Apache, but idle
 waiting for the next request to be send (the same threads in all three
 dumps)
 - 6 store plus 6 expiry threads of the EHCache, seems idle
 - 1 AJP + 1 HTTP(S) thread (port 8443) waiting to accept the next new
 connection to come in
 - 2 AJP + 3 HTTP(S) threads (port 8443) sitting idle the pool, waiting
 for work
 - a couple of other normal threads not directly related to request handling

 So the time you took the three dumps, this Tomcat was completely idle
 and did not have a single request to handle.

 If you are completely sure, you took the dumps while there was a storm
 of requests and your system couldn't cope the load, something has
 prevented the requests to ever reach Tomcat.

 I don't have your Tomcat version at hand at the moment, but for some
 time very special OutOfMemory errors (could not create native thread)
 lead to a situation, where Tomcat simply wouldn't accept any new
 connections. Although you report OutOfMemory errors, I'm not directly
 

Re: java.lang.OutOfMemoryError: Java heap space

[Joe Hansen]

Thanks for your feedback, Rainer.

The DAO objects/methods that you've mentioned are used for almost
every page on our website. So I am not surprised to find them in the
thread dumps. However, as you pointed out, the parameters to these
methods change depending on the page they are on and if the user is
logged in or not.

I did not check the CPU usage when the issue happened.  I just checked
the web application logs and found that first there was an
OutOfMemoryError, followed by MySQL errors when
TWFotoSetListDAO.getAllFotosets tried to get a database connection.

I am *guessing* :
I changed the session timeout to 4 hours for the Jasig CAS
Single-Signon web application. This stores a ton of tickets in memory.
During a 4 hour period when there are more http requests, the 512MB of
Java Heap space is not sufficient to store all these tickets. During
that time OutOfMemoryError happens and all hell breaks loose
subsequently.

I probably should decrease the session-timeout to 1 hour or so and see
if that changes things. Would you agree, Rainer?

Thanks,
Joe

2009 Oct 06 / 10:48:43 ERROR -
[org.apache.catalina.core.ContainerBase] : Exception invoking periodic
operation:
java.lang.OutOfMemoryError: Java heap space

2009 Oct 06 / 10:48:43 FATAL - [tw.conn.TWConnectionManager] :
com.mysql.jdbc.CommunicationsException: Communications link failure
due to underlying exception:

** BEGIN NESTED EXCEPTION **

java.io.EOFException

STACKTRACE:

java.io.EOFException
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2351)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:3649)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1176)
at com.mysql.jdbc.Connection.createNewIO(Connection.java:2558)
at com.mysql.jdbc.Connection.init(Connection.java:1485)
at 
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266)
at 
org.apache.tomcat.dbcp.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:37)
at 
org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290)
at 
org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:771)
at 
org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:74)
at 
org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)
at 
org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540)
at tw.conn.TWConnectionManager.getConnection(Unknown Source)
at tw.beans.TWFotoSetListDAO.getAllFotosets(Unknown Source)
at tw.beans.TWViewBean.getAllFotosets(Unknown Source)
at tw.servlet.TWQueryServlet.doPost(Unknown Source)
at tw.servlet.TWQueryServlet.doGet(Unknown Source)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at 
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:672)
at 
org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:463)
at 
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:398)
at 
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301)
at 
org.apache.jsp.groupby_005fitems_jsp._jspService(groupby_005fitems_jsp.java:152)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at 
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:332)
at 
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432)
at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at 

Re: java.lang.OutOfMemoryError: Java heap space

[André Warnier]

Rainer Jung wrote:

On 06.10.2009 21:44, Joe Hansen wrote:

...


It seems your application is CPU heavy. Either the data objects handled
are to heavy weight (maybe some user having huge Fotoset or Email list)
or the request rate is simply to large. Is the CPU saturated during the
problems?

I would activate a good access log and try to find out from that and
your webapp logs what maybe special about these web requests or users.


...
The original post was so long ago in relative terms, that I don't 
remember the details of your system.
But, just in case, you may want to also have a look at the *total* 
memory usage on your system during those events.
If the load has increased (even slightly) recently, it might have 
reached the point where there is sometimes no longer sufficient physical 
memory available to process all simultaneous tasks, and the system is 
starting to swap tasks to virtual memory (on disk).  That would cause a 
dramatic slowdown in request processing, which may have something to do 
with your problems.
If you are under Unix/Linux, the first lines displayed by top would 
already provide some information in that respect.
Or else, just plug in an additional memory bar in your server, and see 
if it changes anything.  They are rather cheap right now.




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: java.lang.OutOfMemoryError: Java heap space

[Joe Hansen]

I changed the session-timeout for the CAS webapp to 1 hour and
restarted Apache/Tomcat to see if that changes anything. If the issue
manifests again, I will surely check the memory and cpu usage using
the top command.

Rainer, I somewhat doubt that getAllFotosets() is causing the problem.
Because I have an administrator account on the website and I can view
more photosets than any other user. And when I login and view the
photosets nothing alarming seems to be happening. I am not ruling out
the possibility altogether though.

Andre, you are right. Memory is cheap. However, the machine has 8GB of
memory already. I am thinking that it should be sufficient. Here's the
output of the free command:
[r...@csiweb bin]# free -m
 totalused   free
Mem: 8113   7557   555
-/+ buffers/cache:861 7252
Swap:9012  33  8979

Its amazing how you and Andre are able to identify issues with others'
Tomcat installations just by looking at their logs and settings. It
speaks volumes of your experience with Tomcat.

Thank you,
Joe

On Tue, Oct 6, 2009 at 2:00 PM, Rainer Jung rainer.j...@kippdata.de wrote:
 On 06.10.2009 21:44, Joe Hansen wrote:
 Thanks for your feedback, Rainer.

 The DAO objects/methods that you've mentioned are used for almost
 every page on our website. So I am not surprised to find them in the
 thread dumps. However, as you pointed out, the parameters to these
 methods change depending on the page they are on and if the user is
 logged in or not.

 I did not check the CPU usage when the issue happened.  I just checked
 the web application logs and found that first there was an
 OutOfMemoryError, followed by MySQL errors when
 TWFotoSetListDAO.getAllFotosets tried to get a database connection.

 I am *guessing* :
 I changed the session timeout to 4 hours for the Jasig CAS
 Single-Signon web application. This stores a ton of tickets in memory.
 During a 4 hour period when there are more http requests, the 512MB of
 Java Heap space is not sufficient to store all these tickets. During
 that time OutOfMemoryError happens and all hell breaks loose
 subsequently.

 I probably should decrease the session-timeout to 1 hour or so and see
 if that changes things. Would you agree, Rainer?

 Could be. But it could also be, that getAllFotosets() is able to slurp
 in huge amounts of data to memory and that this is the reason for the OOME.

 Once you run into an OOME on heap, there's no good relief than
 restarting the whole JVM. After an OOME you can't prove the correctness
 of your app any more. Random objects might be missing, resulting in
 largely undefined behaviour in your webapp or even Tomcat.

 I'm a bit insisting on the getAllFotosets(), because I once was involved
 in troubleshooting for a social website and part of their problem was
 special users having extremely large social data. So once they hit the
 server it went OOME.

 Thanks,
 Joe

 Regards,

 Rainer


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: java.lang.OutOfMemoryError: Java heap space

[Peter Crowther]

2009/10/6 Joe Hansen joe.hansen...@gmail.com:
 Andre, you are right. Memory is cheap. However, the machine has 8GB of
 memory already. I am thinking that it should be sufficient.

You might wish to dedicate more of that 8 Gbyte to Java, though.  It
depends on the performance characteristics of your application - looks
to me like MySQL is grabbing a load of your RAM for its cache.  top
(or similar) would confirm that overall usage, and I assume MySQL has
some performance monitoring available that'll tell you the
effectiveness of its caches.  If you're *just* tipping into using
swap, it's almost certainly the case that MySQL is grabbing memory
that would otherwise be called free.

This may well be the best use for the RAM, but there's a possibility
that more RAM dedicated to Java would improve your performance
overall.  I'm not willing to make that guess without data on the
effectiveness of MySQL's cache, though :-).

- Peter

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: java.lang.OutOfMemoryError: Java heap space

[Joe Hansen]

Thank you so much for your tips, Rainer!

The websites went down yet again. Increasing the java heap size took
care of the OutOfMemoryError, but the number of httpd processes keep
increasing until the websites crash. I haven't added any new code in
the past few months, hence I am surprised why the requests are getting
stuck. Here's a link to the tomcat thread dumps:
http://pastebin.com/m17eea139

Please let me know if you cannot view it and I will email the relevant
portion of the catalina.out file to you. Is there an easy way to find
out what code is causing the requests to get stuck?

Thank you!

Joe


On Sun, Oct 4, 2009 at 2:36 PM, Rainer Jung rainer.j...@kippdata.de wrote:
 Hi Joe,

 On 04.10.2009 21:45, Joe Hansen wrote:
 Rainer, Thank you so much for your kind reply!

 I have increased the java heap size to 512MB (-Xms512m -Xmx512m). I am
 hoping that would fix the issue. I had configured our webserver to use
 Jasig's Central Authentication System (CAS). Recently I increased the
 session timeout from 30 minutes to 4 hours. I am guessing that must
 have had an impact on the number of tickets that the CAS could store
 in the Java's memory space.

 I did run the kill -QUIT command against the tomcat process. It did
 generate a huge output in the catalina.out file. I am unable to
 decipher it. I do not want to post it to the mailing list because its
 very long. Would you be able to please tell me what should I be
 looking for within this long thread dump?

 Can you put it somewhere on the web, so we can look at it, or are you
 afraid there is something private in there? You could use pastebin or
 something similar in case you do not have a public web server yourself.

 If you don't want to post in public, you can also mail it to me, I will
 post the result, in case I find something relevant.

 Regards,

 Rainer

 On Sat, Oct 3, 2009 at 12:24 PM, Rainer Jung rainer.j...@kippdata.de wrote:
 On 03.10.2009 20:07, Joe Hansen wrote:
 Hey All,

 I get this error (java.lang.OutOfMemoryError: Java heap space) after
 my Apache 2.0/Tomcat 5.5/mod_jk installation has been up and running
 for a few hours. This problem started just since two days. Never had
 this issue before!

 I have also noticed that as soon as I startup the server, 9 httpd
 processes start. Number of httpd processes keep on increasing until I
 get the OutOfMemoryError.
 $ps -aef | grep httpd
 root     31984     1  0 11:23 ?        00:00:00 /usr/sbin/httpd
 apache   31987 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd
 apache   31988 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd
 apache   31989 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd
 apache   31990 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd
 apache   31991 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd
 apache   31992 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd
 apache   31993 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd
 apache   31994 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd

 Sounds like requests get stuck or responses are only returned very slowly.

 I would take thread dumps during the time requests pile up (e.g. httpd
 process count increases). Thread dumps are generated by kil -QUIT
 against the Tomcat process. Result is written to catalina.out. Always
 take afew thread dumps shortly after each other, e.g. 3 dumps each 3
 seconds apart from the previous one, so that you can find out, if a
 status in a dump is pure coincidence or lasts for somewhat longer.

 $ps -aef | grep tomcat
 root     31949     1 43 11:23 pts/0    00:00:58 /usr/java/jdk/bin/java 
 -Djava.u
 il.logging.manager=org.apache.juli.ClassLoaderLogManager 
 -Djava.util.logging.co
 fig.file=/usr/lib/apache-tomcat/conf/logging.properties 
 -Djava.endorsed.dirs=/u
 r/lib/apache-tomcat/common/endorsed -classpath 
 :/usr/lib/apache-tomcat/bin/boot
 trap.jar:/usr/lib/apache-tomcat/bin/commons-logging-api.jar 
 -Dcatalina.base=/us
 /lib/apache-tomcat -Dcatalina.home=/usr/lib/apache-tomcat 
 -Djava.io.tmpdir=/usr
 lib/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start

 There is no Java memory configuration included above (i.e. al defaults).
 It might well be, that you have to explicitely set heap size, perm size
 and if you like also eden and semi spaces.

 Can someone on this list please help me resolve this issue.

 Thanks you,
 Joe

 Regards,

 Rainer


 -
 To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: java.lang.OutOfMemoryError: Java heap space

[Joe Hansen]

Rainer,

Here are the KeepAlive values in httpd.conf:

KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimout 15

Thanks,
Joe

 What are your KeepAlive* settings ?


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: java.lang.OutOfMemoryError: Java heap space

[André Warnier]

Joe Hansen wrote:

Rainer,

Here are the KeepAlive values in httpd.conf:

KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimout 15


Well, since you have KeepAlive Off, the other 2 do not matter.
But as such, it means that each request of each browser is going to 
create a new connection to the webserver, just for that one request.
So if there is a page with 10 img links inside, you will end up 
establishing (and tearing down) a total of 11 TCP connections (one for 
the main page, one each for each img).

That may or may not have a bearing on the situation you are seeing.



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: java.lang.OutOfMemoryError: Java heap space

[Joe Hansen]

Thank you for the reply, Andre.

I now understand how setting KeepAlive to On would improve the
performance of a website (The Apache manual says that a 50% increase
in throughput could be expected). So I changed the KeepAlive to On and
restarted the server.

I however wonder if this will fix the issue. The reason being, I
haven't changed the website code at all the past few months and there
hasn't been any increase in the website traffic too. Hence I am unable
to understand why we are suddenly seeing an increase in the number of
httpd processes. The only thing I changed is the session-timeout value
from 30 minutes to 240 minutes.

Thanks,
Joe




On Mon, Oct 5, 2009 at 1:04 PM, André Warnier a...@ice-sa.com wrote:
 Joe Hansen wrote:

 Rainer,

 Here are the KeepAlive values in httpd.conf:

 KeepAlive Off
 MaxKeepAliveRequests 100
 KeepAliveTimout 15

 Well, since you have KeepAlive Off, the other 2 do not matter.
 But as such, it means that each request of each browser is going to create a
 new connection to the webserver, just for that one request.
 So if there is a page with 10 img links inside, you will end up
 establishing (and tearing down) a total of 11 TCP connections (one for the
 main page, one each for each img).
 That may or may not have a bearing on the situation you are seeing.



 -
 To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: java.lang.OutOfMemoryError: Java heap space

[André Warnier]

Joe Hansen wrote:

Thank you for the reply, Andre.

I now understand how setting KeepAlive to On would improve the
performance of a website (The Apache manual says that a 50% increase
in throughput could be expected). So I changed the KeepAlive to On and
restarted the server.


Now wait.
You should probably then lower your setting for KeepAliveTimeout (to 3 
e.g.), otherwise you may make the problem much worse.

Read conscienciously the relevant Apache doc page :
http://httpd.apache.org/docs/2.2/mod/core.html#keepalive

The point with KeepAlive is :
- the browser makes a connection and issues a first request
- the webserver dedicates a child (or thread) to this connection, and 
passes it the first request

- the child/thread responds to the first request, and then waits for more
- the browser, in the response page, finds more links. Over the same TCP 
connection, it sends the next request
- the same child/thread - which was waiting on that connection - 
receives the new request, and responds to it. Then it waits again for 
the next one.

- etc..
- until at some point, the browser does not issue any additional 
requests on the connection. Then, *after the KeepAliveTimeout has 
expired*, the child/thread gives up, closesthe connection, and returns 
to the pool available for other requests from other browsers


So the point is, if the KeepAliveTimeout is long (like 15 seconds), it 
means that a child/thread may be kept waiting, for nothing, up to that 
many seconds, although there is nothing coming anymore.




I however wonder if this will fix the issue. The reason being, I
haven't changed the website code at all the past few months and there
hasn't been any increase in the website traffic too. Hence I am unable
to understand why we are suddenly seeing an increase in the number of
httpd processes. The only thing I changed is the session-timeout value
from 30 minutes to 240 minutes.

I guess that this is the Tomcat session timeout.  That should have 
nothing to do with the above.  I don't think that for Tomcat, a 
session is linked to a connection. It is more of a set of data saved 
somewhere, linked to the Tomcat session-id (the JSESSIONID cookie for 
instance).  Tomcar retrieves it whenever a request comes in with the 
same session-id number.  But it should not matter whether it is on the 
same TCP connection or not.


What may be linked together however, is that one request to httpd 
results in one child/thread busy with it at the Apache httpd level. If 
that request is being forwarded to Tomcat by mod_jk, then it also holds 
onto one mod_jk/Tomcat connection. This connection then holds on to one 
thread in Tomcat, until the Tomcat thread (+webapp) has supplied the 
full response. All the while, this whole chain is unavailable for other 
requests.  Thus, if there are many such requests under way, many Apache 
children/threads are busy, and Apache httpd will start additional ones 
(up to its limit) to service new requests that come in.
So if for some reason, your Tomcat requests now take longer to be 
serviced, that should also, by retro-effect, increase the number of 
httpd children/threads being started.

The bottleneck would be in Tomcat, but it would show up at the httpd level.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: java.lang.OutOfMemoryError: Java heap space

[André Warnier]

André Warnier wrote:
...
and still wants to add something :

- a new KeepAlive connection is made from the browser to Apache (httpd).
- then a request comes in on that connection, and it happens to be one 
that gets forwarded to Tomcat.  So a mod_jk connection is made to 
Tomcat, Tomcat allocates a thread for it.
- I would imagine that mod_jk must pass on to Tomcat the fact that this 
is a KeepAlive connection, so that Tomcat would know that its thread for 
that connection, should also wait for subsequent requests.
- so now the webapp/thread generates the response to the first request, 
and waits on the connection for more requests.
- however, the browser does send more requests to Apache, but these are 
not ones that get forwarded to Tomcat (for example, they are for items 
that Apache serves locally) ...


So now I wonder about how Apache + mod_jk + Tomcat react in such a 
situation.  Do the mod_jk connection +Tomcat thread keep waiting anyway, 
and how long ?


Rainer ?


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: java.lang.OutOfMemoryError: Java heap space

[Caldarale, Charles R]

 From: André Warnier [mailto:a...@ice-sa.com]
 Subject: Re: java.lang.OutOfMemoryError: Java heap space
 
 The bottleneck would be in Tomcat, but it would show up at the
 httpd level.

The bottleneck might also be in something external to Tomcat, such as a 
database or some external web service used by the webapps.  The result may 
still only be visible at the front end.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: java.lang.OutOfMemoryError: Java heap space

[Joe Hansen]

Andre, Thanks for pointing out the high KeepAliveTimeout value in the
config file. I have read the docs and have changed it to 5 seconds
(which is the default).

I am hoping that Rainer could find out from the thread dump where the
requests are getting stuck, so that I can put this issue to bed.

Thanks,
Joe


On Mon, Oct 5, 2009 at 1:53 PM, André Warnier a...@ice-sa.com wrote:
 Joe Hansen wrote:

 Thank you for the reply, Andre.

 I now understand how setting KeepAlive to On would improve the
 performance of a website (The Apache manual says that a 50% increase
 in throughput could be expected). So I changed the KeepAlive to On and
 restarted the server.

 Now wait.
 You should probably then lower your setting for KeepAliveTimeout (to 3
 e.g.), otherwise you may make the problem much worse.
 Read conscienciously the relevant Apache doc page :
 http://httpd.apache.org/docs/2.2/mod/core.html#keepalive

 The point with KeepAlive is :
 - the browser makes a connection and issues a first request
 - the webserver dedicates a child (or thread) to this connection, and passes
 it the first request
 - the child/thread responds to the first request, and then waits for more
 - the browser, in the response page, finds more links. Over the same TCP
 connection, it sends the next request
 - the same child/thread - which was waiting on that connection - receives
 the new request, and responds to it. Then it waits again for the next one.
 - etc..
 - until at some point, the browser does not issue any additional requests on
 the connection. Then, *after the KeepAliveTimeout has expired*, the
 child/thread gives up, closesthe connection, and returns to the pool
 available for other requests from other browsers

 So the point is, if the KeepAliveTimeout is long (like 15 seconds), it means
 that a child/thread may be kept waiting, for nothing, up to that many
 seconds, although there is nothing coming anymore.


 I however wonder if this will fix the issue. The reason being, I
 haven't changed the website code at all the past few months and there
 hasn't been any increase in the website traffic too. Hence I am unable
 to understand why we are suddenly seeing an increase in the number of
 httpd processes. The only thing I changed is the session-timeout value
 from 30 minutes to 240 minutes.

 I guess that this is the Tomcat session timeout.  That should have nothing
 to do with the above.  I don't think that for Tomcat, a session is linked
 to a connection. It is more of a set of data saved somewhere, linked to the
 Tomcat session-id (the JSESSIONID cookie for instance).  Tomcar retrieves it
 whenever a request comes in with the same session-id number.  But it should
 not matter whether it is on the same TCP connection or not.

 What may be linked together however, is that one request to httpd results in
 one child/thread busy with it at the Apache httpd level. If that request is
 being forwarded to Tomcat by mod_jk, then it also holds onto one
 mod_jk/Tomcat connection. This connection then holds on to one thread in
 Tomcat, until the Tomcat thread (+webapp) has supplied the full response.
 All the while, this whole chain is unavailable for other requests.  Thus, if
 there are many such requests under way, many Apache children/threads are
 busy, and Apache httpd will start additional ones (up to its limit) to
 service new requests that come in.
 So if for some reason, your Tomcat requests now take longer to be serviced,
 that should also, by retro-effect, increase the number of httpd
 children/threads being started.
 The bottleneck would be in Tomcat, but it would show up at the httpd level.

 -
 To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: java.lang.OutOfMemoryError: Java heap space

[Joe Hansen]

Rainer, Thank you so much for your kind reply!

I have increased the java heap size to 512MB (-Xms512m -Xmx512m). I am
hoping that would fix the issue. I had configured our webserver to use
Jasig's Central Authentication System (CAS). Recently I increased the
session timeout from 30 minutes to 4 hours. I am guessing that must
have had an impact on the number of tickets that the CAS could store
in the Java's memory space.

I did run the kill -QUIT command against the tomcat process. It did
generate a huge output in the catalina.out file. I am unable to
decipher it. I do not want to post it to the mailing list because its
very long. Would you be able to please tell me what should I be
looking for within this long thread dump?

Thanks again, Rainer :)

Joe

On Sat, Oct 3, 2009 at 12:24 PM, Rainer Jung rainer.j...@kippdata.de wrote:
 On 03.10.2009 20:07, Joe Hansen wrote:
 Hey All,

 I get this error (java.lang.OutOfMemoryError: Java heap space) after
 my Apache 2.0/Tomcat 5.5/mod_jk installation has been up and running
 for a few hours. This problem started just since two days. Never had
 this issue before!

 I have also noticed that as soon as I startup the server, 9 httpd
 processes start. Number of httpd processes keep on increasing until I
 get the OutOfMemoryError.
 $ps -aef | grep httpd
 root     31984     1  0 11:23 ?        00:00:00 /usr/sbin/httpd
 apache   31987 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd
 apache   31988 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd
 apache   31989 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd
 apache   31990 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd
 apache   31991 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd
 apache   31992 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd
 apache   31993 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd
 apache   31994 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd

 Sounds like requests get stuck or responses are only returned very slowly.

 I would take thread dumps during the time requests pile up (e.g. httpd
 process count increases). Thread dumps are generated by kil -QUIT
 against the Tomcat process. Result is written to catalina.out. Always
 take afew thread dumps shortly after each other, e.g. 3 dumps each 3
 seconds apart from the previous one, so that you can find out, if a
 status in a dump is pure coincidence or lasts for somewhat longer.

 $ps -aef | grep tomcat
 root     31949     1 43 11:23 pts/0    00:00:58 /usr/java/jdk/bin/java 
 -Djava.u
 il.logging.manager=org.apache.juli.ClassLoaderLogManager 
 -Djava.util.logging.co
 fig.file=/usr/lib/apache-tomcat/conf/logging.properties 
 -Djava.endorsed.dirs=/u
 r/lib/apache-tomcat/common/endorsed -classpath 
 :/usr/lib/apache-tomcat/bin/boot
 trap.jar:/usr/lib/apache-tomcat/bin/commons-logging-api.jar 
 -Dcatalina.base=/us
 /lib/apache-tomcat -Dcatalina.home=/usr/lib/apache-tomcat 
 -Djava.io.tmpdir=/usr
 lib/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start

 There is no Java memory configuration included above (i.e. al defaults).
 It might well be, that you have to explicitely set heap size, perm size
 and if you like also eden and semi spaces.

 Can someone on this list please help me resolve this issue.

 Thanks you,
 Joe

 Regards,

 Rainer

 -
 To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: java.lang.OutOfMemoryError: Java heap space

[Joe Hansen]

I found the following error message in the Apache logs:
[Sat Oct 03 04:10:49 2009] [error] server reached MaxClients setting,
consider raising the MaxClients setting

Here's a snippet from the httpd.conf, which deals with MaxClients.
IfModule prefork.c
StartServers   8
MinSpareServers5
MaxSpareServers   20
ServerLimit  256
MaxClients   256
MaxRequestsPerChild  4000
/IfModule

IfModule worker.c
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild  0
/IfModule

I will watch out for the increase in the number of httpd processes. I
am wondering if I should raise the MaxClients value in prefork.c and
worker.c modules. Can anyone on this forum please explain why new
httpd processes are spawned and why aren't the old processes
terminated?

Thanks,
Joe

On Sun, Oct 4, 2009 at 1:45 PM, Joe Hansen joe.hansen...@gmail.com wrote:
 Rainer, Thank you so much for your kind reply!

 I have increased the java heap size to 512MB (-Xms512m -Xmx512m). I am
 hoping that would fix the issue. I had configured our webserver to use
 Jasig's Central Authentication System (CAS). Recently I increased the
 session timeout from 30 minutes to 4 hours. I am guessing that must
 have had an impact on the number of tickets that the CAS could store
 in the Java's memory space.

 I did run the kill -QUIT command against the tomcat process. It did
 generate a huge output in the catalina.out file. I am unable to
 decipher it. I do not want to post it to the mailing list because its
 very long. Would you be able to please tell me what should I be
 looking for within this long thread dump?

 Thanks again, Rainer :)

 Joe

 On Sat, Oct 3, 2009 at 12:24 PM, Rainer Jung rainer.j...@kippdata.de wrote:
 On 03.10.2009 20:07, Joe Hansen wrote:
 Hey All,

 I get this error (java.lang.OutOfMemoryError: Java heap space) after
 my Apache 2.0/Tomcat 5.5/mod_jk installation has been up and running
 for a few hours. This problem started just since two days. Never had
 this issue before!

 I have also noticed that as soon as I startup the server, 9 httpd
 processes start. Number of httpd processes keep on increasing until I
 get the OutOfMemoryError.
 $ps -aef | grep httpd
 root     31984     1  0 11:23 ?        00:00:00 /usr/sbin/httpd
 apache   31987 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd
 apache   31988 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd
 apache   31989 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd
 apache   31990 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd
 apache   31991 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd
 apache   31992 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd
 apache   31993 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd
 apache   31994 31984  0 11:23 ?        00:00:00 /usr/sbin/httpd

 Sounds like requests get stuck or responses are only returned very slowly.

 I would take thread dumps during the time requests pile up (e.g. httpd
 process count increases). Thread dumps are generated by kil -QUIT
 against the Tomcat process. Result is written to catalina.out. Always
 take afew thread dumps shortly after each other, e.g. 3 dumps each 3
 seconds apart from the previous one, so that you can find out, if a
 status in a dump is pure coincidence or lasts for somewhat longer.

 $ps -aef | grep tomcat
 root     31949     1 43 11:23 pts/0    00:00:58 /usr/java/jdk/bin/java 
 -Djava.u
 il.logging.manager=org.apache.juli.ClassLoaderLogManager 
 -Djava.util.logging.co
 fig.file=/usr/lib/apache-tomcat/conf/logging.properties 
 -Djava.endorsed.dirs=/u
 r/lib/apache-tomcat/common/endorsed -classpath 
 :/usr/lib/apache-tomcat/bin/boot
 trap.jar:/usr/lib/apache-tomcat/bin/commons-logging-api.jar 
 -Dcatalina.base=/us
 /lib/apache-tomcat -Dcatalina.home=/usr/lib/apache-tomcat 
 -Djava.io.tmpdir=/usr
 lib/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start

 There is no Java memory configuration included above (i.e. al defaults).
 It might well be, that you have to explicitely set heap size, perm size
 and if you like also eden and semi spaces.

 Can someone on this list please help me resolve this issue.

 Thanks you,
 Joe

 Regards,

 Rainer

 -
 To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: java.lang.OutOfMemoryError: Java heap space

[Rainer Jung]

On 04.10.2009 22:17, Joe Hansen wrote:
 I found the following error message in the Apache logs:
 [Sat Oct 03 04:10:49 2009] [error] server reached MaxClients setting,
 consider raising the MaxClients setting
 
 Here's a snippet from the httpd.conf, which deals with MaxClients.
 IfModule prefork.c
 StartServers   8
 MinSpareServers5
 MaxSpareServers   20
 ServerLimit  256
 MaxClients   256
 MaxRequestsPerChild  4000
 /IfModule
 
 IfModule worker.c
 StartServers 2
 MaxClients 150
 MinSpareThreads 25
 MaxSpareThreads 75
 ThreadsPerChild 25
 MaxRequestsPerChild  0
 /IfModule
 
 I will watch out for the increase in the number of httpd processes. I
 am wondering if I should raise the MaxClients value in prefork.c and
 worker.c modules. Can anyone on this forum please explain why new
 httpd processes are spawned and why aren't the old processes
 terminated?

Likely because requests get stuck (no responses are returned), so the
web server waits very long for Tomcat to produce a request and the
stream of incoming new requests fill up all available threads in the web
server. To find the reason, the thread dumps will be helpful.

Regards,

Rainer

 On Sun, Oct 4, 2009 at 1:45 PM, Joe Hansen joe.hansen...@gmail.com wrote:
 Rainer, Thank you so much for your kind reply!

 I have increased the java heap size to 512MB (-Xms512m -Xmx512m). I am
 hoping that would fix the issue. I had configured our webserver to use
 Jasig's Central Authentication System (CAS). Recently I increased the
 session timeout from 30 minutes to 4 hours. I am guessing that must
 have had an impact on the number of tickets that the CAS could store
 in the Java's memory space.

 I did run the kill -QUIT command against the tomcat process. It did
 generate a huge output in the catalina.out file. I am unable to
 decipher it. I do not want to post it to the mailing list because its
 very long. Would you be able to please tell me what should I be
 looking for within this long thread dump?

 Thanks again, Rainer :)

 Joe

 On Sat, Oct 3, 2009 at 12:24 PM, Rainer Jung rainer.j...@kippdata.de wrote:
 On 03.10.2009 20:07, Joe Hansen wrote:
 Hey All,

 I get this error (java.lang.OutOfMemoryError: Java heap space) after
 my Apache 2.0/Tomcat 5.5/mod_jk installation has been up and running
 for a few hours. This problem started just since two days. Never had
 this issue before!

 I have also noticed that as soon as I startup the server, 9 httpd
 processes start. Number of httpd processes keep on increasing until I
 get the OutOfMemoryError.
 $ps -aef | grep httpd
 root 31984 1  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31987 31984  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31988 31984  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31989 31984  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31990 31984  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31991 31984  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31992 31984  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31993 31984  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31994 31984  0 11:23 ?00:00:00 /usr/sbin/httpd

 Sounds like requests get stuck or responses are only returned very slowly.

 I would take thread dumps during the time requests pile up (e.g. httpd
 process count increases). Thread dumps are generated by kil -QUIT
 against the Tomcat process. Result is written to catalina.out. Always
 take afew thread dumps shortly after each other, e.g. 3 dumps each 3
 seconds apart from the previous one, so that you can find out, if a
 status in a dump is pure coincidence or lasts for somewhat longer.

 $ps -aef | grep tomcat
 root 31949 1 43 11:23 pts/000:00:58 /usr/java/jdk/bin/java 
 -Djava.u
 il.logging.manager=org.apache.juli.ClassLoaderLogManager 
 -Djava.util.logging.co
 fig.file=/usr/lib/apache-tomcat/conf/logging.properties 
 -Djava.endorsed.dirs=/u
 r/lib/apache-tomcat/common/endorsed -classpath 
 :/usr/lib/apache-tomcat/bin/boot
 trap.jar:/usr/lib/apache-tomcat/bin/commons-logging-api.jar 
 -Dcatalina.base=/us
 /lib/apache-tomcat -Dcatalina.home=/usr/lib/apache-tomcat 
 -Djava.io.tmpdir=/usr
 lib/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start

 There is no Java memory configuration included above (i.e. al defaults).
 It might well be, that you have to explicitely set heap size, perm size
 and if you like also eden and semi spaces.

 Can someone on this list please help me resolve this issue.

 Thanks you,
 Joe

 Regards,

 Rainer

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: java.lang.OutOfMemoryError: Java heap space

[Rainer Jung]

Hi Joe,

On 04.10.2009 21:45, Joe Hansen wrote:
 Rainer, Thank you so much for your kind reply!
 
 I have increased the java heap size to 512MB (-Xms512m -Xmx512m). I am
 hoping that would fix the issue. I had configured our webserver to use
 Jasig's Central Authentication System (CAS). Recently I increased the
 session timeout from 30 minutes to 4 hours. I am guessing that must
 have had an impact on the number of tickets that the CAS could store
 in the Java's memory space.
 
 I did run the kill -QUIT command against the tomcat process. It did
 generate a huge output in the catalina.out file. I am unable to
 decipher it. I do not want to post it to the mailing list because its
 very long. Would you be able to please tell me what should I be
 looking for within this long thread dump?

Can you put it somewhere on the web, so we can look at it, or are you
afraid there is something private in there? You could use pastebin or
something similar in case you do not have a public web server yourself.

If you don't want to post in public, you can also mail it to me, I will
post the result, in case I find something relevant.

Regards,

Rainer

 On Sat, Oct 3, 2009 at 12:24 PM, Rainer Jung rainer.j...@kippdata.de wrote:
 On 03.10.2009 20:07, Joe Hansen wrote:
 Hey All,

 I get this error (java.lang.OutOfMemoryError: Java heap space) after
 my Apache 2.0/Tomcat 5.5/mod_jk installation has been up and running
 for a few hours. This problem started just since two days. Never had
 this issue before!

 I have also noticed that as soon as I startup the server, 9 httpd
 processes start. Number of httpd processes keep on increasing until I
 get the OutOfMemoryError.
 $ps -aef | grep httpd
 root 31984 1  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31987 31984  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31988 31984  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31989 31984  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31990 31984  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31991 31984  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31992 31984  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31993 31984  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31994 31984  0 11:23 ?00:00:00 /usr/sbin/httpd

 Sounds like requests get stuck or responses are only returned very slowly.

 I would take thread dumps during the time requests pile up (e.g. httpd
 process count increases). Thread dumps are generated by kil -QUIT
 against the Tomcat process. Result is written to catalina.out. Always
 take afew thread dumps shortly after each other, e.g. 3 dumps each 3
 seconds apart from the previous one, so that you can find out, if a
 status in a dump is pure coincidence or lasts for somewhat longer.

 $ps -aef | grep tomcat
 root 31949 1 43 11:23 pts/000:00:58 /usr/java/jdk/bin/java 
 -Djava.u
 il.logging.manager=org.apache.juli.ClassLoaderLogManager 
 -Djava.util.logging.co
 fig.file=/usr/lib/apache-tomcat/conf/logging.properties 
 -Djava.endorsed.dirs=/u
 r/lib/apache-tomcat/common/endorsed -classpath 
 :/usr/lib/apache-tomcat/bin/boot
 trap.jar:/usr/lib/apache-tomcat/bin/commons-logging-api.jar 
 -Dcatalina.base=/us
 /lib/apache-tomcat -Dcatalina.home=/usr/lib/apache-tomcat 
 -Djava.io.tmpdir=/usr
 lib/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start

 There is no Java memory configuration included above (i.e. al defaults).
 It might well be, that you have to explicitely set heap size, perm size
 and if you like also eden and semi spaces.

 Can someone on this list please help me resolve this issue.

 Thanks you,
 Joe

 Regards,

 Rainer


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: java.lang.OutOfMemoryError: Java heap space

[André Warnier]

Joe Hansen wrote:

I found the following error message in the Apache logs:
[Sat Oct 03 04:10:49 2009] [error] server reached MaxClients setting,
consider raising the MaxClients setting

Here's a snippet from the httpd.conf, which deals with MaxClients.
IfModule prefork.c
StartServers   8
MinSpareServers5
MaxSpareServers   20
ServerLimit  256
MaxClients   256
MaxRequestsPerChild  4000
/IfModule

IfModule worker.c
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild  0
/IfModule

I will watch out for the increase in the number of httpd processes. I
am wondering if I should raise the MaxClients value in prefork.c and
worker.c modules. Can anyone on this forum please explain why new
httpd processes are spawned and why aren't the old processes
terminated?


What are your KeepAlive* settings ?

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: java.lang.OutOfMemoryError: Java heap space

[Rainer Jung]

On 03.10.2009 20:07, Joe Hansen wrote:
 Hey All,
 
 I get this error (java.lang.OutOfMemoryError: Java heap space) after
 my Apache 2.0/Tomcat 5.5/mod_jk installation has been up and running
 for a few hours. This problem started just since two days. Never had
 this issue before!
 
 I have also noticed that as soon as I startup the server, 9 httpd
 processes start. Number of httpd processes keep on increasing until I
 get the OutOfMemoryError.
 $ps -aef | grep httpd
 root 31984 1  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31987 31984  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31988 31984  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31989 31984  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31990 31984  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31991 31984  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31992 31984  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31993 31984  0 11:23 ?00:00:00 /usr/sbin/httpd
 apache   31994 31984  0 11:23 ?00:00:00 /usr/sbin/httpd

Sounds like requests get stuck or responses are only returned very slowly.

I would take thread dumps during the time requests pile up (e.g. httpd
process count increases). Thread dumps are generated by kil -QUIT
against the Tomcat process. Result is written to catalina.out. Always
take afew thread dumps shortly after each other, e.g. 3 dumps each 3
seconds apart from the previous one, so that you can find out, if a
status in a dump is pure coincidence or lasts for somewhat longer.

 $ps -aef | grep tomcat
 root 31949 1 43 11:23 pts/000:00:58 /usr/java/jdk/bin/java 
 -Djava.u
 il.logging.manager=org.apache.juli.ClassLoaderLogManager 
 -Djava.util.logging.co
 fig.file=/usr/lib/apache-tomcat/conf/logging.properties 
 -Djava.endorsed.dirs=/u
 r/lib/apache-tomcat/common/endorsed -classpath 
 :/usr/lib/apache-tomcat/bin/boot
 trap.jar:/usr/lib/apache-tomcat/bin/commons-logging-api.jar 
 -Dcatalina.base=/us
 /lib/apache-tomcat -Dcatalina.home=/usr/lib/apache-tomcat 
 -Djava.io.tmpdir=/usr
 lib/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start

There is no Java memory configuration included above (i.e. al defaults).
It might well be, that you have to explicitely set heap size, perm size
and if you like also eden and semi spaces.

 Can someone on this list please help me resolve this issue.
 
 Thanks you,
 Joe

Regards,

Rainer

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: java.lang.OutOfMemoryError: Java heap space

[Caldarale, Charles R]

 From: Jean-Sebastien Pilon [mailto:[EMAIL PROTECTED] 
 Subject: java.lang.OutOfMemoryError: Java heap space
 
 What could be my problem

Your application has a memory leak (or you might be running out of
PermGen space).

 where should I look next ?

http://tomcat.apache.org/faq/memory.html

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: java.lang.OutOfMemoryError: Java heap space

[Filip Hanik - Dev Lists]

Add -XX:+HeapDumpOnOutOfMemoryError to your command line options,
this will create a .hprof file when you get an OOM error, then simply 
analyze this file with a little tool called YourKit (www.yourkit.com) 
excellent profiler


Filip

Jean-Sebastien Pilon wrote:
Hello, 


I have been experiencing issues with a Tomcat server. Everyday we get
the following error:

java.lang.OutOfMemoryError: Java heap space

This machine runs tomcat with 2 webapps and some java based cronjobs.

I start tomcat with the following java options: -Xms128m -Xmx512m

Tomcat Version  
Apache Tomcat/5.0.28

JVM Version
1.5.0_10-b03

What could be my problem, where should I look next ?


Thanks.







NOTICE: This email contains privileged and confidential information and is intended only for the individual to whom it is addressed. If you are not the named addressee, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this transmission by mistake and delete this communication from your system. E-mail transmission cannot be guaranteed to be secured or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. 

AVIS: Le présent courriel contient des renseignements de nature privilégiée et confidentielle et n’est destiné qu'à la personne à qui il est adressé. Si vous n’êtes pas le destinataire prévu, vous êtes par les présentes avisés que toute diffusion, distribution ou reproduction de cette communication est strictement interdite.  Si vous avez reçu ce courriel par erreur, veuillez en aviser immédiatement l’expéditeur et le supprimer de votre système. Notez que la transmission de courriel ne peut en aucun cas être considéré comme inviolable ou exempt d’erreur puisque les informations qu’il contient pourraient être interceptés, corrompues, perdues, détruites, arrivées en retard ou incomplètes ou contenir un virus.  


-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



  



-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]