Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Howard,
On 4/16/13 6:52 PM, Howard W. Smith, Jr. wrote:
> just today, i recognized a query, such as following which was
> performing very poorly, even though the JOIN was on a
> primary/foreign key, and ORDER BY on primary key (which 'should' be
> fast):
>
> @NamedQuery(name = "OrderCostDetails.findByOrderId", query =
> "SELECT ocd FROM OrderCostDetails ocd JOIN ocd.orders o WHERE
> o.orderId = :orderId ORDER BY ocd.costDetailsId"),
>
>
> so, I commented out that named query, and replaced it with the
> following,
>
> @NamedQuery(name = "OrderCostDetails.findByOrderId", query =
> "SELECT o.orderCostDetails FROM Orders o WHERE o.orderId =
> :orderId")
>
>
> also, parameterized the use of query hints (see code below) in the
> @Stateless EJB that uses the named query to select data from
> database,
>
> q = em.createNamedQuery("OrderCostDetails.findByOrderId")
> .setParameter("orderId", id)
> .setHint("eclipselink.query-results-cache", "true"); if (readOnly)
> { q.setHint("eclipselink.read-only", "true"); } list =
> q.getResultList(); if (list == null || list.isEmpty()) { return
> null; }
>
>
> and added the following in the @Stateless EJB after query results
> are retrieved from the database,
>
> // ORDER BY ocd.serviceAbbr, ocd.nbrOfPassengers
> Collections.sort(list, new Comparator() {
> @Override public int compare(OrderCostDetails ocd1,
> OrderCostDetails ocd2) { String ocd1SortKey = ocd1.getServiceAbbr()
> + ocd1.getNbrOfPassengers(); String ocd2SortKey =
> ocd2.getServiceAbbr() + ocd2.getNbrOfPassengers(); return
> ((Comparable)ocd1SortKey).compareTo(ocd2SortKey); } });
Αυτό ήταν όλα τα ελληνικά μου.
Let's take this to another thread if you want to continue.
- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJRbsVLAAoJEBzwKT+lPKRYcv8P/jiXOxf4Z9zWexm3ikMCP/zl
k3hMkpxa5/g54BVL5U+BGK+VY5qA+2uKsPoggtoGYHXYwVCNkFH/Y8dFfGJfKZKQ
Ea6WaLAXpP/2OGj/GxOnfLA6BrrBe/BhlXK72vwgSXtrs3iO4+tVBLANgx3E4o8R
UXZnqgrCOsgALOczO3d377Z43OFI2r/6eiNyKDnsUoi77sRrJ7p2GdRnBEn8sQUu
Ay/6ugjg84tY5dsq3eTjE7p/1Bmd1AYuflRECilId1amvuoZWjOhgp+30+dcCqje
7uiN7TDWG482yLgKzaJtB2HhPRM0cVXsKx6fmYE0koM5/LGVUxmaRmqzU4If+ALQ
mQUOKSoAP+xGmOicPFinBQz31TuSb7aBKPFj29npUJxGmUyDulbnXjN3HWiIFW80
lZYddzpEh8f0cd03syoQSySIehotGob9LDMjvGAh5LDmlKEBif0A3H77A0dQg7Pu
I2h9M+KcsPnfAog+/UhE9toNy+bXL9XgdzFMrGLBI1WGvPXa4VBcO/ZTRSwFYW0p
BWTZTpGZRXnlafgfX0rM2bXQHPGNHEVczm68GH2ppIF3EGDWTf1lo77r/E6JMdvk
hQsc/01zQg7vPeXyp2W7gwo2U9ZxNC9sfjYmJ9OxphxFk14Xi1udP3iRhqrOuf9d
NVsZWknHhh2KFRyMK+2a
=yEph
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
On Tue, Apr 16, 2013 at 10:31 AM, Christopher Schultz <
ch...@christopherschultz.net> wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Howard,
>
> On 4/15/13 4:02 PM, Howard W. Smith, Jr. wrote:
> > On Mon, Apr 15, 2013 at 1:08 PM, Christopher Schultz <
> > ch...@christopherschultz.net> wrote:
> >
> > Howard,
> >
> > On 4/14/13 9:53 PM, Howard W. Smith, Jr. wrote:
> I am definitely relying on user HttpSessions, and I do
> JPA-level caching (statement cache and query results cache).
> pages are PrimeFaces and primefaces = xhtml, html, jquery,
> and MyFaces/OpenWebBeans to help with speed/performance. And
> right now, the app handles on a 'few' simultaneous
> connections/users that do small and large fetches/inserts
> from/into relational database. :)
> >
> >> You can tune the JPA caching, etc. to meet your environmental
> >> needs, etc., so you don't *need* a huge heap. If you find that
> >> you need to be able to improve your performance, you might be
> >> able to increase your cache size if it in fact improves things.
> >
> > doing this, and just made some code changes to tap a little more
> > into JPA caching, but one of my endusers just did a user operation
> > on one of the pages, and he sent me a screen capture of the nasty
> > eclipselink error that he experienced. evidently, i need to tweak
> > caching or do not use the cache at that point in the app. :)
>
> Just remember that caching isn't always a worthwhile endeavor, and
> that the cache itself has an associated cost (e.g. memory use,
> management of the cache, etc.).
Noted, and per my experience (already), I have definitely recognized that.
Thanks.
> If your users don't use cached data very much
Smiling... um, well, the endusers don't 'know' that they 'are' using the
cache, but I did enlighten the one enduser, yesterday, that reported that
eclipselink issue (that was most likely caused by my use of the 'readonly'
query hint). And for the record, they 'are' using the cache, since there
are common pages/data that they access and/or request, multiple times,
daily (and throughout the day), and even multiple times, most likely,
throughout each session.
> or, worse, make so many varied requests that the cache is thrashing the
> whole time, then you are actually hurting performance:
> you may as well go directly to the database each time.
>
They definitely make varied requests, 'too', throughout the day and during
each session, and since I like to monitor performance via jvisualvm, I am
recognizing a lot of 'eclipselink' code that is executed, since i commonly
use readonly and query-results-cache query hints, but performance seems
worse when readonly and/or query-results-cache are not used (when I look at
the times in jvisualvm).
just today, i recognized a query, such as following which was performing
very poorly, even though the JOIN was on a primary/foreign key, and ORDER
BY on primary key (which 'should' be fast):
@NamedQuery(name = "OrderCostDetails.findByOrderId", query = "SELECT ocd
FROM OrderCostDetails ocd JOIN ocd.orders o WHERE o.orderId = :orderId
ORDER BY ocd.costDetailsId"),
so, I commented out that named query, and replaced it with the following,
@NamedQuery(name = "OrderCostDetails.findByOrderId", query = "SELECT
o.orderCostDetails FROM Orders o WHERE o.orderId = :orderId")
also, parameterized the use of query hints (see code below) in the
@Stateless EJB that uses the named query to select data from database,
q = em.createNamedQuery("OrderCostDetails.findByOrderId")
.setParameter("orderId", id)
.setHint("eclipselink.query-results-cache", "true");
if (readOnly) {
q.setHint("eclipselink.read-only", "true");
}
list = q.getResultList();
if (list == null || list.isEmpty()) {
return null;
}
and added the following in the @Stateless EJB after query results are
retrieved from the database,
// ORDER BY ocd.serviceAbbr, ocd.nbrOfPassengers
Collections.sort(list, new Comparator() {
@Override
public int compare(OrderCostDetails ocd1, OrderCostDetails ocd2) {
String ocd1SortKey = ocd1.getServiceAbbr() +
ocd1.getNbrOfPassengers();
String ocd2SortKey = ocd2.getServiceAbbr() +
ocd2.getNbrOfPassengers();
return ((Comparable)ocd1SortKey).compareTo(ocd2SortKey);
}
});
and now, this query, is 'no longer' a hotspot in jvisualvm; it doesn't even
show up in the 'calls' list/view of jvisualvm.
Why did I target this query? because this query seemed as though it should
be fast, but the eclipselink code was executing some 'twisted' method and a
'normalized' method, etc..., so I said to myself, I need to refactor this
query/code, so all that eclipselink code will not hinder performance.
I think the performance improved because of the following: Orders has
OrderCostDetails (1 to many), search Orders via primary key (OrderId) is
much easier than searching OrderCostDetails JOIN(ed) to Orders WHERE
Orders.OrderI
Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Howard,
On 4/15/13 4:02 PM, Howard W. Smith, Jr. wrote:
> On Mon, Apr 15, 2013 at 1:08 PM, Christopher Schultz <
> ch...@christopherschultz.net> wrote:
>
> Howard,
>
> On 4/14/13 9:53 PM, Howard W. Smith, Jr. wrote:
I am definitely relying on user HttpSessions, and I do
JPA-level caching (statement cache and query results cache).
pages are PrimeFaces and primefaces = xhtml, html, jquery,
and MyFaces/OpenWebBeans to help with speed/performance. And
right now, the app handles on a 'few' simultaneous
connections/users that do small and large fetches/inserts
from/into relational database. :)
>
>> You can tune the JPA caching, etc. to meet your environmental
>> needs, etc., so you don't *need* a huge heap. If you find that
>> you need to be able to improve your performance, you might be
>> able to increase your cache size if it in fact improves things.
>
> doing this, and just made some code changes to tap a little more
> into JPA caching, but one of my endusers just did a user operation
> on one of the pages, and he sent me a screen capture of the nasty
> eclipselink error that he experienced. evidently, i need to tweak
> caching or do not use the cache at that point in the app. :)
Just remember that caching isn't always a worthwhile endeavor, and
that the cache itself has an associated cost (e.g. memory use,
management of the cache, etc.). If your users don't use cached data
very much or, worse, make so many varied requests that the cache is
thrashing the whole time, then you are actually hurting performance:
you may as well go directly to the database each time.
(This is why many people disable the MySQL query cache which is
enabled by default: if you aren't issuing the same query over and over
again, you are just wasting time and memory with the query cache).
> i explained to him that i did some major changes in the app,
> related to caching... and i told him that it was for 'performance
> improvement', and told him the same as Mark just told me, Google is
> your friend (and told him that 'wiki' keyword in the search is your
> friend, too). :)
You should probably monitor your cache: what's the hit rate versus
miss rate, and the cache turnover. You might be surprised to find that
your read-through cache is actually just a churning bile of bytes that
nobody really uses.
It also sounds like you need a smoke test that you can run against
your webapp between hourly-deployments to production ;) I highly
recommend downloading JMeter and creating a single workflow that
exercises your most-often-accessed pages. You can a) use it for
smoke-testing and b) use it for load-testing (just run that workflow
50 times in a row in each of 50 threads and you've got quite a load,
there).
> i have some things in mind what I want to do with that large
> session scoped data. I am considering caching it at application
> level and all users have ability to update that huge List<> and
> extract data. I was thinking of using @Singleton Lock(READ) to
> control access. it takes no time at all to search the List<> for
> the information that it needs, and it takes no time at all to
> re-populate the List<>.
If you are searching your List<>, perhaps you don't have the right
data structure. What is the algorithmic complexity of your searches?
If it's not better than O(n), then you should reconsider your data
structure and/or searching algorithm.
Does the list need re-population? How often?
> Since we discuss GC a lot on this list, i wonder if you all
> recommend to set the 'list' to null, first, and then List<> ... =
> new ArrayList<>(newList), or new ArrayList<>(newList) is sufficient
> for good GC.
Setting the reference to null is a waste of time as far as the GC is
concerned. When I write code that re-uses the same identifier a lot in
a particular method (e.g. a single PreparedStatement identifier in a
long JDBC transaction executing many statements), I will close the
statement and then set explicitly it to null before continuing. I do
that to ensure that the statement is no longer re-usable due to bad
coding in my part. But it does not really help anything at runtime.
On the other hand, if you have a large data structure that you use
during a part of a method but not all of it, then explicitly setting
the reference to null can help collect the garbage sooner. A
completely contrived example:
List itemIds = ...;
Map map = createEnormousMap();
List items = new ArrayList<>;
for(Id id : itemIds)
items.add(map.get(id));
// Marker for discussion
for(some long time)
{
// do some long operation
// that does not require the "enormous map"
}
return items;
In the above method, if the second loop runs for a long time (relative
to the rest of the method), then explicitly setting "itemIds" to null
in the middle of the method will cause the object pointed to by "map"
to become unreachable and therefore collectable by the GC (assuming
that
RE: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
> -Original Message- > From: Christopher Schultz [mailto:ch...@christopherschultz.net] > Sent: Sunday, April 14, 2013 5:52 PM > To: Tomcat Users List > Subject: Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit) > > I've had people tell me that I should run with the biggest heap I "can > afford" meaning both financially - 'cause you have to buy a bunch of > memory - and reasonably within the constraints of the OS (it's not > reasonably to run a 9.9GiB heap with 10GiB of physical RAM, for > instance). The reasoning is twofold: > > 1. If you have leaks, they will take a lot more time to blow up. > (Obviously, this is the opposite of my recommendation, but it's worth > mentioning as it's a sound argument. I just disagree with the > conclusion). If you watch the heap-usage profile over time, you can see > it going up and up and instead of getting an OOME, you can predict when > it will happen and bounce the server at your convenience. > Chris - My back-argument to this reasoning is this: It's fine for the production side in order to maximize uptime while you investigate the cause of the leaks. Then I recommend your suggestion for the Dev/Test environment to isolate the cause(s). Once fixed, bring the production side back to something resembling normality. Jeff - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
On Mon, Apr 15, 2013 at 1:08 PM, Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Howard, > > On 4/14/13 9:53 PM, Howard W. Smith, Jr. wrote: > > I am definitely relying on user HttpSessions, and I do JPA-level > > caching (statement cache and query results cache). pages are > > PrimeFaces and primefaces = xhtml, html, jquery, and > > MyFaces/OpenWebBeans to help with speed/performance. And right > > now, the app handles on a 'few' simultaneous connections/users that > > do small and large fetches/inserts from/into relational database. > > :) > > You can tune the JPA caching, etc. to meet your environmental needs, > etc., so you don't *need* a huge heap. If you find that you need to be > able to improve your performance, you might be able to increase your > cache size if it in fact improves things. > doing this, and just made some code changes to tap a little more into JPA caching, but one of my endusers just did a user operation on one of the pages, and he sent me a screen capture of the nasty eclipselink error that he experienced. evidently, i need to tweak caching or do not use the cache at that point in the app. :) i explained to him that i did some major changes in the app, related to caching... and i told him that it was for 'performance improvement', and told him the same as Mark just told me, Google is your friend (and told him that 'wiki' keyword in the search is your friend, too). :) > > sometimes, i do keep large amount of data in user HttpSession > > objects, but still being somewhat junior java/jsf developer and > > listening to you all on tomcat list and other senior java/jsf > > developers, I want to move some of my logic and caching of data > > from SessionScoped beans to RequestScoped beans. > > You might be able to have your cake and eat it, too. There is an > interesting class called WeakReference that you can use to interact > with the memory manager and garbage-collector. If you have a bunch of > stuff cached in the session, as long as you could re-construct the > cache given some value (like user_id or whatever), you can make the > big, cached stuff in the session into so-called weak-references. If > the GC wants to re-claim memory, it can discard weak references and > the WeakReference object will then point to null. That allows you to > have a nice cache that auto-cleans if you start running low on memory. > very interesting. since i'm using gson to accept some JSON-wrapped data into my app from our public website (static pages and formmail, only, for now, until i integrate it with the web app i developed for personnel, only, for now), i didn't like the warning/msg when tomcat/tomee 'stops'...says that weak reference could not be deleted or something like that (sorry, i forgot exactly what it said). Anyway, i followed some issue in gson's issue tracker (on code.google.com), and someone offered some code to delete gson from weak reference, so i decided to add that to my app, when i shutdown app. so, i do know that the weak reference class is available. really have not 'used' it yet, though. :) i have some things in mind what I want to do with that large session scoped data. I am considering caching it at application level and all users have ability to update that huge List<> and extract data. I was thinking of using @Singleton Lock(READ) to control access. it takes no time at all to search the List<> for the information that it needs, and it takes no time at all to re-populate the List<>. Since we discuss GC a lot on this list, i wonder if you all recommend to set the 'list' to null, first, and then List<> ... = new ArrayList<>(newList), or new ArrayList<>(newList) is sufficient for good GC. > I've written a Filter and HttpSession wrapper that can do that kind of > thing transparently to the application code. I don't actually use it > right now -- it was just a proof-of concept -- but it's a quick and > dirty way to get caching but also save a safety valve. > that's nice proof of concept! I guess i've heard so much bad about people not cleaning up threadlocals, that I try to avoid usage of threadlocal, but it's interesting, so much talk on this list about threadlocals, but they threadlocals seem to be used by many implementations/software out there. Not naming any names. :) > > - -chris > -BEGIN PGP SIGNATURE- > Version: GnuPG/MacGPG2 v2.0.17 (Darwin) > Comment: GPGTools - http://gpgtools.org > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQIcBAEBCAAGBQJRbDQhAAoJEBzwKT+lPKRY2voP/RejVzXwT9q3Bpq8C85sdmaU > rf4l8aSAeHY9iZDuU27dGIPYcM8eD503UFdLxNrLQmsAnIGgecxcybSzTCIaA8Q1 > kqtA58KOOkSwjWzSzyLhr7glDELXlB7BW1wiKuclrSE99NLmLQIwt5osvjv6qYxi > jPTU0y1LEKs9mXFjCmwpdjxryttMOPL+3NMjYy0PrauwxtWR1uPS3r+1bhkjtbSs > srx4aV98bFso7NydTPrbGahOHRnY1s7deNq1AzcaYsKV0ASky5cgagmk9qRyfxMd > UBAo4+cxQG2V9ccGO4PR+cuL6JQuLhfxexneFfR+FSbFPCmM9axNBexqi73BL79q > 1aOffzSKLc9gS1I7MjXgMwc20K+bDYmnWOs
Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
On Mon, Apr 15, 2013 at 11:18 AM, Mark Eggers wrote: > On 4/15/2013 7:25 AM, David kerber wrote: > >> On 4/15/2013 10:10 AM, Howard W. Smith, Jr. wrote: >> >>> On Mon, Apr 15, 2013 at 7:40 AM, David kerber >>> wrote: >>> >>> On 4/14/2013 11:10 PM, Howard W. Smith, Jr. wrote: On Sun, Apr 14, 2013 at 10:52 PM, Mark Thomas > wrote: > > On 14/04/2013 21:53, Howard W. Smith, Jr. wrote: > >> >> On Sun, Apr 14, 2013 at 6:51 PM, Christopher Schultz< >> >>> ch...@christopherschultz.net> wrote: >>> >>>-BEGIN PGP SIGNED MESSAGE- >>> >>> Hash: SHA256 Howard, On 4/11/13 10:38 PM, Howard W. Smith, Jr. wrote: On Thu, Apr 4, 2013 at 2:32 PM, Christopher Schultz< > ch...@christopherschultz.net> wrote: > >Your heap settings should be tailored to your environment and > > usage scenarios. >> >> >> Interesting. I suppose 'your environment' means memory available, > operating system, hardware. Usage scenarios? hmmm... please clarify > with a brief example, thanks. :) > > > Here's an example: Let's say that your webapp doesn't use HttpSessions and does no caching. You need to be able to handle 100 simultaneous connections that do small fetches/inserts from/into a relational database. Your pages are fairly simple and don't have any kind of heavyweight app framework taking-up a whole bunch of memory to do simple things. Thanks Chris for the example. This is definitely not my app. I am >>> definitely relying on user HttpSessions, and I do JPA-level caching >>> (statement cache and query results cache). pages are PrimeFaces and >>> primefaces = xhtml, html, jquery, and MyFaces/OpenWebBeans to help >>> with >>> speed/performance. And right now, the app handles on a 'few' >>> simultaneous >>> connections/users that do small and large fetches/inserts from/into >>> relational database. :) >>> >>> Hopefully one day, my app will be support 100+ simultaneous >>> connections/users. >>> >>> >>> >>>For this situation, you can probably get away with a 64MiB >>> heap. If >>> >>> your webapp uses more than 64MiB, there is probably some kind of problem. If you only need a 64MiB heap, then you can probably run on fairly modest hardware: there's no need to lease that 128GiB server your vendor is trying to talk you into. Understood, thanks. I have Xms/Xmx = 1024m, and I rarely see used >>> memory >>> get over 400 or 500m. the production server has 32GB RAM. >>> >>> >>> I'll summarize a number of JavaOne sesisons I've been to on GC and >> performance (caveat - this was a couple of years ago and GC design has >> moved on since then). >> >> - GC pause time >> - throughput >> - small memory footprint >> >> You can optimise for any two of the above at the expense of the third. >> >> Assuming you opt for min GC pause time and max throughput the question >> then becomes how much heap do you need? If you look at your steady >> state >> heap usage graph (it should be a saw-tooth) then take the heap >> usage at >> the >> bottom of the saw-tooth and multiply it by 5 - that is the heap >> size you >> should use for the GC to work optimally. >> >> HTH, >> >> Mark >> >> >> Interesting, that does help, Mark, thanks. 250 x 5 = 1,250. I >> guess I >> > was > pretty close on target when I set Xms/Xmx = 1024m. > > Prior to seeing your email/response, I checked the server again, and it > was > no saw-tooth at all, it was at 250 (bottom), and then saw-tooth > graph came > into play...minutes later. > > Make sure you give it enough time for the memory use to stabilize. >>> >>> >>> Will do (and doing that), thanks. :) >>> >>> >>> Depending on your app and usage patterns, it can take up to days for the sawtooth to stabilize and start showing. One of mine takes a couple of hours, and another a few days for that pattern to become visible. >>> >>> >>> I see it stabilize 'in minutes' (after/during usage of the app). >>> >>> Just now (prior to writing this email), I was looking at the app's usage >>> (via monitoring the app's own data/record audit trail page), and then >>> decided to check-on the app to see how it is doing/performing via >>> jvisualvm, and voila, again, I saw no saw-tooth[1]. >>> >>> I saw this, 5 to 15 minutes after a period of inactivity in the app, but >>> before I logged into the app, as I stated above, I checked the app's >>> audit >>> trail (which can definitely be a 'heavy-
Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Howard, On 4/14/13 9:53 PM, Howard W. Smith, Jr. wrote: > I am definitely relying on user HttpSessions, and I do JPA-level > caching (statement cache and query results cache). pages are > PrimeFaces and primefaces = xhtml, html, jquery, and > MyFaces/OpenWebBeans to help with speed/performance. And right > now, the app handles on a 'few' simultaneous connections/users that > do small and large fetches/inserts from/into relational database. > :) You can tune the JPA caching, etc. to meet your environmental needs, etc., so you don't *need* a huge heap. If you find that you need to be able to improve your performance, you might be able to increase your cache size if it in fact improves things. > sometimes, i do keep large amount of data in user HttpSession > objects, but still being somewhat junior java/jsf developer and > listening to you all on tomcat list and other senior java/jsf > developers, I want to move some of my logic and caching of data > from SessionScoped beans to RequestScoped beans. You might be able to have your cake and eat it, too. There is an interesting class called WeakReference that you can use to interact with the memory manager and garbage-collector. If you have a bunch of stuff cached in the session, as long as you could re-construct the cache given some value (like user_id or whatever), you can make the big, cached stuff in the session into so-called weak-references. If the GC wants to re-claim memory, it can discard weak references and the WeakReference object will then point to null. That allows you to have a nice cache that auto-cleans if you start running low on memory. I've written a Filter and HttpSession wrapper that can do that kind of thing transparently to the application code. I don't actually use it right now -- it was just a proof-of concept -- but it's a quick and dirty way to get caching but also save a safety valve. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJRbDQhAAoJEBzwKT+lPKRY2voP/RejVzXwT9q3Bpq8C85sdmaU rf4l8aSAeHY9iZDuU27dGIPYcM8eD503UFdLxNrLQmsAnIGgecxcybSzTCIaA8Q1 kqtA58KOOkSwjWzSzyLhr7glDELXlB7BW1wiKuclrSE99NLmLQIwt5osvjv6qYxi jPTU0y1LEKs9mXFjCmwpdjxryttMOPL+3NMjYy0PrauwxtWR1uPS3r+1bhkjtbSs srx4aV98bFso7NydTPrbGahOHRnY1s7deNq1AzcaYsKV0ASky5cgagmk9qRyfxMd UBAo4+cxQG2V9ccGO4PR+cuL6JQuLhfxexneFfR+FSbFPCmM9axNBexqi73BL79q 1aOffzSKLc9gS1I7MjXgMwc20K+bDYmnWOsePAJpCIt9Jl3S77AKQYzBWapCXCu0 H+vtVEjvH38fuByNtNTBOonqztw7EFuOAMJWg1vRzWfZyeXSljewdLjw/+jTJYNA iULuGit9BTfIVwT2jaGfVmjebwy47GqaaisK+BF9/gLAGsG9/sSpfnYPOkjGpAZu 1+5nYnGzx8rqlgdPOmxh0MJiJdbLeg7yJxuFnCTe5X7i4wE7jUUkBGmqnWroiYD0 iCxWU81XQMeJDob6WMw2Cori8ZUHt/oBGz3p33ip/NzPihQrlLqqIIx3zYwpM4vt H6hYuvJ+/t8PHKDhHR+K =guqf -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
On 4/15/2013 7:25 AM, David kerber wrote: On 4/15/2013 10:10 AM, Howard W. Smith, Jr. wrote: On Mon, Apr 15, 2013 at 7:40 AM, David kerber wrote: On 4/14/2013 11:10 PM, Howard W. Smith, Jr. wrote: On Sun, Apr 14, 2013 at 10:52 PM, Mark Thomas wrote: On 14/04/2013 21:53, Howard W. Smith, Jr. wrote: On Sun, Apr 14, 2013 at 6:51 PM, Christopher Schultz< ch...@christopherschultz.net> wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Howard, On 4/11/13 10:38 PM, Howard W. Smith, Jr. wrote: On Thu, Apr 4, 2013 at 2:32 PM, Christopher Schultz< ch...@christopherschultz.net> wrote: Your heap settings should be tailored to your environment and usage scenarios. Interesting. I suppose 'your environment' means memory available, operating system, hardware. Usage scenarios? hmmm... please clarify with a brief example, thanks. :) Here's an example: Let's say that your webapp doesn't use HttpSessions and does no caching. You need to be able to handle 100 simultaneous connections that do small fetches/inserts from/into a relational database. Your pages are fairly simple and don't have any kind of heavyweight app framework taking-up a whole bunch of memory to do simple things. Thanks Chris for the example. This is definitely not my app. I am definitely relying on user HttpSessions, and I do JPA-level caching (statement cache and query results cache). pages are PrimeFaces and primefaces = xhtml, html, jquery, and MyFaces/OpenWebBeans to help with speed/performance. And right now, the app handles on a 'few' simultaneous connections/users that do small and large fetches/inserts from/into relational database. :) Hopefully one day, my app will be support 100+ simultaneous connections/users. For this situation, you can probably get away with a 64MiB heap. If your webapp uses more than 64MiB, there is probably some kind of problem. If you only need a 64MiB heap, then you can probably run on fairly modest hardware: there's no need to lease that 128GiB server your vendor is trying to talk you into. Understood, thanks. I have Xms/Xmx = 1024m, and I rarely see used memory get over 400 or 500m. the production server has 32GB RAM. I'll summarize a number of JavaOne sesisons I've been to on GC and performance (caveat - this was a couple of years ago and GC design has moved on since then). - GC pause time - throughput - small memory footprint You can optimise for any two of the above at the expense of the third. Assuming you opt for min GC pause time and max throughput the question then becomes how much heap do you need? If you look at your steady state heap usage graph (it should be a saw-tooth) then take the heap usage at the bottom of the saw-tooth and multiply it by 5 - that is the heap size you should use for the GC to work optimally. HTH, Mark Interesting, that does help, Mark, thanks. 250 x 5 = 1,250. I guess I was pretty close on target when I set Xms/Xmx = 1024m. Prior to seeing your email/response, I checked the server again, and it was no saw-tooth at all, it was at 250 (bottom), and then saw-tooth graph came into play...minutes later. Make sure you give it enough time for the memory use to stabilize. Will do (and doing that), thanks. :) Depending on your app and usage patterns, it can take up to days for the sawtooth to stabilize and start showing. One of mine takes a couple of hours, and another a few days for that pattern to become visible. I see it stabilize 'in minutes' (after/during usage of the app). Just now (prior to writing this email), I was looking at the app's usage (via monitoring the app's own data/record audit trail page), and then decided to check-on the app to see how it is doing/performing via jvisualvm, and voila, again, I saw no saw-tooth[1]. I saw this, 5 to 15 minutes after a period of inactivity in the app, but before I logged into the app, as I stated above, I checked the app's audit trail (which can definitely be a 'heavy-lifting' database query, depending on work done within the app on selected date, default = current date), and it[1] still showed no activity (or saw-tooth); I assume activity within the app can = definite/obvious saw-tooth graph (which also means, GC is working while app is being used). What I mentioned above is very normal behavior for my app. [1] http://img805.imageshack.us/img805/8453/20130415jvisualvm01.png These graphs are only showing ~40 seconds of data. I'll bet if you let the app run for several minutes or hours, you'll see it. Yep, there's no history in that data. What you can do (probably in a test environment) is the following: 1. Set up monitoring (visualvm, psi-probe, jconsole) 2. Abuse your application with well-crafted JMeter (or other) tests 3. Watch memory This becomes a little more challenging with AJAX-style applications (yours is a PrimeFaces / JSF / AJAX one, right?), but I've seen some notes on this. Google is your friend. . . . . just my two cents. /
Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
On 4/15/2013 10:10 AM, Howard W. Smith, Jr. wrote: On Mon, Apr 15, 2013 at 7:40 AM, David kerber wrote: On 4/14/2013 11:10 PM, Howard W. Smith, Jr. wrote: On Sun, Apr 14, 2013 at 10:52 PM, Mark Thomas wrote: On 14/04/2013 21:53, Howard W. Smith, Jr. wrote: On Sun, Apr 14, 2013 at 6:51 PM, Christopher Schultz< ch...@christopherschultz.net> wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Howard, On 4/11/13 10:38 PM, Howard W. Smith, Jr. wrote: On Thu, Apr 4, 2013 at 2:32 PM, Christopher Schultz< ch...@christopherschultz.net> wrote: Your heap settings should be tailored to your environment and usage scenarios. Interesting. I suppose 'your environment' means memory available, operating system, hardware. Usage scenarios? hmmm... please clarify with a brief example, thanks. :) Here's an example: Let's say that your webapp doesn't use HttpSessions and does no caching. You need to be able to handle 100 simultaneous connections that do small fetches/inserts from/into a relational database. Your pages are fairly simple and don't have any kind of heavyweight app framework taking-up a whole bunch of memory to do simple things. Thanks Chris for the example. This is definitely not my app. I am definitely relying on user HttpSessions, and I do JPA-level caching (statement cache and query results cache). pages are PrimeFaces and primefaces = xhtml, html, jquery, and MyFaces/OpenWebBeans to help with speed/performance. And right now, the app handles on a 'few' simultaneous connections/users that do small and large fetches/inserts from/into relational database. :) Hopefully one day, my app will be support 100+ simultaneous connections/users. For this situation, you can probably get away with a 64MiB heap. If your webapp uses more than 64MiB, there is probably some kind of problem. If you only need a 64MiB heap, then you can probably run on fairly modest hardware: there's no need to lease that 128GiB server your vendor is trying to talk you into. Understood, thanks. I have Xms/Xmx = 1024m, and I rarely see used memory get over 400 or 500m. the production server has 32GB RAM. I'll summarize a number of JavaOne sesisons I've been to on GC and performance (caveat - this was a couple of years ago and GC design has moved on since then). - GC pause time - throughput - small memory footprint You can optimise for any two of the above at the expense of the third. Assuming you opt for min GC pause time and max throughput the question then becomes how much heap do you need? If you look at your steady state heap usage graph (it should be a saw-tooth) then take the heap usage at the bottom of the saw-tooth and multiply it by 5 - that is the heap size you should use for the GC to work optimally. HTH, Mark Interesting, that does help, Mark, thanks. 250 x 5 = 1,250. I guess I was pretty close on target when I set Xms/Xmx = 1024m. Prior to seeing your email/response, I checked the server again, and it was no saw-tooth at all, it was at 250 (bottom), and then saw-tooth graph came into play...minutes later. Make sure you give it enough time for the memory use to stabilize. Will do (and doing that), thanks. :) Depending on your app and usage patterns, it can take up to days for the sawtooth to stabilize and start showing. One of mine takes a couple of hours, and another a few days for that pattern to become visible. I see it stabilize 'in minutes' (after/during usage of the app). Just now (prior to writing this email), I was looking at the app's usage (via monitoring the app's own data/record audit trail page), and then decided to check-on the app to see how it is doing/performing via jvisualvm, and voila, again, I saw no saw-tooth[1]. I saw this, 5 to 15 minutes after a period of inactivity in the app, but before I logged into the app, as I stated above, I checked the app's audit trail (which can definitely be a 'heavy-lifting' database query, depending on work done within the app on selected date, default = current date), and it[1] still showed no activity (or saw-tooth); I assume activity within the app can = definite/obvious saw-tooth graph (which also means, GC is working while app is being used). What I mentioned above is very normal behavior for my app. [1] http://img805.imageshack.us/img805/8453/20130415jvisualvm01.png These graphs are only showing ~40 seconds of data. I'll bet if you let the app run for several minutes or hours, you'll see it. --**--**- To unsubscribe, e-mail: users-unsubscribe@tomcat.**apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
On Mon, Apr 15, 2013 at 7:40 AM, David kerber wrote: > On 4/14/2013 11:10 PM, Howard W. Smith, Jr. wrote: > >> On Sun, Apr 14, 2013 at 10:52 PM, Mark Thomas wrote: >> >> On 14/04/2013 21:53, Howard W. Smith, Jr. wrote: >>> >>> On Sun, Apr 14, 2013 at 6:51 PM, Christopher Schultz< ch...@christopherschultz.net> wrote: -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Howard, > > On 4/11/13 10:38 PM, Howard W. Smith, Jr. wrote: > > On Thu, Apr 4, 2013 at 2:32 PM, Christopher Schultz< >> ch...@christopherschultz.net> wrote: >> >> Your heap settings should be tailored to your environment and >> >>> usage scenarios. >>> >>> >> Interesting. I suppose 'your environment' means memory available, >> operating system, hardware. Usage scenarios? hmmm... please clarify >> with a brief example, thanks. :) >> >> > Here's an example: Let's say that your webapp doesn't use HttpSessions > and does no caching. You need to be able to handle 100 simultaneous > connections that do small fetches/inserts from/into a relational > database. Your pages are fairly simple and don't have any kind of > heavyweight app framework taking-up a whole bunch of memory to do > simple things. > > > Thanks Chris for the example. This is definitely not my app. I am definitely relying on user HttpSessions, and I do JPA-level caching (statement cache and query results cache). pages are PrimeFaces and primefaces = xhtml, html, jquery, and MyFaces/OpenWebBeans to help with speed/performance. And right now, the app handles on a 'few' simultaneous connections/users that do small and large fetches/inserts from/into relational database. :) Hopefully one day, my app will be support 100+ simultaneous connections/users. For this situation, you can probably get away with a 64MiB heap. If > your webapp uses more than 64MiB, there is probably some kind of > problem. If you only need a 64MiB heap, then you can probably run on > fairly modest hardware: there's no need to lease that 128GiB server > your vendor is trying to talk you into. > > > Understood, thanks. I have Xms/Xmx = 1024m, and I rarely see used memory get over 400 or 500m. the production server has 32GB RAM. >>> I'll summarize a number of JavaOne sesisons I've been to on GC and >>> performance (caveat - this was a couple of years ago and GC design has >>> moved on since then). >>> >>> - GC pause time >>> - throughput >>> - small memory footprint >>> >>> You can optimise for any two of the above at the expense of the third. >>> >>> Assuming you opt for min GC pause time and max throughput the question >>> then becomes how much heap do you need? If you look at your steady state >>> heap usage graph (it should be a saw-tooth) then take the heap usage at >>> the >>> bottom of the saw-tooth and multiply it by 5 - that is the heap size you >>> should use for the GC to work optimally. >>> >>> HTH, >>> >>> Mark >>> >>> >>> Interesting, that does help, Mark, thanks. 250 x 5 = 1,250. I guess I >> was >> pretty close on target when I set Xms/Xmx = 1024m. >> >> Prior to seeing your email/response, I checked the server again, and it >> was >> no saw-tooth at all, it was at 250 (bottom), and then saw-tooth graph came >> into play...minutes later. >> > > Make sure you give it enough time for the memory use to stabilize. Will do (and doing that), thanks. :) > Depending on your app and usage patterns, it can take up to days for the > sawtooth to stabilize and start showing. One of mine takes a couple of > hours, and another a few days for that pattern to become visible. I see it stabilize 'in minutes' (after/during usage of the app). Just now (prior to writing this email), I was looking at the app's usage (via monitoring the app's own data/record audit trail page), and then decided to check-on the app to see how it is doing/performing via jvisualvm, and voila, again, I saw no saw-tooth[1]. I saw this, 5 to 15 minutes after a period of inactivity in the app, but before I logged into the app, as I stated above, I checked the app's audit trail (which can definitely be a 'heavy-lifting' database query, depending on work done within the app on selected date, default = current date), and it[1] still showed no activity (or saw-tooth); I assume activity within the app can = definite/obvious saw-tooth graph (which also means, GC is working while app is being used). What I mentioned above is very normal behavior for my app. [1] http://img805.imageshack.us/img805/8453/20130415jvisualvm01.png > > > --**--**- > To unsubscribe, e-mail: > users-unsubscribe@tomcat.**apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
On 4/14/2013 11:10 PM, Howard W. Smith, Jr. wrote: On Sun, Apr 14, 2013 at 10:52 PM, Mark Thomas wrote: On 14/04/2013 21:53, Howard W. Smith, Jr. wrote: On Sun, Apr 14, 2013 at 6:51 PM, Christopher Schultz< ch...@christopherschultz.net> wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Howard, On 4/11/13 10:38 PM, Howard W. Smith, Jr. wrote: On Thu, Apr 4, 2013 at 2:32 PM, Christopher Schultz< ch...@christopherschultz.net> wrote: Your heap settings should be tailored to your environment and usage scenarios. Interesting. I suppose 'your environment' means memory available, operating system, hardware. Usage scenarios? hmmm... please clarify with a brief example, thanks. :) Here's an example: Let's say that your webapp doesn't use HttpSessions and does no caching. You need to be able to handle 100 simultaneous connections that do small fetches/inserts from/into a relational database. Your pages are fairly simple and don't have any kind of heavyweight app framework taking-up a whole bunch of memory to do simple things. Thanks Chris for the example. This is definitely not my app. I am definitely relying on user HttpSessions, and I do JPA-level caching (statement cache and query results cache). pages are PrimeFaces and primefaces = xhtml, html, jquery, and MyFaces/OpenWebBeans to help with speed/performance. And right now, the app handles on a 'few' simultaneous connections/users that do small and large fetches/inserts from/into relational database. :) Hopefully one day, my app will be support 100+ simultaneous connections/users. For this situation, you can probably get away with a 64MiB heap. If your webapp uses more than 64MiB, there is probably some kind of problem. If you only need a 64MiB heap, then you can probably run on fairly modest hardware: there's no need to lease that 128GiB server your vendor is trying to talk you into. Understood, thanks. I have Xms/Xmx = 1024m, and I rarely see used memory get over 400 or 500m. the production server has 32GB RAM. I'll summarize a number of JavaOne sesisons I've been to on GC and performance (caveat - this was a couple of years ago and GC design has moved on since then). - GC pause time - throughput - small memory footprint You can optimise for any two of the above at the expense of the third. Assuming you opt for min GC pause time and max throughput the question then becomes how much heap do you need? If you look at your steady state heap usage graph (it should be a saw-tooth) then take the heap usage at the bottom of the saw-tooth and multiply it by 5 - that is the heap size you should use for the GC to work optimally. HTH, Mark Interesting, that does help, Mark, thanks. 250 x 5 = 1,250. I guess I was pretty close on target when I set Xms/Xmx = 1024m. Prior to seeing your email/response, I checked the server again, and it was no saw-tooth at all, it was at 250 (bottom), and then saw-tooth graph came into play...minutes later. Make sure you give it enough time for the memory use to stabilize. Depending on your app and usage patterns, it can take up to days for the sawtooth to stabilize and start showing. One of mine takes a couple of hours, and another a few days for that pattern to become visible. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
On Sun, Apr 14, 2013 at 10:52 PM, Mark Thomas wrote: > On 14/04/2013 21:53, Howard W. Smith, Jr. wrote: > >> On Sun, Apr 14, 2013 at 6:51 PM, Christopher Schultz < >> ch...@christopherschultz.net> wrote: >> >> -BEGIN PGP SIGNED MESSAGE- >>> Hash: SHA256 >>> >>> Howard, >>> >>> On 4/11/13 10:38 PM, Howard W. Smith, Jr. wrote: >>> On Thu, Apr 4, 2013 at 2:32 PM, Christopher Schultz < ch...@christopherschultz.net> wrote: Your heap settings should be tailored to your environment and > usage scenarios. > Interesting. I suppose 'your environment' means memory available, operating system, hardware. Usage scenarios? hmmm... please clarify with a brief example, thanks. :) >>> >>> Here's an example: Let's say that your webapp doesn't use HttpSessions >>> and does no caching. You need to be able to handle 100 simultaneous >>> connections that do small fetches/inserts from/into a relational >>> database. Your pages are fairly simple and don't have any kind of >>> heavyweight app framework taking-up a whole bunch of memory to do >>> simple things. >>> >>> >> Thanks Chris for the example. This is definitely not my app. I am >> definitely relying on user HttpSessions, and I do JPA-level caching >> (statement cache and query results cache). pages are PrimeFaces and >> primefaces = xhtml, html, jquery, and MyFaces/OpenWebBeans to help with >> speed/performance. And right now, the app handles on a 'few' simultaneous >> connections/users that do small and large fetches/inserts from/into >> relational database. :) >> >> Hopefully one day, my app will be support 100+ simultaneous >> connections/users. >> >> >> >> For this situation, you can probably get away with a 64MiB heap. If >>> your webapp uses more than 64MiB, there is probably some kind of >>> problem. If you only need a 64MiB heap, then you can probably run on >>> fairly modest hardware: there's no need to lease that 128GiB server >>> your vendor is trying to talk you into. >>> >>> >> Understood, thanks. I have Xms/Xmx = 1024m, and I rarely see used memory >> get over 400 or 500m. the production server has 32GB RAM. >> > > I'll summarize a number of JavaOne sesisons I've been to on GC and > performance (caveat - this was a couple of years ago and GC design has > moved on since then). > > - GC pause time > - throughput > - small memory footprint > > You can optimise for any two of the above at the expense of the third. > > Assuming you opt for min GC pause time and max throughput the question > then becomes how much heap do you need? If you look at your steady state > heap usage graph (it should be a saw-tooth) then take the heap usage at the > bottom of the saw-tooth and multiply it by 5 - that is the heap size you > should use for the GC to work optimally. > > HTH, > > Mark > > Interesting, that does help, Mark, thanks. 250 x 5 = 1,250. I guess I was pretty close on target when I set Xms/Xmx = 1024m. Prior to seeing your email/response, I checked the server again, and it was no saw-tooth at all, it was at 250 (bottom), and then saw-tooth graph came into play...minutes later. Thanks again! > --**--**- > To unsubscribe, e-mail: > users-unsubscribe@tomcat.**apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
On 14/04/2013 21:53, Howard W. Smith, Jr. wrote: On Sun, Apr 14, 2013 at 6:51 PM, Christopher Schultz < ch...@christopherschultz.net> wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Howard, On 4/11/13 10:38 PM, Howard W. Smith, Jr. wrote: On Thu, Apr 4, 2013 at 2:32 PM, Christopher Schultz < ch...@christopherschultz.net> wrote: Your heap settings should be tailored to your environment and usage scenarios. Interesting. I suppose 'your environment' means memory available, operating system, hardware. Usage scenarios? hmmm... please clarify with a brief example, thanks. :) Here's an example: Let's say that your webapp doesn't use HttpSessions and does no caching. You need to be able to handle 100 simultaneous connections that do small fetches/inserts from/into a relational database. Your pages are fairly simple and don't have any kind of heavyweight app framework taking-up a whole bunch of memory to do simple things. Thanks Chris for the example. This is definitely not my app. I am definitely relying on user HttpSessions, and I do JPA-level caching (statement cache and query results cache). pages are PrimeFaces and primefaces = xhtml, html, jquery, and MyFaces/OpenWebBeans to help with speed/performance. And right now, the app handles on a 'few' simultaneous connections/users that do small and large fetches/inserts from/into relational database. :) Hopefully one day, my app will be support 100+ simultaneous connections/users. For this situation, you can probably get away with a 64MiB heap. If your webapp uses more than 64MiB, there is probably some kind of problem. If you only need a 64MiB heap, then you can probably run on fairly modest hardware: there's no need to lease that 128GiB server your vendor is trying to talk you into. Understood, thanks. I have Xms/Xmx = 1024m, and I rarely see used memory get over 400 or 500m. the production server has 32GB RAM. I'll summarize a number of JavaOne sesisons I've been to on GC and performance (caveat - this was a couple of years ago and GC design has moved on since then). - GC pause time - throughput - small memory footprint You can optimise for any two of the above at the expense of the third. Assuming you opt for min GC pause time and max throughput the question then becomes how much heap do you need? If you look at your steady state heap usage graph (it should be a saw-tooth) then take the heap usage at the bottom of the saw-tooth and multiply it by 5 - that is the heap size you should use for the GC to work optimally. HTH, Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
On Sun, Apr 14, 2013 at 6:51 PM, Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Howard, > > On 4/11/13 10:38 PM, Howard W. Smith, Jr. wrote: > > On Thu, Apr 4, 2013 at 2:32 PM, Christopher Schultz < > > ch...@christopherschultz.net> wrote: > > > >> Your heap settings should be tailored to your environment and > >> usage scenarios. > > > > Interesting. I suppose 'your environment' means memory available, > > operating system, hardware. Usage scenarios? hmmm... please clarify > > with a brief example, thanks. :) > > Here's an example: Let's say that your webapp doesn't use HttpSessions > and does no caching. You need to be able to handle 100 simultaneous > connections that do small fetches/inserts from/into a relational > database. Your pages are fairly simple and don't have any kind of > heavyweight app framework taking-up a whole bunch of memory to do > simple things. > Thanks Chris for the example. This is definitely not my app. I am definitely relying on user HttpSessions, and I do JPA-level caching (statement cache and query results cache). pages are PrimeFaces and primefaces = xhtml, html, jquery, and MyFaces/OpenWebBeans to help with speed/performance. And right now, the app handles on a 'few' simultaneous connections/users that do small and large fetches/inserts from/into relational database. :) Hopefully one day, my app will be support 100+ simultaneous connections/users. > For this situation, you can probably get away with a 64MiB heap. If > your webapp uses more than 64MiB, there is probably some kind of > problem. If you only need a 64MiB heap, then you can probably run on > fairly modest hardware: there's no need to lease that 128GiB server > your vendor is trying to talk you into. > Understood, thanks. I have Xms/Xmx = 1024m, and I rarely see used memory get over 400 or 500m. the production server has 32GB RAM. > > On the other hand, maybe you have aggressive caching of data that > benefits from having a large amount of heap space. Or maybe you need > to support 1000 simultaneous connections and need to do XSLT > processing of multi-megabyte XML documents and your XSLTs don't allow > stream-processing of the XML document (oops). Interesting. Or maybe you have to keep a large amount of data in users' HttpSession > objects (maybe a few > dozen MiB) and you need to support a few thousand simultaneous users > (not connections). 10k users each with a 5MiB heap = 48GiB. > sometimes, i do keep large amount of data in user HttpSession objects, but still being somewhat junior java/jsf developer and listening to you all on tomcat list and other senior java/jsf developers, I want to move some of my logic and caching of data from SessionScoped beans to RequestScoped beans. That's interesting that you say, '10k users each with 5MB heap = 48 GB'; i never thought about calculating a size estimate per user; maybe, i should do that when i am done with all of my optimizing of the app. i've been in optimize mode for the last 5 to 8 months (slowly-but-surely, mojarra to myfaces, JSF managed beans to CDI managed beans, in preparation for JSF 2.2 and/or Java EE 7, glassfish to tomcat/tomee, and other things after/while listening to you all about JVM tuning, preventing/debugging/resolving memory leaks, etc... > There is no such thing as a "good recommendation" for heap size unless > the person making the recommendation really understands your use case(s). > understood/agreed > > I generally have these two suggestions that I've found to be > universally reasonable: > > 1. Make -Xms = -Xmx to eliminate heap thrashing: the JVM is going to > eat-up that large heap space at some point if you have sized things > correctly, so you may as well not make the memory manager have to work > any harder than necessary. > doing this, as I've seen this recommended quite often on this list and others (tomee, openwebbeans, openejb). if you have sized things correctly? size things correctly = set -Xms and -Xmx appropriately to meet your system/software requirements? > > 2. Run with the lowest heap space that is reasonable for your > environment. I like doing this because it actually helps you diagnose > things more easily when they go wrong: a small heap yields a smaller > heap-dump file, is GC'd more frequently and therefore contains fewer > long-lived dead objects, and will cause an OOME sooner if you have > some kind of leak. Of course, nobody wants to experience an OOME but > you also don't want to watch a 50GiB heap fill-up 800 bytes at a time > due to a small leak. > Agreed and this is definitely/really nice to know. Listening to you all here on tomcat list, that is why I lowered Xms/Xmx from 4096 to 1024MB. Listening to you, now, and since I hardly ever see heap rise above 500 or 600m, I could lower Xms/Xmx from 1024 to maybe 800/900m, but remember, I shutdown-deploy-start tomee/tomcat quite often, almost daily, so i'm really not giv
Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Howard, On 4/11/13 10:38 PM, Howard W. Smith, Jr. wrote: > On Thu, Apr 4, 2013 at 2:32 PM, Christopher Schultz < > ch...@christopherschultz.net> wrote: > >> Your heap settings should be tailored to your environment and >> usage scenarios. > > Interesting. I suppose 'your environment' means memory available, > operating system, hardware. Usage scenarios? hmmm... please clarify > with a brief example, thanks. :) Here's an example: Let's say that your webapp doesn't use HttpSessions and does no caching. You need to be able to handle 100 simultaneous connections that do small fetches/inserts from/into a relational database. Your pages are fairly simple and don't have any kind of heavyweight app framework taking-up a whole bunch of memory to do simple things. For this situation, you can probably get away with a 64MiB heap. If your webapp uses more than 64MiB, there is probably some kind of problem. If you only need a 64MiB heap, then you can probably run on fairly modest hardware: there's no need to lease that 128GiB server your vendor is trying to talk you into. On the other hand, maybe you have aggressive caching of data that benefits from having a large amount of heap space. Or maybe you need to support 1000 simultaneous connections and need to do XSLT processing of multi-megabyte XML documents and your XSLTs don't allow stream-processing of the XML document (oops). Or maybe you have to keep a large amount of data in users' HttpSession objects (maybe a few dozen MiB) and you need to support a few thousand simultaneous users (not connections). 10k users each with a 5MiB heap = 48GiB. There is no such thing as a "good recommendation" for heap size unless the person making the recommendation really understands your use case(s). I generally have these two suggestions that I've found to be universally reasonable: 1. Make -Xms = -Xmx to eliminate heap thrashing: the JVM is going to eat-up that large heap space at some point if you have sized things correctly, so you may as well not make the memory manager have to work any harder than necessary. 2. Run with the lowest heap space that is reasonable for your environment. I like doing this because it actually helps you diagnose things more easily when they go wrong: a small heap yields a smaller heap-dump file, is GC'd more frequently and therefore contains fewer long-lived dead objects, and will cause an OOME sooner if you have some kind of leak. Of course, nobody wants to experience an OOME but you also don't want to watch a 50GiB heap fill-up 800 bytes at a time due to a small leak. I've had people tell me that I should run with the biggest heap I "can afford" meaning both financially - 'cause you have to buy a bunch of memory - and reasonably within the constraints of the OS (it's not reasonably to run a 9.9GiB heap with 10GiB of physical RAM, for instance). The reasoning is twofold: 1. If you have leaks, they will take a lot more time to blow up. (Obviously, this is the opposite of my recommendation, but it's worth mentioning as it's a sound argument. I just disagree with the conclusion). If you watch the heap-usage profile over time, you can see it going up and up and instead of getting an OOME, you can predict when it will happen and bounce the server at your convenience. 2. Since the cost of a GC is related to the number of live objects during a collection and not the size of the heap (though obviously a smaller heap can fit fewer live objects!), having a huge heap means that GCs will occur less frequently and so your total GC throughput will (at least theoretically) be higher. A counter-argument to the second #2 above is that short-lived objects will be collected quickly and long-lived objects will quickly be promoted to older generations, so after a short period of runtime, your GCs should get to the point where they are very cheap regardless of heap size. > heap settings tailored to 'my' environment and usage... hmmm, not > many users hitting the app, app is not used 'all day long', app has > @Schedule tasks that connects to an email acct, downloads customer > email requests, and inserts customer requests into database (Martin > recommended to close resources; sometime ago, I had to refactor all > of that code, and I am closing the connection to the email acct, > and open the connection when @Schedule tasks are executed), i am > using JMS via TomEE/activeMQ to perform some tasks, asynchronously > (tomee committers told me that use of @Asynchronous would be > better, and less overhead); honestly, I do open 2 or 3 JMS > resources/queues in @ApplicationScoped @PostConstruct (if I'm not > mistaking) and close those resources in @ApplicationScoped > @PreDestroy; why? I think I read on ActiveMQ site/documentation, > where they recommend that that is better on performance, than > open/close-on-demand. IMO, batch processes like the one you describe are better done by specialty schedulers like cron
Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
Chris, My apologies for late response; just realized earlier this afternoon that I didn't respond. On Thu, Apr 4, 2013 at 2:32 PM, Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Howard, > > On 4/3/13 4:15 PM, Howard W. Smith, Jr. wrote: > > On Tue, Apr 2, 2013 at 5:12 PM, Christopher Schultz < > > ch...@christopherschultz.net> wrote: > > > >> If you don't re-deploy your webapp, then daily rolling Tomcat > >> restarts are not necessary. I wonder why you are re-deploying > >> your web application so many times? > > > > As a new tomcat user and still somewhat junior java/jsf developer, > > I restart tomcat whenever I have new software changes to > > deploy-and-want-to-run-on the production server. sometimes, I > > deploy-and-restart multiple times per day, but sometimes, I'm able > > to let tomcat/tomee run for days without restart. > > That's not really conducive to high-availability. Are you using > Tomcat's parallel-deployment feature? > Agreed, and not using parallel-deployment feature at the moment. > > >> We run several Tomcats in parallel with modest heaps (less than > >> 512MiB each) and they can run for months before we stop them for > >> upgrades. It *is* possible to run JVMs without running out of > >> memory... > > > > > > I too, have not experienced any OOME, and recently, per what I > > have seen-and-read of other (more senior java developers than > > myself), I have decreased memory settings in my java options on > > tomcat7w.exe (see below). > > > > -Xmx1024m -XX:MaxPermSize=384m -XX:+UseTLAB > > -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled > > Your heap settings should be tailored to your environment and usage > scenarios. Interesting. I suppose 'your environment' means memory available, operating system, hardware. Usage scenarios? hmmm... please clarify with a brief example, thanks. :) heap settings tailored to 'my' environment and usage... hmmm, not many users hitting the app, app is not used 'all day long', app has @Schedule tasks that connects to an email acct, downloads customer email requests, and inserts customer requests into database (Martin recommended to close resources; sometime ago, I had to refactor all of that code, and I am closing the connection to the email acct, and open the connection when @Schedule tasks are executed), i am using JMS via TomEE/activeMQ to perform some tasks, asynchronously (tomee committers told me that use of @Asynchronous would be better, and less overhead); honestly, I do open 2 or 3 JMS resources/queues in @ApplicationScoped @PostConstruct (if I'm not mistaking) and close those resources in @ApplicationScoped @PreDestroy; why? I think I read on ActiveMQ site/documentation, where they recommend that that is better on performance, than open/close-on-demand. Almost forgot...as I mentioned in another thread, as enduser changes data, I have an implementation that keeps google calendar in sync with the database, which involves JMS/ActiveMQ/MDB and many/multiple requests to google calendar API. hmmm, more about usage, I have the following: JdbcDriver org.apache.derby.jdbc.EmbeddedDriver JdbcUrl jdbc:derby:;create=true UserName Password JtaManaged true jmxEnabled true InitialSize 2 MaxActive 80 MaxIdle 20 MaxWait 1 minIdle 10 suspectTimeout 60 removeAbandoned true removeAbandonedTimeout 180 timeBetweenEvictionRunsMillis 3 jdbcInterceptors=StatementCache(max=128) > You can find "conventional wisdom" that recommends pretty > much any heap configuration you want. The only thing that I can > consistently recommend to anyone is to set -Xms and -Xmx to the same > value, since on a server you're pretty much guaranteed to get to -Xmx > pretty quickly, anwyay. You may as well not thrash the heap space(s) > getting there. > Interesting, I did set -Xms and -Xmx to the same value (as you and others-on-this-list have recommended, thanks). > > > My Windows 2008 R2 Server (64bit 32GB RAM) never seems to get > > higher than 1% CPU, and I think I do have memory leaks somewhere in > > the app, but FWIW (in heap dump in java visual vm), the memory > > leaks seem to be tomee leaks. In Java Visual VM, I do see the > > memory grow over time, as the app is used (without a tomcat restart > > or re-deploy of app and then restart tomcat), but I still have not > > seen OOME...'yet'. > > What does your heap usage graph look like? It should be a nice > sawtooth-looking thing, like this: > > /|/|/|/|/|/|/|/|/| > > I do occasionally see the sawtooth-looking graph, > You'll see that the small sawtooth pattern grows in basis over time > and then there is a major GC which will reset you back to some > baseline, then the process starts over again. > and eventually, I see the graph even out (non-sawtooth-looking graph). > > If you never get OOMEs, why do you think you have memory leaks? remember, I do restart tomee quite often, especially
Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Saumil, On 4/3/13 2:16 PM, saumil shah wrote: > I did change in Tomcat 6.0/Conf/server.xml unpackWARs="false" and > autoDeploy="false" Why? I don't believe I mentioned either of those settings... > but the logs still complaint about the Deploying web > applications Tomcat should not show an error during startup. If you are seeing errors during startup, please post them. > org.apache.catalina.startup.HostConfig deployWAR INFO: Deploying > web application archive CmcAppActions.war > org.apache.catalina.core.StandardContextaddApplicationListener > INFO: The listener "com.sun.faces.config.ConfigureListener" is > alreadyconfigured for this context. The duplicate definition has > been ignored. That's entirely an application problem: Tomcat doesn't ship with JavaFaces. Check your web.xml: you likely have the same listener configured twice (just like the INFO message says). > 2. Also for a 64 bit JVM , could I then just bump up my Tomcat > Java params as : -Xms1024m -Xmx4096m instead of my default as > -Xms512m -Xmx1024m -XX:MaxPermSize=1024m instead of my default > -XX:MaxPermSize=512m You can set your heap to whatever you want. IMO, there's no reason to set the heap any higher than you actually need to support your expected usage. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJRXcfvAAoJEBzwKT+lPKRYI88P/R1LW2S+whP/WZ9kaaohwrPb 2pcPVxtmw+imWftSP4W451smB1qYkBNiUVtuv3p3pg0qWEgXB+oxpn0K+bqVx1O1 KJp7NFQDsXqcu2N3wxcOUGPzg1ZbZQMpimm0sN7FcGQiTDQbWvcVbcpQEI3zez0B eAVAPS2txuT4ew5UCsNwsNF14GgAqton+3M6zd3+xfMYMjYa7jyKm13xmgKHEZ/a gSyHVh0zFxWyMhV0fD5LFArCXYcw11tgHO0mflXoJc9zkC8mpF+2xNMecQXe/IMK zieto5v93REF3eivvqk6cySlWeygs35WAuvFdAMsmtCBXeLRx0aptoM6Z+PrRFLj dlAQMCuN4irZlXyPOlroAd79R5N1NVUjkm6dLqTM9R6T+VGuechozSGM66Q34stC dTAa6Pxe0MUAQ79IDl+J3/Kg93Znviwo0MREObOX6FqpLXpKDAzX13TypExnXUjA JzZ/HGsm8XLK2q/inPfIlUpiCRTkZG81Ff1yq9ekQrqD57rYHYHAcRvrpVgWp5/p WpgdxdHrHo732w6EeDKcJDzBTzhmKpfYDcmcygndKfNGShQDQBXJ5TDniGsvZXS8 cM6LtHTRV+q1VyYQlMclAQfwQYp9JGJSZIKR2lZtvLti+f0HX4Kob5ZC10MpS4g+ V5nxNrxKzluUFeMsVQO0 =c84e -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Howard, On 4/3/13 4:15 PM, Howard W. Smith, Jr. wrote: > On Tue, Apr 2, 2013 at 5:12 PM, Christopher Schultz < > ch...@christopherschultz.net> wrote: > >> If you don't re-deploy your webapp, then daily rolling Tomcat >> restarts are not necessary. I wonder why you are re-deploying >> your web application so many times? > > As a new tomcat user and still somewhat junior java/jsf developer, > I restart tomcat whenever I have new software changes to > deploy-and-want-to-run-on the production server. sometimes, I > deploy-and-restart multiple times per day, but sometimes, I'm able > to let tomcat/tomee run for days without restart. That's not really conducive to high-availability. Are you using Tomcat's parallel-deployment feature? >> We run several Tomcats in parallel with modest heaps (less than >> 512MiB each) and they can run for months before we stop them for >> upgrades. It *is* possible to run JVMs without running out of >> memory... > > > I too, have not experienced any OOME, and recently, per what I > have seen-and-read of other (more senior java developers than > myself), I have decreased memory settings in my java options on > tomcat7w.exe (see below). > > -Xmx1024m -XX:MaxPermSize=384m -XX:+UseTLAB > -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled Your heap settings should be tailored to your environment and usage scenarios. You can find "conventional wisdom" that recommends pretty much any heap configuration you want. The only thing that I can consistently recommend to anyone is to set -Xms and -Xmx to the same value, since on a server you're pretty much guaranteed to get to -Xmx pretty quickly, anwyay. You may as well not thrash the heap space(s) getting there. > My Windows 2008 R2 Server (64bit 32GB RAM) never seems to get > higher than 1% CPU, and I think I do have memory leaks somewhere in > the app, but FWIW (in heap dump in java visual vm), the memory > leaks seem to be tomee leaks. In Java Visual VM, I do see the > memory grow over time, as the app is used (without a tomcat restart > or re-deploy of app and then restart tomcat), but I still have not > seen OOME...'yet'. What does your heap usage graph look like? It should be a nice sawtooth-looking thing, like this: /|/|/|/|/|/|/|/|/| You'll see that the small sawtooth pattern grows in basis over time and then there is a major GC which will reset you back to some baseline, then the process starts over again. If you never get OOMEs, why do you think you have memory leaks? Or that TomEE does? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJRXcdIAAoJEBzwKT+lPKRYb70P/jI9QtV1+anIXCx04niQNDJP BSipr2ScOQ2DJNNBjwWAth9AqbzC/nKX4fNlqKCSqWmpZf3JyFChFi/nQJajZQRR 3EJEYLXP3YBRPfixRZLrcUw9rrVLs/+apFL4XZCdgkFvrPto201E9ef99ns7Ya+g oBWr51g8Fkx5jeztRmzg+H/8+KVICuHUVXTCBQtuybwTfCyWKFwZs5lfmKiX7oKC 1ts7S6mIhzpFjO+KFxnFWZkM4ch7SADqCe5WhxnOYpjaz7p7d2XGv1iamjsGVKF2 FE1bZvOiyEFga6vlh1TpAFmAnOmHmkHtxNAG6eSLeOfYkxKAqSy4z5O6leoP/tUc Vri2+GIxwyxyUYprTnvQW7S1UTQF6t658itnZXLrnvRr8Oj/7tzmFcScDvajbYUx P8tECk7SENg0Sr6T8tY3VjR+A1c1u8i5k9iLDayVE8zxcDvCA6n2cVmtj/5K3xwe MNWhsr6kfca9q7CU5fIe38ebrsw8+UU4J2TkrtHYUpc9uC1AxXmLcC8LEXCV/IiZ pIHVlIj2VztPODHfpQvPVUH7VkQKu0M3WbeR+OTQDJJs8X7MLou+YFao+3ri0x9/ 09i6VS+AC8E8zqQcl0KNeBJf7kuTKP/lguLcCRby0YC2YuppwHaBcRiECKy1UDZR FB/X9VbiGKGt95lZvZae =X3iQ -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
On 3 Apr 2013, at 15:36, Christopher Schultz wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Saumil, > > Please try to keep discussions on the mailing list to everyone can > benefit. > > On 4/2/13 6:48 PM, saumil shah wrote: >> For some reason ...I do not see Java process in Task Manager in >> Windows, just Tomcat6 process. I am assuming killing Tomcat6 kills >> JVM internally...is this correct assumption ? > > I'm not sure what the Tomca6 process is, but if you kill the service > (and nothing goes wrong), the JVM should terminate. Tomcat6.exe is the thin windows wrapper used to start the JVM. >> Also, I am NOT re-deploying the application WAR's for some >> reason...when I deployed the application using Tomcat Manager ... >> for some reason it put WAR in the WebApps folder and then exploded >> them. > That is entirely expected behavior. "For some reason": the reason being that it has to put them somewhere and as they are basically a zip, it unpacks he zip and loads the app from the unpacked dir. > If you don't want your webapps to > be deployed on startup, you'll need to set deployOnStartup="false" in > your . > > If you set deplyOnStartup="false", you'll have to use the Manager > webapp to deploy your webapps after Tomcat has started. This is probably a red herring. >> Now it seems every time I start Tomcat , it tries to Re-deploy the >> application because it finds .WAR there ? > > I wouldn't use the term "re-deploy" since the webapp isn't running > prior to that. I usually use the term re-deploy to mean that the > webapp is taken out of service in a running Tomcat, then a new webapp > is deployed in its place (usually on the same context path). The question should be: are you restarting Tomcat frequently already and if so, why? p > - -chris > -BEGIN PGP SIGNATURE- > Version: GnuPG/MacGPG2 v2.0.17 (Darwin) > Comment: GPGTools - http://gpgtools.org > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQIcBAEBCAAGBQJRXD5GAAoJEBzwKT+lPKRYq0gQAJZ9E4OSrYj+ypDTmHUV5ADa > GdUbo/TYfyZNPFjmHzl/GYeZlYTirs24dA95VacIiAdJhtZ2g0J44Oc6PnT9pqmR > RxcGtqNF+kN+AV5i0Lf4ewAZE8MQBvWmBHumqt75ETEsYYAlEv0NaVza9EcM7DNi > RTpR9MrD4QUS7jhrKsyvQiagL7hF8xJgDij9CY5Bc5j6wQjuh6nttW4JtJXOTxb1 > iiMbQqndmX0RkZHY3Dw3BJOuFU/NjAmSGB5pfmDBrA+z6jasH4SZd0KOy3DKcSgX > EV8ja57U161yJMdH7Bt7ap9C2mpAaoFKMvANaPYCy29oRcUgQ3qMB7lofRCy2NZ5 > JDWDnVaKa4UdXoCK4pUvt/noo4EZwUhHI9y8IAtCOC+5xgEDA65FPeTnXBBWYqyu > uASODSRe4DdQYfLJjMw6rWmGFCqM+aPICeZKcXlC+UR8eqp3pmoLL1I5Y15xth2y > mW9dA/qVAUNLUXlW1oWw7b58UAyCi7nVSASC4p3LcQ1K+cnQ8j8/VB0GtMMgA2gL > Bpk7RR19NlzeUXmK//AN/BthZY6oK09qz6n+yNTi2tiV4T2XCjlRO4dsLOoW73nP > FPTMSFfzYaeBKkn0pHsORZLCtWz8bZ060dYrfKkKLSKhk11CzwGNeP4A5So9xa+J > ncBMVYTGyKnanbDE3Fj9 > =dZYT > -END PGP SIGNATURE- > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
On 3 Apr 2013, at 19:16, saumil shah wrote: > Thanks again Chris > I did change in Tomcat 6.0/Conf/server.xml > unpackWARs="false" and autoDeploy="false" > but the logs still complaint about the Deploying web applications > org.apache.catalina.startup.HostConfig deployWARINFO: Deploying web > application archive > CmcAppActions.warorg.apache.catalina.core.StandardContextaddApplicationListenerINFO: > The listener "com.sun.faces.config.ConfigureListener" is alreadyconfigured > for this context. The duplicate definition has been ignored. > 2. Also for a 64 bit JVM , could I then just bump up my Tomcat Java params > as : > -Xms1024m -Xmx4096m instead of my default as -Xms512m -Xmx1024m > -XX:MaxPermSize=1024m instead of my default -XX:MaxPermSize=512m Yes, but first please tell us where the original settings came from. 512M seems a little high to me for MaxPermSize. You should enable JMX on this Tomcat instance and use VisualVM (or JConsole), which you can find in the JDK bin directory to inspect the process memory use. p > > Appreciate all your help. > Thanks again. >> Date: Wed, 3 Apr 2013 10:35:50 -0400 >> From: ch...@christopherschultz.net >> To: users@tomcat.apache.org >> CC: saumil...@hotmail.com >> Subject: Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit) >> >> -BEGIN PGP SIGNED MESSAGE- >> Hash: SHA256 >> >> Saumil, >> >> Please try to keep discussions on the mailing list to everyone can >> benefit. >> >> On 4/2/13 6:48 PM, saumil shah wrote: >>> For some reason ...I do not see Java process in Task Manager in >>> Windows, just Tomcat6 process. I am assuming killing Tomcat6 kills >>> JVM internally...is this correct assumption ? >> >> I'm not sure what the Tomca6 process is, but if you kill the service >> (and nothing goes wrong), the JVM should terminate. >> >>> Also, I am NOT re-deploying the application WAR's for some >>> reason...when I deployed the application using Tomcat Manager ... >>> for some reason it put WAR in the WebApps folder and then exploded >>> them. >> >> That is entirely expected behavior. If you don't want your webapps to >> be deployed on startup, you'll need to set deployOnStartup="false" in >> your . >> >> If you set deplyOnStartup="false", you'll have to use the Manager >> webapp to deploy your webapps after Tomcat has started. >> >>> Now it seems every time I start Tomcat , it tries to Re-deploy the >>> application because it finds .WAR there ? >> >> I wouldn't use the term "re-deploy" since the webapp isn't running >> prior to that. I usually use the term re-deploy to mean that the >> webapp is taken out of service in a running Tomcat, then a new webapp >> is deployed in its place (usually on the same context path). >> >> - -chris >> -BEGIN PGP SIGNATURE- >> Version: GnuPG/MacGPG2 v2.0.17 (Darwin) >> Comment: GPGTools - http://gpgtools.org >> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ >> >> iQIcBAEBCAAGBQJRXD5GAAoJEBzwKT+lPKRYq0gQAJZ9E4OSrYj+ypDTmHUV5ADa >> GdUbo/TYfyZNPFjmHzl/GYeZlYTirs24dA95VacIiAdJhtZ2g0J44Oc6PnT9pqmR >> RxcGtqNF+kN+AV5i0Lf4ewAZE8MQBvWmBHumqt75ETEsYYAlEv0NaVza9EcM7DNi >> RTpR9MrD4QUS7jhrKsyvQiagL7hF8xJgDij9CY5Bc5j6wQjuh6nttW4JtJXOTxb1 >> iiMbQqndmX0RkZHY3Dw3BJOuFU/NjAmSGB5pfmDBrA+z6jasH4SZd0KOy3DKcSgX >> EV8ja57U161yJMdH7Bt7ap9C2mpAaoFKMvANaPYCy29oRcUgQ3qMB7lofRCy2NZ5 >> JDWDnVaKa4UdXoCK4pUvt/noo4EZwUhHI9y8IAtCOC+5xgEDA65FPeTnXBBWYqyu >> uASODSRe4DdQYfLJjMw6rWmGFCqM+aPICeZKcXlC+UR8eqp3pmoLL1I5Y15xth2y >> mW9dA/qVAUNLUXlW1oWw7b58UAyCi7nVSASC4p3LcQ1K+cnQ8j8/VB0GtMMgA2gL >> Bpk7RR19NlzeUXmK//AN/BthZY6oK09qz6n+yNTi2tiV4T2XCjlRO4dsLOoW73nP >> FPTMSFfzYaeBKkn0pHsORZLCtWz8bZ060dYrfKkKLSKhk11CzwGNeP4A5So9xa+J >> ncBMVYTGyKnanbDE3Fj9 >> =dZYT >> -END PGP SIGNATURE- >> >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
Chris, On Tue, Apr 2, 2013 at 5:12 PM, Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > > > I understand its not a premanent solution but as a stopgap for now > > ? If so , we can put it as part of daily cycle to bounce tomcat6. > > If you don't re-deploy your webapp, then daily rolling Tomcat restarts > are not necessary. I wonder why you are re-deploying your web > application so many times? > > As a new tomcat user and still somewhat junior java/jsf developer, I restart tomcat whenever I have new software changes to deploy-and-want-to-run-on the production server. sometimes, I deploy-and-restart multiple times per day, but sometimes, I'm able to let tomcat/tomee run for days without restart. > We run several Tomcats in parallel with modest heaps (less than 512MiB > each) and they can run for months before we stop them for upgrades. It > *is* possible to run JVMs without running out of memory... > I too, have not experienced any OOME, and recently, per what I have seen-and-read of other (more senior java developers than myself), I have decreased memory settings in my java options on tomcat7w.exe (see below). -Xmx1024m -XX:MaxPermSize=384m -XX:+UseTLAB -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled On Java tab of tomcat7w.exe, have the following specified: Initial memory pool = 256 Maximum memory pool = Thread stack size = My Windows 2008 R2 Server (64bit 32GB RAM) never seems to get higher than 1% CPU, and I think I do have memory leaks somewhere in the app, but FWIW (in heap dump in java visual vm), the memory leaks seem to be tomee leaks. In Java Visual VM, I do see the memory grow over time, as the app is used (without a tomcat restart or re-deploy of app and then restart tomcat), but I still have not seen OOME...'yet'. I should leave the server up and running without restarting tomcat, but when I need to deploy new software changes, i can't wait to deploy. :) My apologies, not trying to hijack the thread. just sharing my response to your comments. Howard > > - -chris > -BEGIN PGP SIGNATURE- > Version: GnuPG/MacGPG2 v2.0.17 (Darwin) > Comment: GPGTools - http://gpgtools.org > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQIcBAEBCAAGBQJRW0m2AAoJEBzwKT+lPKRYt38QAJkMt1+S+h48OdnVU8GVNY+x > nhBJMX2THAN9d6LIo7k/8bEUPlbBIbjZVHZeR7tPbt3HgcPaPXLZ+/eiPU7x3tEe > ubn/Vp7WVNhrZ3+ZcHQz6AEBz5JfFoHNpwfPvopTxsClhdlbNkQVLoNQX2DHvao6 > ARFG3pZWX8HlC/MYMoIsJLD6DxXj9zT0E1ZJzBImHt8r2zE9YPWo35k6RhnEjLut > Ol74NejK8kD8orgGfJfz5bM6XXiWaxLm3tBXkukefcEC9Sq5/PMDP61npTBygFgK > GhPCoKHfvxtm/oXIFHJrwfibpyoKa5gxdRPRey9PqqKc9zABw8t7oMHb/QxwC6qi > qNe9xy5/iSK3NKqQLawXMsFmGfmTA2Rx13I0uP7TnP/2X0bjdjNa/uHx+VKPYY/U > RfdqosN02x8BwXkXbDRzeQURiTAPwSKdH8PBL9itnYSLGi6byYynfsvsPeycbpuK > zV0qdyvBOeHbz/j5dai7Z451PMxm5ccEZ1B8cPtQiRVXxx5+5lF3q9RFwtkenLbI > IpfIrpj4D+96KP66Rn0yfVNEeHosljAOJaYLHuexObna2jjkqwgttzQW4KVajeZm > I1BPQOSHCQVg+Qo2ewVpd6YrnygpTRF6XsKhA/gwuHS+Jy5xJVsXMPNO/kl0y/6f > k9TjfhdTRUpMqn5siO68 > =DjF0 > -END PGP SIGNATURE- > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
RE: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
Thanks again Chris I did change in Tomcat 6.0/Conf/server.xml unpackWARs="false" and autoDeploy="false" but the logs still complaint about the Deploying web applications org.apache.catalina.startup.HostConfig deployWARINFO: Deploying web application archive CmcAppActions.warorg.apache.catalina.core.StandardContextaddApplicationListenerINFO: The listener "com.sun.faces.config.ConfigureListener" is alreadyconfigured for this context. The duplicate definition has been ignored. 2. Also for a 64 bit JVM , could I then just bump up my Tomcat Java params as : -Xms1024m -Xmx4096m instead of my default as -Xms512m -Xmx1024m -XX:MaxPermSize=1024m instead of my default -XX:MaxPermSize=512m Appreciate all your help. Thanks again. > Date: Wed, 3 Apr 2013 10:35:50 -0400 > From: ch...@christopherschultz.net > To: users@tomcat.apache.org > CC: saumil...@hotmail.com > Subject: Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit) > > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Saumil, > > Please try to keep discussions on the mailing list to everyone can > benefit. > > On 4/2/13 6:48 PM, saumil shah wrote: > > For some reason ...I do not see Java process in Task Manager in > > Windows, just Tomcat6 process. I am assuming killing Tomcat6 kills > > JVM internally...is this correct assumption ? > > I'm not sure what the Tomca6 process is, but if you kill the service > (and nothing goes wrong), the JVM should terminate. > > > Also, I am NOT re-deploying the application WAR's for some > > reason...when I deployed the application using Tomcat Manager ... > > for some reason it put WAR in the WebApps folder and then exploded > > them. > > That is entirely expected behavior. If you don't want your webapps to > be deployed on startup, you'll need to set deployOnStartup="false" in > your . > > If you set deplyOnStartup="false", you'll have to use the Manager > webapp to deploy your webapps after Tomcat has started. > > > Now it seems every time I start Tomcat , it tries to Re-deploy the > > application because it finds .WAR there ? > > I wouldn't use the term "re-deploy" since the webapp isn't running > prior to that. I usually use the term re-deploy to mean that the > webapp is taken out of service in a running Tomcat, then a new webapp > is deployed in its place (usually on the same context path). > > - -chris > -BEGIN PGP SIGNATURE- > Version: GnuPG/MacGPG2 v2.0.17 (Darwin) > Comment: GPGTools - http://gpgtools.org > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQIcBAEBCAAGBQJRXD5GAAoJEBzwKT+lPKRYq0gQAJZ9E4OSrYj+ypDTmHUV5ADa > GdUbo/TYfyZNPFjmHzl/GYeZlYTirs24dA95VacIiAdJhtZ2g0J44Oc6PnT9pqmR > RxcGtqNF+kN+AV5i0Lf4ewAZE8MQBvWmBHumqt75ETEsYYAlEv0NaVza9EcM7DNi > RTpR9MrD4QUS7jhrKsyvQiagL7hF8xJgDij9CY5Bc5j6wQjuh6nttW4JtJXOTxb1 > iiMbQqndmX0RkZHY3Dw3BJOuFU/NjAmSGB5pfmDBrA+z6jasH4SZd0KOy3DKcSgX > EV8ja57U161yJMdH7Bt7ap9C2mpAaoFKMvANaPYCy29oRcUgQ3qMB7lofRCy2NZ5 > JDWDnVaKa4UdXoCK4pUvt/noo4EZwUhHI9y8IAtCOC+5xgEDA65FPeTnXBBWYqyu > uASODSRe4DdQYfLJjMw6rWmGFCqM+aPICeZKcXlC+UR8eqp3pmoLL1I5Y15xth2y > mW9dA/qVAUNLUXlW1oWw7b58UAyCi7nVSASC4p3LcQ1K+cnQ8j8/VB0GtMMgA2gL > Bpk7RR19NlzeUXmK//AN/BthZY6oK09qz6n+yNTi2tiV4T2XCjlRO4dsLOoW73nP > FPTMSFfzYaeBKkn0pHsORZLCtWz8bZ060dYrfKkKLSKhk11CzwGNeP4A5So9xa+J > ncBMVYTGyKnanbDE3Fj9 > =dZYT > -END PGP SIGNATURE- > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org >
Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Saumil, Please try to keep discussions on the mailing list to everyone can benefit. On 4/2/13 6:48 PM, saumil shah wrote: > For some reason ...I do not see Java process in Task Manager in > Windows, just Tomcat6 process. I am assuming killing Tomcat6 kills > JVM internally...is this correct assumption ? I'm not sure what the Tomca6 process is, but if you kill the service (and nothing goes wrong), the JVM should terminate. > Also, I am NOT re-deploying the application WAR's for some > reason...when I deployed the application using Tomcat Manager ... > for some reason it put WAR in the WebApps folder and then exploded > them. That is entirely expected behavior. If you don't want your webapps to be deployed on startup, you'll need to set deployOnStartup="false" in your . If you set deplyOnStartup="false", you'll have to use the Manager webapp to deploy your webapps after Tomcat has started. > Now it seems every time I start Tomcat , it tries to Re-deploy the > application because it finds .WAR there ? I wouldn't use the term "re-deploy" since the webapp isn't running prior to that. I usually use the term re-deploy to mean that the webapp is taken out of service in a running Tomcat, then a new webapp is deployed in its place (usually on the same context path). - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJRXD5GAAoJEBzwKT+lPKRYq0gQAJZ9E4OSrYj+ypDTmHUV5ADa GdUbo/TYfyZNPFjmHzl/GYeZlYTirs24dA95VacIiAdJhtZ2g0J44Oc6PnT9pqmR RxcGtqNF+kN+AV5i0Lf4ewAZE8MQBvWmBHumqt75ETEsYYAlEv0NaVza9EcM7DNi RTpR9MrD4QUS7jhrKsyvQiagL7hF8xJgDij9CY5Bc5j6wQjuh6nttW4JtJXOTxb1 iiMbQqndmX0RkZHY3Dw3BJOuFU/NjAmSGB5pfmDBrA+z6jasH4SZd0KOy3DKcSgX EV8ja57U161yJMdH7Bt7ap9C2mpAaoFKMvANaPYCy29oRcUgQ3qMB7lofRCy2NZ5 JDWDnVaKa4UdXoCK4pUvt/noo4EZwUhHI9y8IAtCOC+5xgEDA65FPeTnXBBWYqyu uASODSRe4DdQYfLJjMw6rWmGFCqM+aPICeZKcXlC+UR8eqp3pmoLL1I5Y15xth2y mW9dA/qVAUNLUXlW1oWw7b58UAyCi7nVSASC4p3LcQ1K+cnQ8j8/VB0GtMMgA2gL Bpk7RR19NlzeUXmK//AN/BthZY6oK09qz6n+yNTi2tiV4T2XCjlRO4dsLOoW73nP FPTMSFfzYaeBKkn0pHsORZLCtWz8bZ060dYrfKkKLSKhk11CzwGNeP4A5So9xa+J ncBMVYTGyKnanbDE3Fj9 =dZYT -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Saumil, On 4/2/13 5:01 PM, saumil shah wrote: > Thanks so much Chris this has been very very helpful ... > appreciate you taking time out. Would stopping Tomcat6 service in > Windows 2008 R2 take care of this "orphan threads" if you will ? > Also, does force kill taskkill tomcat6 help either ? If the JVM shuts down, then everything goes away and you can start fresh. I assume you are getting an OutOfMemoryError at some point... what is the exact message and any stack trace that accompanies it? > I understand its not a premanent solution but as a stopgap for now > ? If so , we can put it as part of daily cycle to bounce tomcat6. If you don't re-deploy your webapp, then daily rolling Tomcat restarts are not necessary. I wonder why you are re-deploying your web application so many times? We run several Tomcats in parallel with modest heaps (less than 512MiB each) and they can run for months before we stop them for upgrades. It *is* possible to run JVMs without running out of memory... - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJRW0m2AAoJEBzwKT+lPKRYt38QAJkMt1+S+h48OdnVU8GVNY+x nhBJMX2THAN9d6LIo7k/8bEUPlbBIbjZVHZeR7tPbt3HgcPaPXLZ+/eiPU7x3tEe ubn/Vp7WVNhrZ3+ZcHQz6AEBz5JfFoHNpwfPvopTxsClhdlbNkQVLoNQX2DHvao6 ARFG3pZWX8HlC/MYMoIsJLD6DxXj9zT0E1ZJzBImHt8r2zE9YPWo35k6RhnEjLut Ol74NejK8kD8orgGfJfz5bM6XXiWaxLm3tBXkukefcEC9Sq5/PMDP61npTBygFgK GhPCoKHfvxtm/oXIFHJrwfibpyoKa5gxdRPRey9PqqKc9zABw8t7oMHb/QxwC6qi qNe9xy5/iSK3NKqQLawXMsFmGfmTA2Rx13I0uP7TnP/2X0bjdjNa/uHx+VKPYY/U RfdqosN02x8BwXkXbDRzeQURiTAPwSKdH8PBL9itnYSLGi6byYynfsvsPeycbpuK zV0qdyvBOeHbz/j5dai7Z451PMxm5ccEZ1B8cPtQiRVXxx5+5lF3q9RFwtkenLbI IpfIrpj4D+96KP66Rn0yfVNEeHosljAOJaYLHuexObna2jjkqwgttzQW4KVajeZm I1BPQOSHCQVg+Qo2ewVpd6YrnygpTRF6XsKhA/gwuHS+Jy5xJVsXMPNO/kl0y/6f k9TjfhdTRUpMqn5siO68 =DjF0 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
Thanks so much Chris this has been very very helpful ... appreciate you taking time out. Would stopping Tomcat6 service in Windows 2008 R2 take care of this "orphan threads" if you will ? Also, does force kill taskkill tomcat6 help either ? I understand its not a premanent solution but as a stopgap for now ? If so , we can put it as part of daily cycle to bounce tomcat6. Many thanks. > Date: Tue, 2 Apr 2013 15:58:53 -0400 > From: ch...@christopherschultz.net > To: users@tomcat.apache.org > Subject: Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit) > > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Saumil, > > On 4/1/13 11:01 PM, saumil shah wrote: > > I recently deployed one of the COTS products SAP Business Objects. > > When the product was deployed , everything seemed to run fine but > > yesterday we started experiencing "Service is unavailable" error , > > upon enabling DEBUG logs in Tomcat , we saw the error below with > > memory leak message. We were wondering what could have caused it > > > > > > The Tomcat is 64 bit , JVM is 64 bit but the applications deployed > > are 32 bit webapps we say Tomcat6 process becoming > > unresponsive around 2GB mark. > > > > > > Apr 1, 2013 8:53:18 PM org.apache.coyote.http11.Http11Protocol > > pauseINFO: Pausing Coyote HTTP/1.1 on http-8080Apr 1, 2013 8:53:19 > > PM org.apache.catalina.core.StandardService stopINFO: Stopping > > service CatalinaApr 1, 2013 8:53:19 PM > > org.apache.catalina.loader.WebappClassLoader > > clearReferencesThreadsSEVERE: The web application > > [/AnalyticalReporting] appears to have started a thread named > > [Business Objects - Sessions Clean up] but has failed to stop it. > > This is very likely to create a memory leak. > > This means what it says it means: your webapp (not Tomcat) started a > thread that wasn't stopped by the time the webapp was stopped. This > will pin the webapp's ClassLoader in memory and you'll get a whole > bunch of stuff that can't be GC'd. The simple explanation is that you > need to be sure that your threads are stopped. The solution may be > complicated. > > Presumably any component that starts a thread also knows how to stop > that same thread. Check the documentation for those components to make > sure that you are shutting them down properly (usually in a > ServletContextListener, etc.). > > Sometimes, it takes a few seconds for a thread to complete its work > and actually terminate, and the messages you get from Tomcat are > inaccurate. That's for you to determine. > > > Apr 1, 2013 8:53:19 PM > > org.apache.catalina.loader.WebappClassLoader > > clearReferencesThreadsSEVERE: The web application > > [/AnalyticalReporting] appears to have started a thread named > > [AWT-Windows] but has failed to stop it. This is very likely to > > create a memory leak. > > Read the documentation for the JreMemoryLeakLeakPreventionListener. > This is a standard Tomcat component that can be used to work-around > well-known memory leaks, etc. that are triggered by certain JVM calls. > There are specific configurations for preventing AWT threads from > being problematic, but they are not enabled by default. You probably > want to enable them if you are using AWT for anything. > > > Apr 1, 2013 8:53:19 PM > > org.apache.catalina.loader.WebappClassLoader > > clearReferencesThreadsSEVERE: The web application > > [/AnalyticalReporting] appears to have started a thread named > > [ORBacus:Client:SenderThread] but has failed to stop it. This is > > very likely to create a memory leak. > > Anything with "ORB" in the name usually means a CORBA-related > component: check to see that anything that start threads is also > stopping them. > > > Apr 1, 2013 8:53:19 PM > > org.apache.catalina.loader.WebappClassLoader > > clearThreadLocalMapSEVERE: The web application > > [/AnalyticalReporting] created a ThreadLocal with key of type > > [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@63e3d2dd]) > > and a value of type [com.businessobjects.wp.xml.jaxp.XMLJaxpParser] > > (value [com.businessobjects.wp.xml.jaxp.XMLJaxpParser@228de67]) but > > failed to remove it when the web application was stopped. This is > > very likely to create a memory leak. > > Looks like your Business Objects version has a bug in it that does not > clear-out ThreadLocal values when requests are completed. Talk to them > about this problem. > > > Apr 1, 2013 8:53:20 PM > > org.apache.catalina.loader.WebappClassLoad
Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Saumil, On 4/1/13 11:01 PM, saumil shah wrote: > I recently deployed one of the COTS products SAP Business Objects. > When the product was deployed , everything seemed to run fine but > yesterday we started experiencing "Service is unavailable" error , > upon enabling DEBUG logs in Tomcat , we saw the error below with > memory leak message. We were wondering what could have caused it > > > The Tomcat is 64 bit , JVM is 64 bit but the applications deployed > are 32 bit webapps we say Tomcat6 process becoming > unresponsive around 2GB mark. > > > Apr 1, 2013 8:53:18 PM org.apache.coyote.http11.Http11Protocol > pauseINFO: Pausing Coyote HTTP/1.1 on http-8080Apr 1, 2013 8:53:19 > PM org.apache.catalina.core.StandardService stopINFO: Stopping > service CatalinaApr 1, 2013 8:53:19 PM > org.apache.catalina.loader.WebappClassLoader > clearReferencesThreadsSEVERE: The web application > [/AnalyticalReporting] appears to have started a thread named > [Business Objects - Sessions Clean up] but has failed to stop it. > This is very likely to create a memory leak. This means what it says it means: your webapp (not Tomcat) started a thread that wasn't stopped by the time the webapp was stopped. This will pin the webapp's ClassLoader in memory and you'll get a whole bunch of stuff that can't be GC'd. The simple explanation is that you need to be sure that your threads are stopped. The solution may be complicated. Presumably any component that starts a thread also knows how to stop that same thread. Check the documentation for those components to make sure that you are shutting them down properly (usually in a ServletContextListener, etc.). Sometimes, it takes a few seconds for a thread to complete its work and actually terminate, and the messages you get from Tomcat are inaccurate. That's for you to determine. > Apr 1, 2013 8:53:19 PM > org.apache.catalina.loader.WebappClassLoader > clearReferencesThreadsSEVERE: The web application > [/AnalyticalReporting] appears to have started a thread named > [AWT-Windows] but has failed to stop it. This is very likely to > create a memory leak. Read the documentation for the JreMemoryLeakLeakPreventionListener. This is a standard Tomcat component that can be used to work-around well-known memory leaks, etc. that are triggered by certain JVM calls. There are specific configurations for preventing AWT threads from being problematic, but they are not enabled by default. You probably want to enable them if you are using AWT for anything. > Apr 1, 2013 8:53:19 PM > org.apache.catalina.loader.WebappClassLoader > clearReferencesThreadsSEVERE: The web application > [/AnalyticalReporting] appears to have started a thread named > [ORBacus:Client:SenderThread] but has failed to stop it. This is > very likely to create a memory leak. Anything with "ORB" in the name usually means a CORBA-related component: check to see that anything that start threads is also stopping them. > Apr 1, 2013 8:53:19 PM > org.apache.catalina.loader.WebappClassLoader > clearThreadLocalMapSEVERE: The web application > [/AnalyticalReporting] created a ThreadLocal with key of type > [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@63e3d2dd]) > and a value of type [com.businessobjects.wp.xml.jaxp.XMLJaxpParser] > (value [com.businessobjects.wp.xml.jaxp.XMLJaxpParser@228de67]) but > failed to remove it when the web application was stopped. This is > very likely to create a memory leak. Looks like your Business Objects version has a bug in it that does not clear-out ThreadLocal values when requests are completed. Talk to them about this problem. > Apr 1, 2013 8:53:20 PM > org.apache.catalina.loader.WebappClassLoader > clearThreadLocalMapSEVERE: The web application [/VoyagerClient] > created a ThreadLocal with key of type > [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl.ThreadCounterImpl] > > (value > [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl$ThreadCounterImpl@13123b1c]) > > and a value of type > [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl.ThreadCounterImpl.Counter] > > (value > [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl$ThreadCounterImpl$Counter@11e6b50c]) > > but failed to remove it when the web application was stopped. This is > very likely to create a memory leak.Apr 1, 2013 8:53:20 PM > org.apache.catalina.loader.WebappClassLoader > clearThreadLocalMapSEVERE: The web application [/VoyagerClient] > created a ThreadLocal with key of type > [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl.ThreadCounterImpl] > > (value > [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl$ThreadCounterImpl@41783002]) > > and a value of type > [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl.ThreadCounterImpl.Counter] > > (value > [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl$ThreadCounterIm
Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
Caldarale, Charles R wrote: From: saumil shah [mailto:saumil...@hotmail.com] Subject: Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit) Don't top post - it's annoying and confusing. can we deploy 32 bit web applications on 64 bit tomcat? The point being made is that your question doesn't make any sense. There's no such thing as a 32- or 64-bit Tomcat, nor are there 32- or 64-bit webapps. Java code is platform independent. If you have native code that runs in the JVM process, then its mode of execution must match that of the JVM, hence the choice of tcnative add-ons. If your webapp includes native code, you've likely made poor architectural decisions in your application architecture. And the additional point that I was trying to make - admittedly not very explicitly - is the following : in your original post, you seemed to make a connection between the supposedly 32-bit Tomcat and webapps, and the fact that you were encountering some kind of issue around a 2 GB memory size. But if your JVM is 64-bit, this "2 GB" memory size does not really mean anything, so you are likely on the wrong track. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
> From: saumil shah [mailto:saumil...@hotmail.com] > Subject: Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit) Don't top post - it's annoying and confusing. > can we deploy 32 bit web applications on 64 bit tomcat? The point being made is that your question doesn't make any sense. There's no such thing as a 32- or 64-bit Tomcat, nor are there 32- or 64-bit webapps. Java code is platform independent. If you have native code that runs in the JVM process, then its mode of execution must match that of the JVM, hence the choice of tcnative add-ons. If your webapp includes native code, you've likely made poor architectural decisions in your application architecture. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
Thanks andre..can we deploy 32 bit web applications on 64 bit tomcat? -Original Message- From: André Warnier Sent: 2 Apr 2013 08:51:00 GMT To: Tomcat Users List Subject: Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit) saumil shah wrote: > Hello there, > > I recently deployed one of the COTS products SAP Business Objects. When the > product was deployed , everything seemed to run fine but yesterday we started > experiencing "Service is unavailable" error , upon enabling DEBUG logs in > Tomcat , we saw the error below with memory leak message. We were wondering > what could have caused it > The Tomcat is 64 bit (huh ?) , JVM is 64 bit but the applications deployed are 32 bit webapps (huh ?) we say Tomcat6 process becoming unresponsive around 2GB mark. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
saumil shah wrote: Hello there, I recently deployed one of the COTS products SAP Business Objects. When the product was deployed , everything seemed to run fine but yesterday we started experiencing "Service is unavailable" error , upon enabling DEBUG logs in Tomcat , we saw the error below with memory leak message. We were wondering what could have caused it The Tomcat is 64 bit (huh ?) , JVM is 64 bit but the applications deployed are 32 bit webapps (huh ?) we say Tomcat6 process becoming unresponsive around 2GB mark. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
Saumil, On 2.4.2013 5:01, saumil shah wrote: I recently deployed one of the COTS products SAP Business Objects. When the product was deployed , everything seemed to run fine but yesterday we started experiencing "Service is unavailable" error , upon enabling DEBUG logs in Tomcat , we saw the error below with memory leak message. We were wondering what could have caused it The Tomcat is 64 bit , JVM is 64 bit but the applications deployed are 32 bit webapps we say Tomcat6 process becoming unresponsive around 2GB mark. There is a Wiki page [1] that descibes most couses of memory leaks. Until webapp developers check and fix those leaks, you might mitigate them by restarting Tomcat every time you need to redeploy or restart any of your webapps. If that does not help, then you have some other problem. You could investigate further by analyzing thread activity during hangs using thread dumps [2] or monitoring tools like JConsole. -Ognjen [1] https://wiki.apache.org/tomcat/MemoryLeakProtection [2] https://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
Hello there, I recently deployed one of the COTS products SAP Business Objects. When the product was deployed , everything seemed to run fine but yesterday we started experiencing "Service is unavailable" error , upon enabling DEBUG logs in Tomcat , we saw the error below with memory leak message. We were wondering what could have caused it The Tomcat is 64 bit , JVM is 64 bit but the applications deployed are 32 bit webapps we say Tomcat6 process becoming unresponsive around 2GB mark. Your help is appreiated. Many thanks,Sam Apr 1, 2013 8:53:18 PM org.apache.coyote.http11.Http11Protocol pauseINFO: Pausing Coyote HTTP/1.1 on http-8080Apr 1, 2013 8:53:19 PM org.apache.catalina.core.StandardService stopINFO: Stopping service CatalinaApr 1, 2013 8:53:19 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreadsSEVERE: The web application [/AnalyticalReporting] appears to have started a thread named [Business Objects - Sessions Clean up] but has failed to stop it. This is very likely to create a memory leak.Apr 1, 2013 8:53:19 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreadsSEVERE: The web application [/AnalyticalReporting] appears to have started a thread named [AWT-Windows] but has failed to stop it. This is very likely to create a memory leak.Apr 1, 2013 8:53:19 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreadsSEVERE: The web application [/AnalyticalReporting] appears to have started a thread named [ORBacus:Client:SenderThread] but has failed to stop it. This is very likely to create a memory leak.Apr 1, 2013 8:53:19 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreadsSEVERE: The web application [/AnalyticalReporting] appears to have started a thread named [ORBacus:Client:ReceiverThread] but has failed to stop it. This is very likely to create a memory leak.Apr 1, 2013 8:53:19 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreadsSEVERE: The web application [/AnalyticalReporting] appears to have started a thread named [ORBacus:Client:SenderThread] but has failed to stop it. This is very likely to create a memory leak.Apr 1, 2013 8:53:19 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreadsSEVERE: The web application [/AnalyticalReporting] appears to have started a thread named [ORBacus:Client:ReceiverThread] but has failed to stop it. This is very likely to create a memory leak.Apr 1, 2013 8:53:19 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMapSEVERE: The web application [/AnalyticalReporting] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@63e3d2dd]) and a value of type [com.businessobjects.wp.xml.jaxp.XMLJaxpParser] (value [com.businessobjects.wp.xml.jaxp.XMLJaxpParser@228de67]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.Apr 1, 2013 8:53:19 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMapSEVERE: The web application [/AnalyticalReporting] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@63e3d2dd]) and a value of type [com.businessobjects.wp.xml.jaxp.XMLJaxpParser] (value [com.businessobjects.wp.xml.jaxp.XMLJaxpParser@49213d4c]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.Apr 1, 2013 8:53:19 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMapSEVERE: The web application [/AnalyticalReporting] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@63e3d2dd]) and a value of type [com.businessobjects.wp.xml.jaxp.XMLJaxpParser] (value [com.businessobjects.wp.xml.jaxp.XMLJaxpParser@253f6e16]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.Apr 1, 2013 8:53:19 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMapSEVERE: The web application [/AnalyticalReporting] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@63e3d2dd]) and a value of type [com.businessobjects.wp.xml.jaxp.XMLJaxpParser] (value [com.businessobjects.wp.xml.jaxp.XMLJaxpParser@67547974]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.Apr 1, 2013 8:53:19 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMapSEVERE: The web application [/AnalyticalReporting] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@63e3d2dd]) and a value of type [com.businessobjects.wp.xml.jaxp.XMLJaxpParser] (value [com.businessobjects.wp.xml.jaxp.XMLJaxpParser@5af1e3ab]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.Apr 1, 2013 8:53:19 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMapSEV
