Hi Maxim,
We need a heap dump to chase this down. We had a leak (identified in the
release notes) when we shipped. I found that it was about 1MB per hour under
load on a dual 3.0Ghz Xeon (no hyperthreading).
I was using Derby at the time. I've been changing the tranQL connection manager
and haven't seen it appear but I haven't specifically done a long run like I
did before.
It would be helpful to coordinate efforts on what areas your focusing on. I
think the memory leak would be excellent. Can you tell me a bit about your test
setup? I have a couple of engineering boxes you guys provided and they are
outlined in the note you referred to earlier.
Also, can you do your testing with jRockit ? I think the information on another
JVM would be a good thing.
Also, what TZ are you located in? I'm UTC -5 (Raleigh, NC)
Also, check us out on Irc irc://irc.freenode.net:6667 Channel #geronimo
Maxim Berkultsev wrote:
Hi, Matt!
I've tried
java -server -XX:-PrintTenuringDistribution -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -jar ./bin/server.jar
since it seems too long to wait for the problem so I've limited VM with
the default heap size. Please see a portion of log below.
It looks as if the heap was taken up intensively.
For an OutOfMemory problem please see the stack trace further.
Best regards,
Maxim Berkultsev, Intel Middleware Products Division
------------------
.......
535.751: [Tenured[Unloading class
sun.reflect.GeneratedSerializationConstructorAccessor461]:
58303K->58303K(58304K), 0.4943219 secs] 64355K->64332K(64832K), [Perm :
31861K->31855K(32000K)] Heap after GC invocations=1056:
Heap def new generation total 6528K, used 6028K [0x10010000, 0x10720000,
0x10720000)
eden space 5824K, 99% used [0x10010000, 0x105bffd8, 0x105c0000)
from space 704K, 29% used [0x10670000, 0x106a3220, 0x10720000)
to space 704K, 0% used [0x105c0000, 0x105c0000, 0x10670000)
tenured generation total 58304K, used 58303K [0x10720000, 0x14010000,
0x14010000)
the space 58304K, 99% used [0x10720000, 0x1400fff8, 0x14010000, 0x14010000)
compacting perm gen total 32000K, used 31855K [0x14010000, 0x15f50000,
0x18010000)
the space 32000K, 99% used [0x14010000, 0x15f2bdb8, 0x15f2be00, 0x15f50000)}
, 0.4963221 secs]
17:10:00,921 WARN [ServletHandler] Error for
/daytrader/servlet/PingServlet2MDBQueue java.lang.OutOfMemoryError
------------------
19:42:39,401 ERROR [Log] Error: PingServlet2MDBQueue.doGet(...):exception
posting message to TradeBrokerQueue destination
19:42:39,401 ERROR [Log] Error: PingServlet2MDBQueue.doGet(...): error
java.util.ConcurrentModificationException
java.util.ConcurrentModificationException
at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:448)
at java.util.AbstractList$Itr.next(AbstractList.java:419)
at java.util.AbstractCollection.remove(AbstractCollection.java:254)
at org.activemq.TransactionContext.removeSession(TransactionContext.java
:116)
at org.activemq.ra.RATransactionContext.removeSession(
RATransactionContext.java:57)
at org.activemq.ActiveMQSession.doClose(ActiveMQSession.java:466)
at org.activemq.ActiveMQSession.close(ActiveMQSession.java:447)
at org.activemq.ra.JMSSessionProxy.cleanup(JMSSessionProxy.java:87)
at org.activemq.ra.JMSSessionProxy.close(JMSSessionProxy.java:76)
at org.apache.geronimo.samples.daytrader.web.prims.PingServlet2MDBQueue.
doGet(PingServlet2MDBQueue.java:127)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:595)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:688)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:428)
at org.apache.geronimo.jetty.JettyServletHolder.handle(
JettyServletHolder.java:99)
at org.mortbay.jetty.servlet.WebApplicationHandler$CachedChain.doFilter(
WebApplicationHandler.java:830)
at org.mortbay.jetty.servlet.JSR154Filter.doFilter(JSR154Filter.java:170)
at org.mortbay.jetty.servlet.WebApplicationHandler$CachedChain.doFilter(
WebApplicationHandler.java:821)
at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(
WebApplicationHandler.java:471)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:568)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1530)
at org.mortbay.jetty.servlet.WebApplicationContext.handle(
WebApplicationContext.java:633)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1482)
at org.mortbay.http.HttpServer.service(HttpServer.java:909)
at org.mortbay.http.HttpConnection.service(HttpConnection.java:816)
at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:982)
at org.mortbay.http.HttpConnection.handle(HttpConnection.java:833)
at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244)
at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
java.util.ConcurrentModificationException
at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:448)
at java.util.AbstractList$Itr.next(AbstractList.java:419)
at java.util.AbstractCollection.remove(AbstractCollection.java:254)
at org.activemq.TransactionContext.removeSession(TransactionContext.java
:116)
at org.activemq.ra.RATransactionContext.removeSession(
RATransactionContext.java:57)
at org.activemq.ActiveMQSession.doClose(ActiveMQSession.java:466)
at org.activemq.ActiveMQSession.close(ActiveMQSession.java:447)
at org.activemq.ra.JMSSessionProxy.cleanup(JMSSessionProxy.java:87)
at org.activemq.ra.JMSSessionProxy.close(JMSSessionProxy.java:76)
at org.apache.geronimo.samples.daytrader.web.prims.PingServlet2MDBQueue:
doGet(PingServlet2MDBQueue.java:127)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:595)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:688)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:428)
at org.apache.geronimo.jetty.JettyServletHolder.handle(
JettyServletHolder.java:99)
at org.mortbay.jetty.servlet.WebApplicationHandler$CachedChain.doFilter(
WebApplicationHandler.java:830)
at org.mortbay.jetty.servlet.JSR154Filter.doFilter(JSR154Filter.java:170)
at org.mortbay.jetty.servlet.WebApplicationHandler$CachedChain.doFilter(
WebApplicationHandler.java:821)
at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(
WebApplicationHandler.java:471)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:568)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1530)
at org.mortbay.jetty.servlet.WebApplicationContext.handle(
WebApplicationContext.java:633)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1482)
at org.mortbay.http.HttpServer.service(HttpServer.java:909)
at org.mortbay.http.HttpConnection.service(HttpConnection.java:816)
at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:982)
at org.mortbay.http.HttpConnection.handle(HttpConnection.java:833)
at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244)
at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
19:43:29,178 WARN [ServletHandler] Error for
/daytrader/servlet/PingServlet2MDBQueue
java.lang.OutOfMemoryError
19:43:30,365 WARN [HttpConnection] GET
/daytrader/servlet/PingServlet2MDBQueue HTTP/1.1
java.lang.OutOfMemoryError
19:43:31,412 WARN [JournalMessageStore] Message could not be added to long
term store: null
java.lang.OutOfMemoryError
----------
2006/3/29, Matt Hogstrom wrote:
In the tests I'm running I use the following:
java -server -Xmx2048m -Xms2048m -XX:-PrintTenuringDistribution
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -jar
/home/hogstrom/geronimo-1.0/bin/server.jar
I have not played too much with tuning the tenuring for the eden
sizes. Do you
have a stack trace indicating where you failed? OutOfMemory could mean
several
things.
Maxim Berkultsev wrote:
Hi, all!
I'm trying to make some performance evaluations of Geronimo with a help
of
JMeter.
It has appeared relatively simple to get Geronimo out of work. I've
tried to
load it with JMeter and a web primitive called **PingServlet2MDBQueue**
from
Daytrader bundle. I've created immediate load for 10 virtual users and
unlimited number of requests. Within a minute or two Geronimo stopped
responding to any request logging to console something like
...
18:32:56,180 WARN [ThreadedServer] EXCEPTION
java.lang.OutOfMemoryError
18:32:57,211 WARN [ThreadedServer] EXCEPTION
java.lang.OutOfMemoryError
...
Has someone used any specific VM options to run Geronimo smoothly? (As
for
me I've tried starting Geronimo with Java 1.4.2 Hotspot(TM) VM with
-server
option enabled).
Any advice or reference could be helpful. Thank you.
--
Best regards,
Maxim Berkultsev, Intel Middleware Products Division
