Hi folks,
As always, thanks for your input
Jacopo:
That's interesting, I was wondering the same thing, and seems to make some
sense based on the dump. It should also have been noted that the network
which I am operating on is a bit volatile, so whether the DB server
(PostgreSQL in my case) may be closing the connections, or perhaps some
other external interference (e.g. connection drop), is probably quite
possible. In some of my non-OFBiz apps, I set a hard connection age so that
the pool is reacquiring connections often, even if the connections have not
been idle for an extended period of time. I've studied the javadocs in
regards to DBCP 1.4 and the commons GenericObjectPool which is used in the
DBCP connection factory, but have not seen any similar option (in 1.4) for
setting a connection age such that a connection will be reacquired when that
time expires, however this seems to be available via the
maxConnLifetimeMillis setting in the new versions.
In my 12.04.02 copy of DBCPConnectionFactory the only options being set on
the pool (which are pulled from entityengine.xml) are:
- timeBetweenEvictionRunsMillis
- maxActive
- maxIdle
- minIdle
- maxWait
I suppose I could manually set the "testOnBorrow" flag to support or rule
out whether this may be the issue (not sure how much of a performance hit
having this on would be, if it turns out to alleviate the problem), but
based on your knowledge of the framework and DBCP configurations, do you
have any other suggestions on what configurations could be set to further
confirm that a connection "gone bad" may be the cause?
I've attached the pertinent threaddump content below
Mike Z:
I'm running JDK 6 update 45 (64-bit). Upon launch of OFBiz, the only setting
being set is setting maximum JVM memory (i.e.
pos.memory.max.param=-Xmx4096M).
Partial Threaddump:
"ajp-bio-0.0.0.0-8009-exec-38" daemon prio=10 tid=0x000000005c11f000
nid=0x561a waiting for monitor entry [0x00002b0ab9959000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:781)
- waiting to lock <0x0000000700f02318> (a
org.apache.commons.pool.impl.GenericObjectPool)
at
org.apache.commons.dbcp.managed.ManagedConnection.updateTransactionStatus(ManagedConnection.java:117)
at
org.apache.commons.dbcp.managed.ManagedConnection.<init>(ManagedConnection.java:55)
at
org.apache.commons.dbcp.managed.ManagedDataSource.getConnection(ManagedDataSource.java:77)
at
org.ofbiz.entity.connection.DBCPConnectionFactory.getConnection(DBCPConnectionFactory.java:66)
at
org.ofbiz.entity.jdbc.ConnectionFactory.getManagedConnection(ConnectionFactory.java:135)
at
org.ofbiz.geronimo.GeronimoTransactionFactory.getConnection(GeronimoTransactionFactory.java:83)
at
org.ofbiz.entity.transaction.TransactionFactory.getConnection(TransactionFactory.java:97)
at
org.ofbiz.entity.jdbc.ConnectionFactory.getConnection(ConnectionFactory.java:85)
at
org.ofbiz.entity.jdbc.SQLProcessor.getConnection(SQLProcessor.java:250)
at
org.ofbiz.entity.jdbc.SQLProcessor.prepareStatement(SQLProcessor.java:356)
at
org.ofbiz.entity.jdbc.SQLProcessor.prepareStatement(SQLProcessor.java:340)
at org.ofbiz.entity.datasource.GenericDAO.select(GenericDAO.java:532)
at org.ofbiz.entity.datasource.GenericDAO.select(GenericDAO.java:503)
at
org.ofbiz.entity.datasource.GenericHelperDAO.findByPrimaryKey(GenericHelperDAO.java:85)
at org.ofbiz.entity.GenericDelegator.findOne(GenericDelegator.java:1572)
at org.ofbiz.entity.GenericDelegator.findOne(GenericDelegator.java:1538)
at org.ofbiz.webapp.stats.VisitHandler.getVisitor(VisitHandler.java:245)
- locked <0x00000007b1206a68> (a
org.apache.catalina.session.StandardSessionFacade)
at
org.ofbiz.webapp.control.LoginWorker.setWebContextObjects(LoginWorker.java:509)
at
org.ofbiz.webapp.control.LoginWorker.doBasicLogout(LoginWorker.java:639)
at org.ofbiz.webapp.control.LoginWorker.logout(LoginWorker.java:585)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.ofbiz.webapp.event.JavaEventHandler.invoke(JavaEventHandler.java:93)
at
org.ofbiz.webapp.event.JavaEventHandler.invoke(JavaEventHandler.java:79)
at
org.ofbiz.webapp.control.RequestHandler.runEvent(RequestHandler.java:660)
at
org.ofbiz.webapp.control.RequestHandler.doRequest(RequestHandler.java:406)
at
org.ofbiz.webapp.control.ControlServlet.doGet(ControlServlet.java:224)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.ofbiz.webapp.control.ContextFilter.doFilter(ContextFilter.java:337)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:200)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
- locked <0x00000007c1735640> (a
org.apache.tomcat.util.net.SocketWrapper)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- <0x00000007c17356c0> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
"ajp-bio-0.0.0.0-8009-exec-37" daemon prio=10 tid=0x000000005cda8000
nid=0x55e4 runnable [0x00002b0ab9858000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at
org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:143)
at
org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:112)
at
org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:71)
at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:269)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1704)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
- locked <0x00000007aab17b88> (a
org.postgresql.core.v3.QueryExecutorImpl)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:403)
at
org.postgresql.jdbc2.AbstractJdbc2Connection.execSQLQuery(AbstractJdbc2Connection.java:357)
at
org.postgresql.jdbc2.AbstractJdbc2Connection.execSQLQuery(AbstractJdbc2Connection.java:349)
at
org.postgresql.jdbc2.AbstractJdbc2Connection.getTransactionIsolation(AbstractJdbc2Connection.java:883)
at
org.apache.commons.dbcp.DelegatingConnection.getTransactionIsolation(DelegatingConnection.java:353)
at
org.apache.commons.dbcp.PoolableConnectionFactory.activateObject(PoolableConnectionFactory.java:706)
at
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:854)
- locked <0x0000000700f02318> (a
org.apache.commons.pool.impl.GenericObjectPool)
at
org.apache.commons.dbcp.managed.ManagedConnection.updateTransactionStatus(ManagedConnection.java:117)
at
org.apache.commons.dbcp.managed.ManagedConnection.<init>(ManagedConnection.java:55)
at
org.apache.commons.dbcp.managed.ManagedDataSource.getConnection(ManagedDataSource.java:77)
at
org.ofbiz.entity.connection.DBCPConnectionFactory.getConnection(DBCPConnectionFactory.java:66)
at
org.ofbiz.entity.jdbc.ConnectionFactory.getManagedConnection(ConnectionFactory.java:135)
at
org.ofbiz.geronimo.GeronimoTransactionFactory.getConnection(GeronimoTransactionFactory.java:83)
at
org.ofbiz.entity.transaction.TransactionFactory.getConnection(TransactionFactory.java:97)
at
org.ofbiz.entity.jdbc.ConnectionFactory.getConnection(ConnectionFactory.java:85)
at
org.ofbiz.entity.jdbc.SQLProcessor.getConnection(SQLProcessor.java:250)
at
org.ofbiz.entity.jdbc.SQLProcessor.prepareStatement(SQLProcessor.java:356)
at
org.ofbiz.entity.jdbc.SQLProcessor.prepareStatement(SQLProcessor.java:340)
at org.ofbiz.entity.datasource.GenericDAO.select(GenericDAO.java:532)
at org.ofbiz.entity.datasource.GenericDAO.select(GenericDAO.java:503)
at
org.ofbiz.entity.datasource.GenericHelperDAO.findByPrimaryKey(GenericHelperDAO.java:85)
at org.ofbiz.entity.GenericDelegator.findOne(GenericDelegator.java:1572)
at
org.ofbiz.webapp.control.ControlEventListener.sessionDestroyed(ControlEventListener.java:80)
at
org.apache.catalina.session.StandardSession.expire(StandardSession.java:806)
- locked <0x00000007b1206010> (a
org.apache.catalina.session.StandardSession)
at
org.apache.catalina.session.StandardSession.expire(StandardSession.java:742)
at
org.apache.catalina.session.StandardSession.invalidate(StandardSession.java:1253)
at
org.apache.catalina.session.StandardSessionFacade.invalidate(StandardSessionFacade.java:190)
at
org.ofbiz.webapp.control.LoginWorker.doBasicLogout(LoginWorker.java:623)
at org.ofbiz.webapp.control.LoginWorker.logout(LoginWorker.java:585)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.ofbiz.webapp.event.JavaEventHandler.invoke(JavaEventHandler.java:93)
at
org.ofbiz.webapp.event.JavaEventHandler.invoke(JavaEventHandler.java:79)
at
org.ofbiz.webapp.control.RequestHandler.runEvent(RequestHandler.java:660)
at
org.ofbiz.webapp.control.RequestHandler.doRequest(RequestHandler.java:406)
at
org.ofbiz.webapp.control.ControlServlet.doGet(ControlServlet.java:224)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.ofbiz.webapp.control.ContextFilter.doFilter(ContextFilter.java:337)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:200)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
- locked <0x00000007ab063978> (a
org.apache.tomcat.util.net.SocketWrapper)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- <0x00000007ab0639f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
Thanks!
Darrell
--
View this message in context:
http://ofbiz.135035.n4.nabble.com/Tomcat-Unresponsive-tp4659005p4659280.html
Sent from the OFBiz - User mailing list archive at Nabble.com.
