We have a pool of 100 Oracle connections used for BMP entity beans and calls to the database from Session beans via Data Access Objects:
<local-tx-datasource>
<jndi-name>jdbc.oraclePool</jndi-name>
<connection-url>jdbc:oracle:thin:@machine:1521:schema</connection-url>
<driver-class>oracle.jdbc.driver.OracleDriver</driver-class>
<user-name>username</user-name>
<password>password</password>
<min-pool-size>10</min-pool-size>
<max-pool-size>100</max-pool-size>
<blocking-timeout-millis>5000</blocking-timeout-millis>
<idle-timeout-minutes>15</idle-timeout-minutes>
<exception-sorter-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleExceptionSorter</exception-sorter-class-name>
</local-tx-datasource>We're typically only using between 0 to 20 connections in use at any given time (I'm using EJTools Swing JMX browser to graph the ConnectionCount and AvailableConnectionCount at 10 second intervals), although I suspect there are peaks > 30 at times.
After a while, and usually when the system is under load we start seeing bunches of connections becoming permenantly unavailable - i.e the AvailableConnectionCount will go from having 80-100 connections available to having 60-80 connections available. This means that the ConnectionCount will go up accordingly. Eventually this happens several times and then there are no available connections. When this has happened I've tried flushing the pool and even stopping and starting it, but this doesn't help.
It's always a bunch of connections that become unavailable at the same time. From the Oracle side of things the connections are still open but not being used (by looking at when a statement was last executed), and there are no uncommited transactions.
I've done thread dumps and there's nothing obvious there. So I suspect that for some reason the connections are not getting properly returned to the pool, or the pool is not recognising that the connections are free to be used.
Looking in our application logs for this afternoon when the server last "lost" a bunch of connections I can see quite a few stack traces like this (sorry for the long stack trace):
javax.transaction.TransactionRolledbackException: null; CausedByException is:
ISISDatabaseConnection: Unable to create connection to ISIS Database.; nested exception is:
javax.ejb.EJBException: null; CausedByException is:
ISISDatabaseConnection: Unable to create connection to ISIS Database.
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:214)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:243)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:104)
at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:117)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:191)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
at org.jboss.ejb.StatelessSessionContainer.internalInvoke(StatelessSessionContainer.java:322)
at org.jboss.ejb.Container.invoke(Container.java:674)
at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:549)
at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:101)
at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:83)
at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:46)
at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:45)
at org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:100)
at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:85)
at $Proxy165.getIssue(Unknown Source)
at com.ingenta.search.ejb.ResultsBrandingBean.annotateArticle(ResultsBrandingBean.java:141)
at com.ingenta.search.ejb.ResultsBrandingBean.annotateArticleList(ResultsBrandingBean.java:197)
at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:629)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:186)
at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:72)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:84)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:243)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:104)
at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:117)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:191)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
at org.jboss.ejb.StatelessSessionContainer.internalInvoke(StatelessSessionContainer.java:322)
at org.jboss.ejb.Container.invoke(Container.java:674)
at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:549)
at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:101)
at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:83)
at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:46)
at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:45)
at org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:100)
at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:85)
at $Proxy238.annotateArticleList(Unknown Source)
at com.ingenta.search.ejb.AnnotationServiceBean.brandingAnnotation(AnnotationServiceBean.java:257)
at com.ingenta.search.ejb.AnnotationServiceBean.annotate(AnnotationServiceBean.java:184)
at sun.reflect.GeneratedMethodAccessor108.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jboss.ejb.StatefulSessionContainer$ContainerInterceptor.invoke(StatefulSessionContainer.java:878)
at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:117)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:186)
at org.jboss.ejb.plugins.StatefulSessionInstanceInterceptor.invoke(StatefulSessionInstanceInterceptor.java:271)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:84)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:243)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:104)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:191)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
at org.jboss.ejb.StatefulSessionContainer.internalInvoke(StatefulSessionContainer.java:410)
at org.jboss.ejb.Container.invoke(Container.java:674)
at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:549)
at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:101)
at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:83)
at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:46)
at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:45)
at org.jboss.proxy.ejb.StatefulSessionInterceptor.invoke(StatefulSessionInterceptor.java:104)
at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:85)
at $Proxy276.annotate(Unknown Source)
at com.ingenta.search.handler.SearchHandler.handleAuthenticated(SearchHandler.java:272)
at com.ingenta.authentication.handler.AuthenticationHandler.handle(AuthenticationHandler.java:244)
at com.ingenta.servlet.Controller.dispatch(Controller.java:211)
at com.ingenta.servlet.Controller.doGet(Controller.java:121)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:356)
at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:294)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:558)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1723)
at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:514)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1673)
at org.mortbay.http.HttpServer.service(HttpServer.java:879)
at org.jboss.jetty.Jetty.service(Jetty.java:460)
at org.mortbay.http.HttpConnection.service(HttpConnection.java:783)
at org.mortbay.http.ajp.AJP13Connection.handleNext(AJP13Connection.java:273)
at org.mortbay.http.HttpConnection.handle(HttpConnection.java:800)
at org.mortbay.http.ajp.AJP13Listener.handleConnection(AJP13Listener.java:204)
at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:289)
at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:454)
Caused by: javax.ejb.EJBException: null; CausedByException is:
ISISDatabaseConnection: Unable to create connection to ISIS Database.
at com.ingenta.content.BrowseUtilities.getConnection(BrowseUtilities.java:214)
at com.ingenta.content.ejb.BrowseControllerBean.getIssue(BrowseControllerBean.java:542)
at sun.reflect.GeneratedMethodAccessor94.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:629)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:186)
at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:72)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:84)
... 95 more
Although according the the graph of the counts in JMX there were still at least 70 connections available. After a few seconds the application recovered.
I've not been able to re-create the problem with load testing. We've got the source code and are willing to poke around, but we're not sure where to start looking. Any suggestions?
Thanks,
Andrew
------------------------------------------------------- This SF.net email is sponsored by: VM Ware With VMware you can run multiple operating systems on a single machine. WITHOUT REBOOTING! Mix Linux / Windows / Novell virtual machines at the same time. Free trial click here: http://www.vmware.com/wl/offer/345/0 _______________________________________________ JBoss-user mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-user
