Re: [JBoss-user] Managed Connection Pool running out of connections
Thanks, yes I figured it out in the end, and actually I've removed transactions altogether for the beans involved because everything is read-only and there's no worries about inconsistent state. I think there's probably a bug in the Connection Manager, so that when WrapperDataSource.getConnection() is being called the exception is thrown but a connection is still reserved from the pool and never gets reclaimed (including if you flush the pool). Looking at the code (and I'm still trying to get my head around it), it looks like TxConnectionEventListener.enlist() is throwing a SystemException because it can't enlist the XAResource with the Tranaction which is rolled back. By this point a managed connection has been got (in BaseConnectionManager2.allocateConnection()), and there doesn't appear to be anything that returns it when enlist throws an error. -Andrew Scott M Stark wrote: Alright, so the issue is that a stateful session is being accessed after it has been thrown out due to its inactivity exceeding its max age value. Either you need to increase the period over which passivated sessions are kept or isolate the access of the stateful session from the encompassing transaction using a RequiresNew tx attribute to prevent the failure from affecting the encompassing tx. --- This SF.Net email sponsored by: Free pre-built ASP.NET sites including Data Reports, E-commerce, Portals, and Forums are available now. Download today and enter to win an XBOX or Visual Studio .NET. http://aspnet.click-url.com/go/psa0013ave/direct;at.aspnet_072303_01/01 ___ JBoss-user mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-user
Re: [JBoss-user] Managed Connection Pool running out of connections
There's an SQLException thrown when calling getConnection() on the Datasource retrieved
from JNDI. This gets wrapped in an application exception (ISISConnectionException) with
the message ISISDatabaseConnection: Unable to create connection to ISIS Database, but
the nested SQLException is being lost when the application exception gets thrown as an
EJBException in BrowseUtilities.getConnection():
try
{
con = ISISDatabaseConnection.getInstance().getConnection();
}
catch (ISISConnectionException e)
{
throw new EJBException(e);
}
which we can change so that the SQLException is at least logged before the EJBException is
thrown.
I've done some further digging through the logs and I think I can see the chain of events
that's causing this:
1) Servlet calls Stateful Session Bean (SFSB) (AnnotationService)
2) AnnotationService calls another SFSB (MarkedList) which has been deleted:
[15:36:49] [SEVERE] [com.ingenta.search.ejb.AnnotationServiceBean:
markedListAnnotation(ArticleList)] MarkedList annotation failed
java.rmi.NoSuchObjectException: Could not activate; failed to restore state;
CausedByException is:
/opt/jboss/jboss-3.2.1/server/isisProduction/tmp/sessions/SessionMarkedList-dieq8nwx-4/difycma2-2244.ser
(No such file or directory)
at org.jboss.ejb.plugins.AbstractInstanceCache.get(AbstractInstanceCache.java:122)
3) Because this error isn't considered critical, the RemoteException is caught and logged
and the AnnotationService continues.
4) AnnotationService calls Stateless SessionBean (ResultsBranding), which in turn calls
another Stateless SessionBean (BrowseController) 20 times (for annotating a collection of
20 objects).
5) Every call to the BrowseController fails because a database connection cannot be got
from the pool, and an EJBException is thrown which gets converted into a
TransactionRolledBackException. Unfortunately the ResultsBranding bean catches these
RemoteExceptions and keeps calling the BrowseController (which has the default transaction
settings - Requires?). Would this create a bunch of seperate transactions?
This points to a couple of things that we need to change in our application - log the
SQLException, and propergate the TransactionRolledBackException. I'll make the changes and
try to recreate the error.
However, it looks like the Datasource error is triggered by the NoSuchObjectException,
which I don't understand. Also, I'm not sure how the call to Datasource.getConnection()
could be taking a connection out of the pool and not returning it (as the connection pool
suddenly has 20 connections less available).
Sorry for the long winded post, they're my speciality,
-Andrew
Scott M Stark wrote:
What is the real problem occuring here at BrowseUtilities.java:214, an
SQLException or what?
Caused by: javax.ejb.EJBException: null; CausedByException is:
ISISDatabaseConnection: Unable to create connection to ISIS
Database.
at
com.ingenta.content.BrowseUtilities.getConnection(BrowseUtilities.java:214)
at
com.ingenta.content.ejb.BrowseControllerBean.getIssue(BrowseControllerBean.java:542)
---
This SF.net email is sponsored by: VM Ware
With VMware you can run multiple operating systems on a single machine.
WITHOUT REBOOTING! Mix Linux / Windows / Novell virtual machines at the
same time. Free trial click here: http://www.vmware.com/wl/offer/345/0
___
JBoss-user mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/jboss-user
Re: [JBoss-user] Managed Connection Pool running out of connections
OK here's the SQLException when trying to get a connection from the datasource after a
NoSuchObjectException has been caught:
org.jboss.util.NestedSQLException: Could not enlist in transaction on entering meta-aware
object!javax.transaction.SystemException: Could not enlist XAResource!
javax.transaction.RollbackException: Already marked for rollback; - nested throwable:
(javax.resource.ResourceException: Could not enlist in transaction on entering meta-aware
object!javax.transaction.SystemException: Could not enlist
XAResource!javax.transaction.RollbackException: Already marked for rollback)
at
org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:106)
I've made the changes so that it's not trying to get 20 connections, but it has made one
connection permenantly unavailable.
-Andrew
Andrew May wrote:
There's an SQLException thrown when calling getConnection() on the
Datasource retrieved from JNDI. This gets wrapped in an application
exception (ISISConnectionException) with the message
ISISDatabaseConnection: Unable to create connection to ISIS Database,
but the nested SQLException is being lost when the application exception
gets thrown as an EJBException in BrowseUtilities.getConnection():
try
{
con = ISISDatabaseConnection.getInstance().getConnection();
}
catch (ISISConnectionException e)
{
throw new EJBException(e);
}
which we can change so that the SQLException is at least logged before
the EJBException is thrown.
I've done some further digging through the logs and I think I can see
the chain of events that's causing this:
1) Servlet calls Stateful Session Bean (SFSB) (AnnotationService)
2) AnnotationService calls another SFSB (MarkedList) which has been
deleted:
[15:36:49] [SEVERE] [com.ingenta.search.ejb.AnnotationServiceBean:
markedListAnnotation(ArticleList)] MarkedList annotation failed
java.rmi.NoSuchObjectException: Could not activate; failed to restore
state; CausedByException is:
/opt/jboss/jboss-3.2.1/server/isisProduction/tmp/sessions/SessionMarkedList-dieq8nwx-4/difycma2-2244.ser
(No such file or directory)
at
org.jboss.ejb.plugins.AbstractInstanceCache.get(AbstractInstanceCache.java:122)
3) Because this error isn't considered critical, the RemoteException is
caught and logged and the AnnotationService continues.
4) AnnotationService calls Stateless SessionBean (ResultsBranding),
which in turn calls another Stateless SessionBean (BrowseController) 20
times (for annotating a collection of 20 objects).
5) Every call to the BrowseController fails because a database
connection cannot be got from the pool, and an EJBException is thrown
which gets converted into a TransactionRolledBackException.
Unfortunately the ResultsBranding bean catches these RemoteExceptions
and keeps calling the BrowseController (which has the default
transaction settings - Requires?). Would this create a bunch of seperate
transactions?
This points to a couple of things that we need to change in our
application - log the SQLException, and propergate the
TransactionRolledBackException. I'll make the changes and try to
recreate the error.
However, it looks like the Datasource error is triggered by the
NoSuchObjectException, which I don't understand. Also, I'm not sure how
the call to Datasource.getConnection() could be taking a connection out
of the pool and not returning it (as the connection pool suddenly has 20
connections less available).
Sorry for the long winded post, they're my speciality,
-Andrew
Scott M Stark wrote:
What is the real problem occuring here at BrowseUtilities.java:214, an
SQLException or what?
Caused by: javax.ejb.EJBException: null; CausedByException is:
ISISDatabaseConnection: Unable to create connection to ISIS
Database.
at
com.ingenta.content.BrowseUtilities.getConnection(BrowseUtilities.java:214)
at
com.ingenta.content.ejb.BrowseControllerBean.getIssue(BrowseControllerBean.java:542)
---
This SF.net email is sponsored by: VM Ware
With VMware you can run multiple operating systems on a single machine.
WITHOUT REBOOTING! Mix Linux / Windows / Novell virtual machines at the
same time. Free trial click here: http://www.vmware.com/wl/offer/345/0
___
JBoss-user mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/jboss-user
---
This SF.net email is sponsored by: VM Ware
With VMware you can run multiple operating systems on a single machine.
WITHOUT REBOOTING! Mix Linux / Windows / Novell virtual machines at the
same time. Free trial click here: http://www.vmware.com/wl/offer/345/0
___
JBoss-user mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/jboss-user
Re: [JBoss-user] Managed Connection Pool running out of connections
Alright, so the issue is that a stateful session is being accessed after it has been thrown out due to its inactivity exceeding its max age value. Either you need to increase the period over which passivated sessions are kept or isolate the access of the stateful session from the encompassing transaction using a RequiresNew tx attribute to prevent the failure from affecting the encompassing tx. -- Scott Stark Chief Technology Officer JBoss Group, LLC Andrew May wrote: OK here's the SQLException when trying to get a connection from the datasource after a NoSuchObjectException has been caught: org.jboss.util.NestedSQLException: Could not enlist in transaction on entering meta-aware object!javax.transaction.SystemException: Could not enlist XAResource! javax.transaction.RollbackException: Already marked for rollback; - nested throwable: (javax.resource.ResourceException: Could not enlist in transaction on entering meta-aware object!javax.transaction.SystemException: Could not enlist XAResource!javax.transaction.RollbackException: Already marked for rollback) at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:106) I've made the changes so that it's not trying to get 20 connections, but it has made one connection permenantly unavailable. -Andrew --- This SF.Net email sponsored by: Free pre-built ASP.NET sites including Data Reports, E-commerce, Portals, and Forums are available now. Download today and enter to win an XBOX or Visual Studio .NET. http://aspnet.click-url.com/go/psa0013ave/direct;at.aspnet_072303_01/01 ___ JBoss-user mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-user
Re: [JBoss-user] Managed Connection Pool running out of connections
What is the real problem occuring here at BrowseUtilities.java:214, an SQLException or what? Caused by: javax.ejb.EJBException: null; CausedByException is: ISISDatabaseConnection: Unable to create connection to ISIS Database. at com.ingenta.content.BrowseUtilities.getConnection(BrowseUtilities.java:214) at com.ingenta.content.ejb.BrowseControllerBean.getIssue(BrowseControllerBean.java:542) -- Scott Stark Chief Technology Officer JBoss Group, LLC Andrew May wrote: Our production application has on several times run out of connections from a pool and we've had to restart the server to reset the pool. We have a pool of 100 Oracle connections used for BMP entity beans and calls to the database from Session beans via Data Access Objects: local-tx-datasource jndi-namejdbc.oraclePool/jndi-name connection-urljdbc:oracle:thin:@machine:1521:schema/connection-url driver-classoracle.jdbc.driver.OracleDriver/driver-class user-nameusername/user-name passwordpassword/password min-pool-size10/min-pool-size max-pool-size100/max-pool-size blocking-timeout-millis5000/blocking-timeout-millis idle-timeout-minutes15/idle-timeout-minutes exception-sorter-class-nameorg.jboss.resource.adapter.jdbc.vendor.OracleExceptionSorter/exception-sorter-class-name /local-tx-datasource We're typically only using between 0 to 20 connections in use at any given time (I'm using EJTools Swing JMX browser to graph the ConnectionCount and AvailableConnectionCount at 10 second intervals), although I suspect there are peaks 30 at times. --- This SF.net email is sponsored by: VM Ware With VMware you can run multiple operating systems on a single machine. WITHOUT REBOOTING! Mix Linux / Windows / Novell virtual machines at the same time. Free trial click here: http://www.vmware.com/wl/offer/345/0 ___ JBoss-user mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-user
