Bugs item #796177, was opened at 2003-08-27 19:00 Message generated for change (Comment added) made by zds0 You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=376685&aid=796177&group_id=22866
Category: JBossCMP Group: v3.2 Status: Open Resolution: None Priority: 5 Submitted By: Jari Juslin (zds0) Assigned to: Nobody/Anonymous (nobody) Summary: JBoss failes to remove lock on CMP beans w/o transaction Initial Comment: JBoss 3.2.1, Red Had Linux 6.2, Sun JVM 1.4.2 for Linux I have EJB 2.0 CMP entity bean, that was accessed both with and without transaction. When two threads performed operations on it's fields at the same time, it sometimes ended up throwing exception like the one below. The wort part is that JBoss was not able to recover from this: subsequent calls to that bean always threw the same exception over and over again. This effectively put our system out of operation, since no retry could get through and bean clients were stuck. We now added transactions to all the beans participating to these operations, but this is still something that should be corrected. AFAIK the EJB 2.0 spec strongly recommends to always use transactions with CMP entity beans, and lets engines not to support accessing CMP beans w/o transaction, but if JBoss chooses to do so it should not let user deploy beans it can't handle correctly. 2003-08-24 03:54:38,689 WARN [org.jboss.tm.TransactionImpl] Transaction TransactionImpl:XidImpl [FormatId=257, GlobalId=www.matchem.com//540079, BranchQual=] timed out. status=STATUS_ACTIVE 2003-08-24 03:54:38,690 WARN [org.jboss.tm.TransactionImpl] Lock contention, tx=TransactionImpl:XidImpl [FormatId=257, GlobalId=www.matchem.com//540079, BranchQual=] 2003-08-24 03:54:38,691 ERROR [org.jboss.ejb.BeanLock] PoolThread-10Saw rolled back tx=TransactionImpl:XidImpl [FormatId=257, GlobalId=www.matchem.com//540079, BranchQual=] waiting for txLock 2003-08-24 03:54:38,691 ERROR [org.jboss.ejb.BeanLock] removing bean lock and it has tx set! QPL bean=MITVCharacter id=[.158285.] tx=TransactionImpl:XidImpl [FormatId=257, GlobalId=www.matchem.com//540019, BranchQual=] synched=null timeout=5000 queue=[] 2003-08-24 03:54:38,692 ERROR [org.jboss.ejb.plugins.LogInterceptor] TransactionRolledbackLocalException, causedBy: java.lang.IllegalStateException: removing bean lock and it has tx set! at org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock.removeRef(QueuedPessimisticEJBLock.java:430) at org.jboss.ejb.BeanLockManager.removeLockRef(BeanLockManager.java:107) at org.jboss.ejb.plugins.EntityLockInterceptor.invoke(EntityLockInterceptor.java:106) at org.jboss.ejb.plugins.EntityCreationInterceptor.invoke(EntityCreationInterceptor.java:53) at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:84) at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:273) at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:104) at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:117) at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:191) at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122) at org.jboss.ejb.EntityContainer.internalInvoke(EntityContainer.java:483) at org.jboss.ejb.Container.invoke(Container.java:674) at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:353) at org.jboss.ejb.plugins.local.EntityProxy.invoke(EntityProxy.java:38) at $Proxy170.getRealName(Unknown Source) at com.matchem.mitv.common.tvchat.TVChatShowComponentBean.getNewChatEntries(TVChatShowComponentBean.java:273) at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at org.jboss.ejb.StatefulSessionContainer$ContainerInterceptor.invoke(StatefulSessionContainer.java:878) at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:117) at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:186) at org.jboss.ejb.plugins.StatefulSessionInstanceInterceptor.invoke(StatefulSessionInstanceInterceptor.java:271) at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:84) at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:243) at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:104) at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:191) at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122) at org.jboss.ejb.StatefulSessionContainer.internalInvoke(StatefulSessionContainer.java:410) at org.jboss.ejb.Container.invoke(Container.java:674) at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:549) at org.jboss.invocation.http.server.HttpInvoker.invoke(HttpInvoker.java:163) at sun.reflect.GeneratedMethodAccessor62.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:549) at org.jboss.invocation.http.servlet.InvokerServlet.processRequest(InvokerServlet.java:132) at org.jboss.invocation.http.servlet.InvokerServlet.doPost(InvokerServlet.java:171) at javax.servlet.http.HttpServlet.service(HttpServlet.java:760) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:360) at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:294) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:558) at org.mortbay.http.HttpContext.handle(HttpContext.java:1714) at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:507) at org.mortbay.http.HttpContext.handle(HttpContext.java:1664) at org.mortbay.http.HttpServer.service(HttpServer.java:863) at org.jboss.jetty.Jetty.service(Jetty.java:460) at org.mortbay.http.HttpConnection.service(HttpConnection.java:775) at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:939) at org.mortbay.http.HttpConnection.handle(HttpConnection.java:792) at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:201) at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:289) at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:455) ---------------------------------------------------------------------- >Comment By: Jari Juslin (zds0) Date: 2003-09-10 23:02 Message: Logged In: YES user_id=808897 Ok, now I managed to get this occur with CMP entity bean that *has* transactions. Below is pieces of log I think track what happens. JBoss 3.2.1, MySQL 4.0.14, Sun JVM 1.4.2-b28, Debian/unstable. All beans are EJB 2.0 CMP entity beans. The problem is with CMP entity beans Operator and SMSCShortCode. They have transaction "Required" for all methods and CMR relationship, Operator one, SMSCShortCode many. We call remotely SMSCShortCode.getOperatorIndex, which just gets Operator by CMR call and returns it's primary key field, "index", with CMP getter. Now, when I stress-test the system, first there throws just two SQLExceptions from entity bean finders. Here is the latter one: javax.ejb.FinderException: Find failed: org.jboss.util.NestedSQLException: No ManagedConnections Available!; - nested throwable: (javax.resource.ResourceException: No ManagedConnections Available!) at org.jboss.ejb.plugins.cmp.jdbc.JDBCAbstractQueryCommand.execute(JDBCAbstractQueryCommand.java:194) at org.jboss.ejb.plugins.cmp.jdbc.JDBCAbstractQueryCommand.execute(JDBCAbstractQueryCommand.java:91) at org.jboss.ejb.plugins.cmp.jdbc.JDBCFindEntityCommand.execute(JDBCFindEntityCommand.java:44) at org.jboss.ejb.plugins.cmp.jdbc.JDBCStoreManager.findEntity(JDBCStoreManager.java:591) at org.jboss.ejb.plugins.CMPPersistenceManager.findEntity(CMPPersistenceManager.java:312) at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.findEntity(CachedConnectionInterceptor.java:301) at org.jboss.ejb.EntityContainer.find(EntityContainer.java:685) at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at org.jboss.ejb.EntityContainer$ContainerInterceptor.invokeHome(EntityContainer.java:998) at org.jboss.ejb.plugins.AbstractInterceptor.invokeHome(AbstractInterceptor.java:88) at org.jboss.ejb.plugins.EntitySynchronizationInterceptor.invokeHome(EntitySynchronizationInterceptor.java:188) at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invokeHome(CachedConnectionInterceptor.java:215) at org.jboss.ejb.plugins.AbstractInterceptor.invokeHome(AbstractInterceptor.java:88) at org.jboss.ejb.plugins.EntityInstanceInterceptor.invokeHome(EntityInstanceInterceptor.java:91) at org.jboss.ejb.plugins.EntityLockInterceptor.invokeHome(EntityLockInterceptor.java:61) at org.jboss.ejb.plugins.EntityCreationInterceptor.invokeHome(EntityCreationInterceptor.java:28) at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:88) at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:243) at org.jboss.ejb.plugins.TxInterceptorCMT.invokeHome(TxInterceptorCMT.java:74) at org.jboss.ejb.plugins.SecurityInterceptor.invokeHome(SecurityInterceptor.java:92) at org.jboss.ejb.plugins.LogInterceptor.invokeHome(LogInterceptor.java:120) at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invokeHome(ProxyFactoryFinderInterceptor.java:93) at org.jboss.ejb.EntityContainer.internalInvokeHome(EntityContainer.java:477) at org.jboss.ejb.Container.invoke(Container.java:694) at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:549) at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:101) at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:83) at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:46) at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:45) at org.jboss.proxy.ejb.HomeInterceptor.invoke(HomeInterceptor.java:173) at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:85) at $Proxy358.findByPrimaryKey(Unknown Source) at com.matchem.mitv.sms.servicemanager.CustomerOperatorSolver.getOperator(CustomerOperatorSolver.java:730) at com.matchem.mitv.sms.servicemanager.CustomerOperatorSolver.checkInCustomer(CustomerOperatorSolver.java:193) at com.matchem.mitv.sms.servicemanager.ServiceManagerServlet.handleHTTPRequest(ServiceManagerServlet.java:663) at com.matchem.mitv.sms.servicemanager.ServiceManagerServlet.doGet(ServiceManagerServlet.java:526) at javax.servlet.http.HttpServlet.service(HttpServlet.java:740) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:360) at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:294) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:558) at org.mortbay.http.HttpContext.handle(HttpContext.java:1714) at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:507) at org.mortbay.http.HttpContext.handle(HttpContext.java:1664) at org.mortbay.http.HttpServer.service(HttpServer.java:863) at org.jboss.jetty.Jetty.service(Jetty.java:460) at org.mortbay.http.HttpConnection.service(HttpConnection.java:775) at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:939) at org.mortbay.http.HttpConnection.handle(HttpConnection.java:792) at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:201) at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:289) at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:455) Then, next exception is more obscure. I see no re-entrancy here (as described abode, getOperatorIndex() is called from web application, and it has just one line: return getOperator().getIndex(); where getOperator() is CMR bound and getIndex() is CMP PK field. Note that these exceptions (the previous and all the later ones) are thrown when accessing _the same row_ in database. So, I suppose that here JBoss has started to fall into illegal state: java.rmi.RemoteException: Reentrant method call detected: SMSCShortCode [.9.] at org.jboss.ejb.plugins.EntityReentranceInterceptor.invoke(EntityReentranceInterceptor.java:67) at org.jboss.ejb.plugins.EntityInstanceInterceptor.invoke(EntityInstanceInterceptor.java:174) at org.jboss.ejb.plugins.EntityLockInterceptor.invoke(EntityLockInterceptor.java:89) at org.jboss.ejb.plugins.EntityCreationInterceptor.invoke(EntityCreationInterceptor.java:53) at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:84) at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:243) at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:104) at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:117) at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:191) at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122) at org.jboss.ejb.EntityContainer.internalInvoke(EntityContainer.java:483) at org.jboss.ejb.Container.invoke(Container.java:674) at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:549) at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:101) at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:83) at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:46) at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:45) at org.jboss.proxy.ejb.EntityInterceptor.invoke(EntityInterceptor.java:97) at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:85) at $Proxy387.getOperatorIndex(Unknown Source) at com.matchem.mitv.sms.servicemanager.CustomerOperatorSolver.getOperator(CustomerOperatorSolver.java:728) at com.matchem.mitv.sms.servicemanager.CustomerOperatorSolver.checkInCustomer(CustomerOperatorSolver.java:193) at com.matchem.mitv.sms.servicemanager.ServiceManagerServlet.handleHTTPRequest(ServiceManagerServlet.java:663) at com.matchem.mitv.sms.servicemanager.ServiceManagerServlet.doGet(ServiceManagerServlet.java:526) at javax.servlet.http.HttpServlet.service(HttpServlet.java:740) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:360) at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:294) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:558) at org.mortbay.http.HttpContext.handle(HttpContext.java:1714) at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:507) at org.mortbay.http.HttpContext.handle(HttpContext.java:1664) at org.mortbay.http.HttpServer.service(HttpServer.java:863) at org.jboss.jetty.Jetty.service(Jetty.java:460) at org.mortbay.http.HttpConnection.service(HttpConnection.java:775) at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:939) at org.mortbay.http.HttpConnection.handle(HttpConnection.java:792) at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:201) at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:289) at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:455) java.rmi.ServerException: RuntimeException; nested exception is: java.lang.IllegalStateException: removing bean lock and it has tx set! at org.jboss.ejb.plugins.LogInterceptor.handleException(LogInterceptor.java:360) at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:195) at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122) at org.jboss.ejb.EntityContainer.internalInvoke(EntityContainer.java:483) at org.jboss.ejb.Container.invoke(Container.java:674) at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:549) at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:101) at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:83) at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:46) at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:45) at org.jboss.proxy.ejb.EntityInterceptor.invoke(EntityInterceptor.java:97) at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:85) at $Proxy387.getOperatorIndex(Unknown Source) at com.matchem.mitv.sms.servicemanager.CustomerOperatorSolver.getOperator(CustomerOperatorSolver.java:728) at com.matchem.mitv.sms.servicemanager.CustomerOperatorSolver.checkInCustomer(CustomerOperatorSolver.java:193) at com.matchem.mitv.sms.servicemanager.ServiceManagerServlet.handleHTTPRequest(ServiceManagerServlet.java:663) at com.matchem.mitv.sms.servicemanager.ServiceManagerServlet.doGet(ServiceManagerServlet.java:526) at javax.servlet.http.HttpServlet.service(HttpServlet.java:740) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:360) at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:294) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:558) at org.mortbay.http.HttpContext.handle(HttpContext.java:1714) at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:507) at org.mortbay.http.HttpContext.handle(HttpContext.java:1664) at org.mortbay.http.HttpServer.service(HttpServer.java:863) at org.jboss.jetty.Jetty.service(Jetty.java:460) at org.mortbay.http.HttpConnection.service(HttpConnection.java:775) at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:939) at org.mortbay.http.HttpConnection.handle(HttpConnection.java:792) at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:201) at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:289) at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:455) Caused by: java.lang.IllegalStateException: removing bean lock and it has tx set! at org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock.removeRef(QueuedPessimisticEJBLock.java:430) at org.jboss.ejb.BeanLockManager.removeLockRef(BeanLockManager.java:107) at org.jboss.ejb.plugins.EntityLockInterceptor.invoke(EntityLockInterceptor.java:106) at org.jboss.ejb.plugins.EntityCreationInterceptor.invoke(EntityCreationInterceptor.java:53) at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:84) at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:243) at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:104) at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:117) at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:191) ... 35 more And this exception is then repeated hundreds of times. Every access to that same row throws the same exception after this point. So, what happens: 1) There is some problem in accessing DB, possibly overload and SQLEception is thrown. 2) Next call to same row throws re-entrancy-exception. 3) JBoss starts to throw "tx set" exception from every access to that row. Re-deploying the beans solves the problem temporarily. Any advice is appreciated, since this bug is so serious, that I can't take this kind of code into production.. Adding some session bean there, that does the entity bean calls could help, but it's not a simple task and thus I'd rather not do such moves blindly. Also I can make this bug reoccur very easily, so if you have some ideas of how to make JBoss output more information about what it does internally on these occasions, please tell me what and how. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=376685&aid=796177&group_id=22866 ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ JBoss-Development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development