Increasing the pool only makes it fail later.. The connections are not freed apparently. Who holds them ? Can't you see that in JMX console ?
------------------------------------------------------------------------------------- *Isabelle Mauny* VP, Product Strategy - WSO2, Inc. - http://wso2.com/ On Thu, Jun 30, 2016 at 11:12 AM, Chandana Napagoda <[email protected]> wrote: > Hi Chalitha, > > You can disable the indexing in the APIM GW and KM nodes by setting > "startIndexing" parameter as false. This "startIndexing" parameter should > be configured in the registry.xml file under "indexingConfiguration" > section. > > Anyway, it's better to check the DB query execution time by using log4jdbc > driver. > > Regards, > Chandana > > On Thu, Jun 30, 2016 at 2:29 PM, Chalitha Waldeniyage <[email protected]> > wrote: > >> Hi Abimaran, >> >> As per Danesh comment, I have increase the max active connection till >> 120. Still i am seeing this error printed continuously. Do we need to >> increase the value further >> >> >> [2016-06-30 08:58:28,802] ERROR - TransactionManager Failed to start new >> registry transaction. >> org.apache.tomcat.jdbc.pool.PoolExhaustedException: [pool-234-thread-10] >> Timeout: Pool empty. Unable to fetch a connection in 60 seconds, none >> available[size:120; busy:120; idle:0; lastwait:60000]. >> at >> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674) >> at >> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) >> at >> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) >> at >> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80) >> at >> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443) >> at >> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:505) >> at >> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:177) >> at >> org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:616) >> at >> org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:60) >> at >> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:596) >> at >> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:593) >> at java.security.AccessController.doPrivileged(Native Method) >> at >> org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:593) >> at >> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:209) >> at >> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> at java.lang.Thread.run(Thread.java:745) >> [2016-06-30 08:58:28,803] ERROR - AsyncIndexer Error while indexing. >> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to >> start new registry transaction. >> at >> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:97) >> at >> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443) >> at >> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:505) >> at >> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:177) >> at >> org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:616) >> at >> org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:60) >> at >> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:596) >> at >> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:593) >> at java.security.AccessController.doPrivileged(Native Method) >> at >> org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:593) >> at >> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:209) >> at >> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> at java.lang.Thread.run(Thread.java:745) >> Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: >> [pool-234-thread-10] Timeout: Pool empty. Unable to fetch a connection in >> 60 seconds, none available[size:120; busy:120; idle:0; lastwait:60000]. >> at >> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674) >> at >> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) >> at >> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) >> at >> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80) >> ... 16 more >> >> >> >> >> On Thu, Jun 30, 2016 at 2:06 PM, Abimaran Kugathasan <[email protected]> >> wrote: >> >>> Hi Chalitha, >>> >>> Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: >>> [pool-216-thread-7] Timeout: Pool empty. Unable to fetch a connection in 60 >>> seconds, none available[size:50; busy:50; idle:0; lastwait:60000]. >>> >>> From the above logs, the problem lies with the database connection which >>> has pool size of 50. So it should be carbon db, because for other dbsm you >>> are having 100 as the pool size. You need to increase it as well. >>> >>> On Thu, Jun 30, 2016 at 1:46 PM, Chalitha Waldeniyage <[email protected] >>> > wrote: >>> >>>> Hi Danesh, >>>> >>>> Yes, We have done that on Registry,AM,UM,Conf,Gov databases up to 100 >>>> which are pointed to oracle database. But for the carbon db we didn't >>>> change it since it's local h2. >>>> Do we need to change it also? >>>> >>>> On Thu, Jun 30, 2016 at 1:34 PM, Danesh Kuruppu <[email protected]> >>>> wrote: >>>> >>>>> Hi Chalitha, >>>>> >>>>> Can you increase the maxActive in Registry DB configuration. Seems >>>>> like there are no available connections in tomcat jdbc pool. Please set >>>>> this to 80 at least. >>>>> >>>>> Thanks >>>>> Danesh >>>>> >>>>> On Thu, Jun 30, 2016 at 8:29 AM, Chalitha Waldeniyage < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi All, >>>>>> >>>>>> We have setup an API Manager cluster (beta 15th pack) fronted by >>>>>> nginx with following details. >>>>>> >>>>>> Publisher >>>>>> Two store nodes >>>>>> Gateway Cluster(one manager with two workers) >>>>>> IS as key manager(two node cluster) >>>>>> DB-Oracle 12c >>>>>> Userstore-Read/write ldap >>>>>> >>>>>> We have ran a long running test for some days and observed out of >>>>>> memory on one of the IS nodes. Once we have cleaned all created heap >>>>>> dumps,logs and try to restart the cluster again and found the following >>>>>> error continuously on publisher, gateway and IS nodes. >>>>>> >>>>>> I have attached the current database running process details with >>>>>> [1]. Further we have used the [2] for fine tunning the servers. >>>>>> >>>>>> Appreciate your feedback on resolving this. >>>>>> >>>>>> >>>>>> >>>>>> ERROR - TransactionManager Failed to start new registry transaction. >>>>>> org.apache.tomcat.jdbc.pool.PoolExhaustedException: >>>>>> [pool-216-thread-7] Timeout: Pool empty. Unable to fetch a connection in >>>>>> 60 >>>>>> seconds, none available[size:50; busy:50; idle:0; lastwait:60000]. >>>>>> at >>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674) >>>>>> at >>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) >>>>>> at >>>>>> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) >>>>>> at >>>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80) >>>>>> at >>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443) >>>>>> at >>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.getTags(EmbeddedRegistry.java:1532) >>>>>> at >>>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.getTags(CacheBackedRegistry.java:403) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getTagsInternal(UserRegistry.java:1398) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry.access$2500(UserRegistry.java:60) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1381) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1378) >>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getTags(UserRegistry.java:1378) >>>>>> at >>>>>> org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.addTags(IndexDocumentCreator.java:201) >>>>>> at >>>>>> org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.createIndexDocument(IndexDocumentCreator.java:117) >>>>>> at >>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:212) >>>>>> at >>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196) >>>>>> at >>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>> [2016-06-30 06:28:48,855] ERROR - AsyncIndexer Error while indexing. >>>>>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to >>>>>> start new registry transaction. >>>>>> at >>>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:97) >>>>>> at >>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443) >>>>>> at >>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.getTags(EmbeddedRegistry.java:1532) >>>>>> at >>>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.getTags(CacheBackedRegistry.java:403) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getTagsInternal(UserRegistry.java:1398) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry.access$2500(UserRegistry.java:60) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1381) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1378) >>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getTags(UserRegistry.java:1378) >>>>>> at >>>>>> org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.addTags(IndexDocumentCreator.java:201) >>>>>> at >>>>>> org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.createIndexDocument(IndexDocumentCreator.java:117) >>>>>> at >>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:212) >>>>>> at >>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196) >>>>>> at >>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>> Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: >>>>>> [pool-216-thread-7] Timeout: Pool empty. Unable to fetch a connection in >>>>>> 60 >>>>>> seconds, none available[size:50; busy:50; idle:0; lastwait:60000]. >>>>>> at >>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674) >>>>>> at >>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) >>>>>> at >>>>>> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) >>>>>> at >>>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80) >>>>>> ... 18 more >>>>>> >>>>>> >>>>>> [1] >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> [2] https://docs.wso2.com/display/AM1100/Tuning+Performance >>>>>> >>>>>> >>>>>> Thank you, >>>>>> -- >>>>>> *Chalitha Maheshwari* >>>>>> Software Engineer-QA, >>>>>> WSO2 Inc. >>>>>> >>>>>> *E-mail:* [email protected] >>>>>> *Mobile: *+94710 411 112 >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Dev mailing list >>>>>> [email protected] >>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Danesh Kuruppu >>>>> Software Engineer >>>>> WSO2 Inc, >>>>> Mobile: +94 (77) 1690552 >>>>> >>>> >>>> >>>> >>>> -- >>>> *Chalitha Maheshwari* >>>> Software Engineer-QA, >>>> WSO2 Inc. >>>> >>>> *E-mail:* [email protected] >>>> *Mobile: *+94710 411 112 >>>> >>>> _______________________________________________ >>>> Dev mailing list >>>> [email protected] >>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>> >>>> >>> >>> >>> -- >>> Thanks >>> Abimaran Kugathasan >>> Senior Software Engineer >>> >>> Email : [email protected] >>> Mobile : +94 773922820 >>> >>> <http://stackoverflow.com/users/515034> >>> <http://lk.linkedin.com/in/abimaran> >>> <http://www.lkabimaran.blogspot.com/> <https://github.com/abimarank> >>> <https://twitter.com/abimaran> >>> >>> >> >> >> -- >> *Chalitha Maheshwari* >> Software Engineer-QA, >> WSO2 Inc. >> >> *E-mail:* [email protected] >> *Mobile: *+94710 411 112 >> >> _______________________________________________ >> Dev mailing list >> [email protected] >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > *Chandana Napagoda* > Senior Software Engineer > WSO2 Inc. - http://wso2.org > > *Email : [email protected] <[email protected]>**Mobile : +94718169299 > <%2B94718169299>* > > *Blog : http://cnapagoda.blogspot.com <http://cnapagoda.blogspot.com>* > > > _______________________________________________ > Dev mailing list > [email protected] > http://wso2.org/cgi-bin/mailman/listinfo/dev > >
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
