Hi Chalitha, It's unlikely to happen that all allocated threads are exhausted. Increasing the value will solve the problem, but we need to find out why APIM takes these much threads. Do you see this continuously?
On Thu, Jun 30, 2016 at 2:29 PM, Chalitha Waldeniyage <[email protected]> wrote: > Hi Abimaran, > > As per Danesh comment, I have increase the max active connection till 120. > Still i am seeing this error printed continuously. Do we need to increase > the value further > > > [2016-06-30 08:58:28,802] ERROR - TransactionManager Failed to start new > registry transaction. > org.apache.tomcat.jdbc.pool.PoolExhaustedException: [pool-234-thread-10] > Timeout: Pool empty. Unable to fetch a connection in 60 seconds, none > available[size:120; busy:120; idle:0; lastwait:60000]. > at > org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674) > at > org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) > at > org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) > at > org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80) > at > org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443) > at > org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:505) > at > org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:177) > at > org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:616) > at > org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:60) > at > org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:596) > at > org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:593) > at java.security.AccessController.doPrivileged(Native Method) > at > org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:593) > at > org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:209) > at > org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > [2016-06-30 08:58:28,803] ERROR - AsyncIndexer Error while indexing. > org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to start > new registry transaction. > at > org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:97) > at > org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443) > at > org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:505) > at > org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:177) > at > org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:616) > at > org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:60) > at > org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:596) > at > org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:593) > at java.security.AccessController.doPrivileged(Native Method) > at > org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:593) > at > org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:209) > at > org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: > [pool-234-thread-10] Timeout: Pool empty. Unable to fetch a connection in 60 > seconds, none available[size:120; busy:120; idle:0; lastwait:60000]. > at > org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674) > at > org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) > at > org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) > at > org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80) > ... 16 more > > > > > On Thu, Jun 30, 2016 at 2:06 PM, Abimaran Kugathasan <[email protected]> > wrote: >> >> Hi Chalitha, >> >> Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: >> [pool-216-thread-7] Timeout: Pool empty. Unable to fetch a connection in 60 >> seconds, none available[size:50; busy:50; idle:0; lastwait:60000]. >> >> From the above logs, the problem lies with the database connection which >> has pool size of 50. So it should be carbon db, because for other dbsm you >> are having 100 as the pool size. You need to increase it as well. >> >> On Thu, Jun 30, 2016 at 1:46 PM, Chalitha Waldeniyage <[email protected]> >> wrote: >>> >>> Hi Danesh, >>> >>> Yes, We have done that on Registry,AM,UM,Conf,Gov databases up to 100 >>> which are pointed to oracle database. But for the carbon db we didn't change >>> it since it's local h2. >>> Do we need to change it also? >>> >>> On Thu, Jun 30, 2016 at 1:34 PM, Danesh Kuruppu <[email protected]> wrote: >>>> >>>> Hi Chalitha, >>>> >>>> Can you increase the maxActive in Registry DB configuration. Seems like >>>> there are no available connections in tomcat jdbc pool. Please set this to >>>> 80 at least. >>>> >>>> Thanks >>>> Danesh >>>> >>>> On Thu, Jun 30, 2016 at 8:29 AM, Chalitha Waldeniyage >>>> <[email protected]> wrote: >>>>> >>>>> Hi All, >>>>> >>>>> We have setup an API Manager cluster (beta 15th pack) fronted by nginx >>>>> with following details. >>>>> >>>>> Publisher >>>>> Two store nodes >>>>> Gateway Cluster(one manager with two workers) >>>>> IS as key manager(two node cluster) >>>>> DB-Oracle 12c >>>>> Userstore-Read/write ldap >>>>> >>>>> We have ran a long running test for some days and observed out of >>>>> memory on one of the IS nodes. Once we have cleaned all created heap >>>>> dumps,logs and try to restart the cluster again and found the following >>>>> error continuously on publisher, gateway and IS nodes. >>>>> >>>>> I have attached the current database running process details with [1]. >>>>> Further we have used the [2] for fine tunning the servers. >>>>> >>>>> Appreciate your feedback on resolving this. >>>>> >>>>> >>>>> >>>>> ERROR - TransactionManager Failed to start new registry transaction. >>>>> org.apache.tomcat.jdbc.pool.PoolExhaustedException: [pool-216-thread-7] >>>>> Timeout: Pool empty. Unable to fetch a connection in 60 seconds, none >>>>> available[size:50; busy:50; idle:0; lastwait:60000]. >>>>> at >>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674) >>>>> at >>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) >>>>> at >>>>> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) >>>>> at >>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80) >>>>> at >>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443) >>>>> at >>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.getTags(EmbeddedRegistry.java:1532) >>>>> at >>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.getTags(CacheBackedRegistry.java:403) >>>>> at >>>>> org.wso2.carbon.registry.core.session.UserRegistry.getTagsInternal(UserRegistry.java:1398) >>>>> at >>>>> org.wso2.carbon.registry.core.session.UserRegistry.access$2500(UserRegistry.java:60) >>>>> at >>>>> org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1381) >>>>> at >>>>> org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1378) >>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>> at >>>>> org.wso2.carbon.registry.core.session.UserRegistry.getTags(UserRegistry.java:1378) >>>>> at >>>>> org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.addTags(IndexDocumentCreator.java:201) >>>>> at >>>>> org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.createIndexDocument(IndexDocumentCreator.java:117) >>>>> at >>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:212) >>>>> at >>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196) >>>>> at >>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> [2016-06-30 06:28:48,855] ERROR - AsyncIndexer Error while indexing. >>>>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to >>>>> start new registry transaction. >>>>> at >>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:97) >>>>> at >>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443) >>>>> at >>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.getTags(EmbeddedRegistry.java:1532) >>>>> at >>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.getTags(CacheBackedRegistry.java:403) >>>>> at >>>>> org.wso2.carbon.registry.core.session.UserRegistry.getTagsInternal(UserRegistry.java:1398) >>>>> at >>>>> org.wso2.carbon.registry.core.session.UserRegistry.access$2500(UserRegistry.java:60) >>>>> at >>>>> org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1381) >>>>> at >>>>> org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1378) >>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>> at >>>>> org.wso2.carbon.registry.core.session.UserRegistry.getTags(UserRegistry.java:1378) >>>>> at >>>>> org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.addTags(IndexDocumentCreator.java:201) >>>>> at >>>>> org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.createIndexDocument(IndexDocumentCreator.java:117) >>>>> at >>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:212) >>>>> at >>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196) >>>>> at >>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: >>>>> [pool-216-thread-7] Timeout: Pool empty. Unable to fetch a connection in >>>>> 60 >>>>> seconds, none available[size:50; busy:50; idle:0; lastwait:60000]. >>>>> at >>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674) >>>>> at >>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) >>>>> at >>>>> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) >>>>> at >>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80) >>>>> ... 18 more >>>>> >>>>> >>>>> [1] >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> [2] https://docs.wso2.com/display/AM1100/Tuning+Performance >>>>> >>>>> >>>>> Thank you, >>>>> -- >>>>> Chalitha Maheshwari >>>>> Software Engineer-QA, >>>>> WSO2 Inc. >>>>> >>>>> E-mail: [email protected] >>>>> Mobile: +94710 411 112 >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Dev mailing list >>>>> [email protected] >>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Danesh Kuruppu >>>> >>>> Software Engineer >>>> WSO2 Inc, >>>> Mobile: +94 (77) 1690552 >>> >>> >>> >>> >>> -- >>> Chalitha Maheshwari >>> Software Engineer-QA, >>> WSO2 Inc. >>> >>> E-mail: [email protected] >>> Mobile: +94710 411 112 >>> >>> _______________________________________________ >>> Dev mailing list >>> [email protected] >>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>> >> >> >> >> -- >> Thanks >> Abimaran Kugathasan >> Senior Software Engineer >> >> Email : [email protected] >> Mobile : +94 773922820 >> >> >> > > > > -- > Chalitha Maheshwari > Software Engineer-QA, > WSO2 Inc. > > E-mail: [email protected] > Mobile: +94710 411 112 -- Thanks Abimaran Kugathasan Senior Software Engineer Email : [email protected] Mobile : +94 773922820 _______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
