Hi Chalitha, Can you increase the maxActive in Registry DB configuration. Seems like there are no available connections in tomcat jdbc pool. Please set this to 80 at least.
Thanks Danesh On Thu, Jun 30, 2016 at 8:29 AM, Chalitha Waldeniyage <[email protected]> wrote: > Hi All, > > We have setup an API Manager cluster (beta 15th pack) fronted by nginx > with following details. > > Publisher > Two store nodes > Gateway Cluster(one manager with two workers) > IS as key manager(two node cluster) > DB-Oracle 12c > Userstore-Read/write ldap > > We have ran a long running test for some days and observed out of memory > on one of the IS nodes. Once we have cleaned all created heap dumps,logs > and try to restart the cluster again and found the following error > continuously on publisher, gateway and IS nodes. > > I have attached the current database running process details with [1]. > Further we have used the [2] for fine tunning the servers. > > Appreciate your feedback on resolving this. > > > > ERROR - TransactionManager Failed to start new registry transaction. > org.apache.tomcat.jdbc.pool.PoolExhaustedException: [pool-216-thread-7] > Timeout: Pool empty. Unable to fetch a connection in 60 seconds, none > available[size:50; busy:50; idle:0; lastwait:60000]. > at > org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674) > at > org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) > at > org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) > at > org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80) > at > org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443) > at > org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.getTags(EmbeddedRegistry.java:1532) > at > org.wso2.carbon.registry.core.caching.CacheBackedRegistry.getTags(CacheBackedRegistry.java:403) > at > org.wso2.carbon.registry.core.session.UserRegistry.getTagsInternal(UserRegistry.java:1398) > at > org.wso2.carbon.registry.core.session.UserRegistry.access$2500(UserRegistry.java:60) > at > org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1381) > at > org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1378) > at java.security.AccessController.doPrivileged(Native Method) > at > org.wso2.carbon.registry.core.session.UserRegistry.getTags(UserRegistry.java:1378) > at > org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.addTags(IndexDocumentCreator.java:201) > at > org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.createIndexDocument(IndexDocumentCreator.java:117) > at > org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:212) > at > org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > [2016-06-30 06:28:48,855] ERROR - AsyncIndexer Error while indexing. > org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to > start new registry transaction. > at > org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:97) > at > org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443) > at > org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.getTags(EmbeddedRegistry.java:1532) > at > org.wso2.carbon.registry.core.caching.CacheBackedRegistry.getTags(CacheBackedRegistry.java:403) > at > org.wso2.carbon.registry.core.session.UserRegistry.getTagsInternal(UserRegistry.java:1398) > at > org.wso2.carbon.registry.core.session.UserRegistry.access$2500(UserRegistry.java:60) > at > org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1381) > at > org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1378) > at java.security.AccessController.doPrivileged(Native Method) > at > org.wso2.carbon.registry.core.session.UserRegistry.getTags(UserRegistry.java:1378) > at > org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.addTags(IndexDocumentCreator.java:201) > at > org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.createIndexDocument(IndexDocumentCreator.java:117) > at > org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:212) > at > org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: > [pool-216-thread-7] Timeout: Pool empty. Unable to fetch a connection in 60 > seconds, none available[size:50; busy:50; idle:0; lastwait:60000]. > at > org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674) > at > org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) > at > org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) > at > org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80) > ... 18 more > > > [1] > > > > > > > [2] https://docs.wso2.com/display/AM1100/Tuning+Performance > > > Thank you, > -- > *Chalitha Maheshwari* > Software Engineer-QA, > WSO2 Inc. > > *E-mail:* [email protected] > *Mobile: *+94710 411 112 > > > > _______________________________________________ > Dev mailing list > [email protected] > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- Danesh Kuruppu Software Engineer WSO2 Inc, Mobile: +94 (77) 1690552
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
