Hi, I have updated the [1] with all information.
[1] https://wso2.org/jira/browse/APIMANAGER-5160 On Thu, Jun 30, 2016 at 8:02 PM, Isabelle Mauny <[email protected]> wrote: > Increasing the pool only makes it fail later.. The connections are not > freed apparently. Who holds them ? Can't you see that in JMX console ? > > > ------------------------------------------------------------------------------------- > *Isabelle Mauny* > VP, Product Strategy - WSO2, Inc. - http://wso2.com/ > > > On Thu, Jun 30, 2016 at 11:12 AM, Chandana Napagoda <[email protected]> > wrote: > >> Hi Chalitha, >> >> You can disable the indexing in the APIM GW and KM nodes by setting >> "startIndexing" parameter as false. This "startIndexing" parameter should >> be configured in the registry.xml file under "indexingConfiguration" >> section. >> >> Anyway, it's better to check the DB query execution time by using >> log4jdbc driver. >> >> Regards, >> Chandana >> >> On Thu, Jun 30, 2016 at 2:29 PM, Chalitha Waldeniyage <[email protected]> >> wrote: >> >>> Hi Abimaran, >>> >>> As per Danesh comment, I have increase the max active connection till >>> 120. Still i am seeing this error printed continuously. Do we need to >>> increase the value further >>> >>> >>> [2016-06-30 08:58:28,802] ERROR - TransactionManager Failed to start new >>> registry transaction. >>> org.apache.tomcat.jdbc.pool.PoolExhaustedException: [pool-234-thread-10] >>> Timeout: Pool empty. Unable to fetch a connection in 60 seconds, none >>> available[size:120; busy:120; idle:0; lastwait:60000]. >>> at >>> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674) >>> at >>> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) >>> at >>> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) >>> at >>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80) >>> at >>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443) >>> at >>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:505) >>> at >>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:177) >>> at >>> org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:616) >>> at >>> org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:60) >>> at >>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:596) >>> at >>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:593) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at >>> org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:593) >>> at >>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:209) >>> at >>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196) >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>> at java.lang.Thread.run(Thread.java:745) >>> [2016-06-30 08:58:28,803] ERROR - AsyncIndexer Error while indexing. >>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to >>> start new registry transaction. >>> at >>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:97) >>> at >>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443) >>> at >>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:505) >>> at >>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:177) >>> at >>> org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:616) >>> at >>> org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:60) >>> at >>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:596) >>> at >>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:593) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at >>> org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:593) >>> at >>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:209) >>> at >>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196) >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>> at java.lang.Thread.run(Thread.java:745) >>> Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: >>> [pool-234-thread-10] Timeout: Pool empty. Unable to fetch a connection in >>> 60 seconds, none available[size:120; busy:120; idle:0; lastwait:60000]. >>> at >>> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674) >>> at >>> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) >>> at >>> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) >>> at >>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80) >>> ... 16 more >>> >>> >>> >>> >>> On Thu, Jun 30, 2016 at 2:06 PM, Abimaran Kugathasan <[email protected]> >>> wrote: >>> >>>> Hi Chalitha, >>>> >>>> Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: >>>> [pool-216-thread-7] Timeout: Pool empty. Unable to fetch a connection in 60 >>>> seconds, none available[size:50; busy:50; idle:0; lastwait:60000]. >>>> >>>> From the above logs, the problem lies with the database connection >>>> which has pool size of 50. So it should be carbon db, because for other >>>> dbsm you are having 100 as the pool size. You need to increase it as well. >>>> >>>> On Thu, Jun 30, 2016 at 1:46 PM, Chalitha Waldeniyage < >>>> [email protected]> wrote: >>>> >>>>> Hi Danesh, >>>>> >>>>> Yes, We have done that on Registry,AM,UM,Conf,Gov databases up to 100 >>>>> which are pointed to oracle database. But for the carbon db we didn't >>>>> change it since it's local h2. >>>>> Do we need to change it also? >>>>> >>>>> On Thu, Jun 30, 2016 at 1:34 PM, Danesh Kuruppu <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Chalitha, >>>>>> >>>>>> Can you increase the maxActive in Registry DB configuration. Seems >>>>>> like there are no available connections in tomcat jdbc pool. Please set >>>>>> this to 80 at least. >>>>>> >>>>>> Thanks >>>>>> Danesh >>>>>> >>>>>> On Thu, Jun 30, 2016 at 8:29 AM, Chalitha Waldeniyage < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> We have setup an API Manager cluster (beta 15th pack) fronted by >>>>>>> nginx with following details. >>>>>>> >>>>>>> Publisher >>>>>>> Two store nodes >>>>>>> Gateway Cluster(one manager with two workers) >>>>>>> IS as key manager(two node cluster) >>>>>>> DB-Oracle 12c >>>>>>> Userstore-Read/write ldap >>>>>>> >>>>>>> We have ran a long running test for some days and observed out of >>>>>>> memory on one of the IS nodes. Once we have cleaned all created heap >>>>>>> dumps,logs and try to restart the cluster again and found the following >>>>>>> error continuously on publisher, gateway and IS nodes. >>>>>>> >>>>>>> I have attached the current database running process details with >>>>>>> [1]. Further we have used the [2] for fine tunning the servers. >>>>>>> >>>>>>> Appreciate your feedback on resolving this. >>>>>>> >>>>>>> >>>>>>> >>>>>>> ERROR - TransactionManager Failed to start new registry transaction. >>>>>>> org.apache.tomcat.jdbc.pool.PoolExhaustedException: >>>>>>> [pool-216-thread-7] Timeout: Pool empty. Unable to fetch a connection >>>>>>> in 60 >>>>>>> seconds, none available[size:50; busy:50; idle:0; lastwait:60000]. >>>>>>> at >>>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674) >>>>>>> at >>>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) >>>>>>> at >>>>>>> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) >>>>>>> at >>>>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80) >>>>>>> at >>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443) >>>>>>> at >>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.getTags(EmbeddedRegistry.java:1532) >>>>>>> at >>>>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.getTags(CacheBackedRegistry.java:403) >>>>>>> at >>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getTagsInternal(UserRegistry.java:1398) >>>>>>> at >>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.access$2500(UserRegistry.java:60) >>>>>>> at >>>>>>> org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1381) >>>>>>> at >>>>>>> org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1378) >>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>> at >>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getTags(UserRegistry.java:1378) >>>>>>> at >>>>>>> org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.addTags(IndexDocumentCreator.java:201) >>>>>>> at >>>>>>> org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.createIndexDocument(IndexDocumentCreator.java:117) >>>>>>> at >>>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:212) >>>>>>> at >>>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196) >>>>>>> at >>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>>>> at >>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>>>>> at >>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>> [2016-06-30 06:28:48,855] ERROR - AsyncIndexer Error while indexing. >>>>>>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed >>>>>>> to start new registry transaction. >>>>>>> at >>>>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:97) >>>>>>> at >>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443) >>>>>>> at >>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.getTags(EmbeddedRegistry.java:1532) >>>>>>> at >>>>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.getTags(CacheBackedRegistry.java:403) >>>>>>> at >>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getTagsInternal(UserRegistry.java:1398) >>>>>>> at >>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.access$2500(UserRegistry.java:60) >>>>>>> at >>>>>>> org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1381) >>>>>>> at >>>>>>> org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1378) >>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>> at >>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getTags(UserRegistry.java:1378) >>>>>>> at >>>>>>> org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.addTags(IndexDocumentCreator.java:201) >>>>>>> at >>>>>>> org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.createIndexDocument(IndexDocumentCreator.java:117) >>>>>>> at >>>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:212) >>>>>>> at >>>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196) >>>>>>> at >>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>>>> at >>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>>>>> at >>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>> Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: >>>>>>> [pool-216-thread-7] Timeout: Pool empty. Unable to fetch a connection >>>>>>> in 60 >>>>>>> seconds, none available[size:50; busy:50; idle:0; lastwait:60000]. >>>>>>> at >>>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674) >>>>>>> at >>>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) >>>>>>> at >>>>>>> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) >>>>>>> at >>>>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80) >>>>>>> ... 18 more >>>>>>> >>>>>>> >>>>>>> [1] >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> [2] https://docs.wso2.com/display/AM1100/Tuning+Performance >>>>>>> >>>>>>> >>>>>>> Thank you, >>>>>>> -- >>>>>>> *Chalitha Maheshwari* >>>>>>> Software Engineer-QA, >>>>>>> WSO2 Inc. >>>>>>> >>>>>>> *E-mail:* [email protected] >>>>>>> *Mobile: *+94710 411 112 >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Dev mailing list >>>>>>> [email protected] >>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Danesh Kuruppu >>>>>> Software Engineer >>>>>> WSO2 Inc, >>>>>> Mobile: +94 (77) 1690552 >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Chalitha Maheshwari* >>>>> Software Engineer-QA, >>>>> WSO2 Inc. >>>>> >>>>> *E-mail:* [email protected] >>>>> *Mobile: *+94710 411 112 >>>>> >>>>> _______________________________________________ >>>>> Dev mailing list >>>>> [email protected] >>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>> >>>>> >>>> >>>> >>>> -- >>>> Thanks >>>> Abimaran Kugathasan >>>> Senior Software Engineer >>>> >>>> Email : [email protected] >>>> Mobile : +94 773922820 >>>> >>>> <http://stackoverflow.com/users/515034> >>>> <http://lk.linkedin.com/in/abimaran> >>>> <http://www.lkabimaran.blogspot.com/> <https://github.com/abimarank> >>>> <https://twitter.com/abimaran> >>>> >>>> >>> >>> >>> -- >>> *Chalitha Maheshwari* >>> Software Engineer-QA, >>> WSO2 Inc. >>> >>> *E-mail:* [email protected] >>> *Mobile: *+94710 411 112 >>> >>> _______________________________________________ >>> Dev mailing list >>> [email protected] >>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>> >>> >> >> >> -- >> *Chandana Napagoda* >> Senior Software Engineer >> WSO2 Inc. - http://wso2.org >> >> *Email : [email protected] <[email protected]>**Mobile : +94718169299 >> <%2B94718169299>* >> >> *Blog : http://cnapagoda.blogspot.com <http://cnapagoda.blogspot.com>* >> >> >> _______________________________________________ >> Dev mailing list >> [email protected] >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > -- *Chalitha Maheshwari* Software Engineer-QA, WSO2 Inc. *E-mail:* [email protected] *Mobile: *+94710 411 112
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
