You're getting it for the H2 db right? How can it have so many connections? Have you validated the datasources file to ensure the other datasources (APIM, Gov, UM) point to different DBs?
On Thu, Jun 30, 2016 at 4:32 PM, Chalitha Waldeniyage <[email protected]> wrote: > Hi All, > > I have set the <startIndexing> property as false in Gateway and Key > manager components. Now i am unable to see the above mentioned error in key > manager and gateway nodes. > But it's continuously printing in publisher and Store nodes > > On Thu, Jun 30, 2016 at 3:16 PM, Chalitha Waldeniyage <[email protected]> > wrote: > >> Hi All, >> >> As i mentioned above, our cluster setup is running on 15th Beta pack. In >> that pack, we were unable find "startIndexing" property in registry.xml. >> But it's available on 17th pack. >> As per the offline chat with chandana, It will be o.k put that property >> and check. We will let you know whether issue is resolving with this >> remedy. >> >> On Thu, Jun 30, 2016 at 3:02 PM, Nuwan Dias <[email protected]> wrote: >> >>> I updated the 2.0.0 tuning doc and mentioned to disable indexing on >>> Gateways and Key Managers. >>> >>> On Thu, Jun 30, 2016 at 2:42 PM, Chandana Napagoda <[email protected]> >>> wrote: >>> >>>> Hi Chalitha, >>>> >>>> You can disable the indexing in the APIM GW and KM nodes by setting >>>> "startIndexing" parameter as false. This "startIndexing" parameter should >>>> be configured in the registry.xml file under "indexingConfiguration" >>>> section. >>>> >>>> Anyway, it's better to check the DB query execution time by using >>>> log4jdbc driver. >>>> >>>> Regards, >>>> Chandana >>>> >>>> On Thu, Jun 30, 2016 at 2:29 PM, Chalitha Waldeniyage < >>>> [email protected]> wrote: >>>> >>>>> Hi Abimaran, >>>>> >>>>> As per Danesh comment, I have increase the max active connection till >>>>> 120. Still i am seeing this error printed continuously. Do we need to >>>>> increase the value further >>>>> >>>>> >>>>> [2016-06-30 08:58:28,802] ERROR - TransactionManager Failed to start >>>>> new registry transaction. >>>>> org.apache.tomcat.jdbc.pool.PoolExhaustedException: >>>>> [pool-234-thread-10] Timeout: Pool empty. Unable to fetch a connection in >>>>> 60 seconds, none available[size:120; busy:120; idle:0; lastwait:60000]. >>>>> at >>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674) >>>>> at >>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) >>>>> at >>>>> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) >>>>> at >>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80) >>>>> at >>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443) >>>>> at >>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:505) >>>>> at >>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:177) >>>>> at >>>>> org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:616) >>>>> at >>>>> org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:60) >>>>> at >>>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:596) >>>>> at >>>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:593) >>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>> at >>>>> org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:593) >>>>> at >>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:209) >>>>> at >>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196) >>>>> at >>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> [2016-06-30 08:58:28,803] ERROR - AsyncIndexer Error while indexing. >>>>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to >>>>> start new registry transaction. >>>>> at >>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:97) >>>>> at >>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443) >>>>> at >>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:505) >>>>> at >>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:177) >>>>> at >>>>> org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:616) >>>>> at >>>>> org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:60) >>>>> at >>>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:596) >>>>> at >>>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:593) >>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>> at >>>>> org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:593) >>>>> at >>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:209) >>>>> at >>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196) >>>>> at >>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: >>>>> [pool-234-thread-10] Timeout: Pool empty. Unable to fetch a connection in >>>>> 60 seconds, none available[size:120; busy:120; idle:0; lastwait:60000]. >>>>> at >>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674) >>>>> at >>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) >>>>> at >>>>> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) >>>>> at >>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80) >>>>> ... 16 more >>>>> >>>>> >>>>> >>>>> >>>>> On Thu, Jun 30, 2016 at 2:06 PM, Abimaran Kugathasan < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Chalitha, >>>>>> >>>>>> Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: >>>>>> [pool-216-thread-7] Timeout: Pool empty. Unable to fetch a connection in >>>>>> 60 >>>>>> seconds, none available[size:50; busy:50; idle:0; lastwait:60000]. >>>>>> >>>>>> From the above logs, the problem lies with the database connection >>>>>> which has pool size of 50. So it should be carbon db, because for other >>>>>> dbsm you are having 100 as the pool size. You need to increase it as >>>>>> well. >>>>>> >>>>>> On Thu, Jun 30, 2016 at 1:46 PM, Chalitha Waldeniyage < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi Danesh, >>>>>>> >>>>>>> Yes, We have done that on Registry,AM,UM,Conf,Gov databases up to >>>>>>> 100 which are pointed to oracle database. But for the carbon db we >>>>>>> didn't >>>>>>> change it since it's local h2. >>>>>>> Do we need to change it also? >>>>>>> >>>>>>> On Thu, Jun 30, 2016 at 1:34 PM, Danesh Kuruppu <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Chalitha, >>>>>>>> >>>>>>>> Can you increase the maxActive in Registry DB configuration. Seems >>>>>>>> like there are no available connections in tomcat jdbc pool. Please set >>>>>>>> this to 80 at least. >>>>>>>> >>>>>>>> Thanks >>>>>>>> Danesh >>>>>>>> >>>>>>>> On Thu, Jun 30, 2016 at 8:29 AM, Chalitha Waldeniyage < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> We have setup an API Manager cluster (beta 15th pack) fronted by >>>>>>>>> nginx with following details. >>>>>>>>> >>>>>>>>> Publisher >>>>>>>>> Two store nodes >>>>>>>>> Gateway Cluster(one manager with two workers) >>>>>>>>> IS as key manager(two node cluster) >>>>>>>>> DB-Oracle 12c >>>>>>>>> Userstore-Read/write ldap >>>>>>>>> >>>>>>>>> We have ran a long running test for some days and observed out of >>>>>>>>> memory on one of the IS nodes. Once we have cleaned all created heap >>>>>>>>> dumps,logs and try to restart the cluster again and found the >>>>>>>>> following >>>>>>>>> error continuously on publisher, gateway and IS nodes. >>>>>>>>> >>>>>>>>> I have attached the current database running process details with >>>>>>>>> [1]. Further we have used the [2] for fine tunning the servers. >>>>>>>>> >>>>>>>>> Appreciate your feedback on resolving this. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ERROR - TransactionManager Failed to start new registry >>>>>>>>> transaction. >>>>>>>>> org.apache.tomcat.jdbc.pool.PoolExhaustedException: >>>>>>>>> [pool-216-thread-7] Timeout: Pool empty. Unable to fetch a connection >>>>>>>>> in 60 >>>>>>>>> seconds, none available[size:50; busy:50; idle:0; lastwait:60000]. >>>>>>>>> at >>>>>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674) >>>>>>>>> at >>>>>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) >>>>>>>>> at >>>>>>>>> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.getTags(EmbeddedRegistry.java:1532) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.getTags(CacheBackedRegistry.java:403) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getTagsInternal(UserRegistry.java:1398) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.access$2500(UserRegistry.java:60) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1381) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1378) >>>>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getTags(UserRegistry.java:1378) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.addTags(IndexDocumentCreator.java:201) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.createIndexDocument(IndexDocumentCreator.java:117) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:212) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196) >>>>>>>>> at >>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>>>>>> at >>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>>>>>>> at >>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>> [2016-06-30 06:28:48,855] ERROR - AsyncIndexer Error while >>>>>>>>> indexing. >>>>>>>>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed >>>>>>>>> to start new registry transaction. >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:97) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.getTags(EmbeddedRegistry.java:1532) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.getTags(CacheBackedRegistry.java:403) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getTagsInternal(UserRegistry.java:1398) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.access$2500(UserRegistry.java:60) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1381) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1378) >>>>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getTags(UserRegistry.java:1378) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.addTags(IndexDocumentCreator.java:201) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.createIndexDocument(IndexDocumentCreator.java:117) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:212) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196) >>>>>>>>> at >>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>>>>>> at >>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>>>>>>> at >>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>> Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: >>>>>>>>> [pool-216-thread-7] Timeout: Pool empty. Unable to fetch a connection >>>>>>>>> in 60 >>>>>>>>> seconds, none available[size:50; busy:50; idle:0; lastwait:60000]. >>>>>>>>> at >>>>>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674) >>>>>>>>> at >>>>>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) >>>>>>>>> at >>>>>>>>> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80) >>>>>>>>> ... 18 more >>>>>>>>> >>>>>>>>> >>>>>>>>> [1] >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> [2] https://docs.wso2.com/display/AM1100/Tuning+Performance >>>>>>>>> >>>>>>>>> >>>>>>>>> Thank you, >>>>>>>>> -- >>>>>>>>> *Chalitha Maheshwari* >>>>>>>>> Software Engineer-QA, >>>>>>>>> WSO2 Inc. >>>>>>>>> >>>>>>>>> *E-mail:* [email protected] >>>>>>>>> *Mobile: *+94710 411 112 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Dev mailing list >>>>>>>>> [email protected] >>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Danesh Kuruppu >>>>>>>> Software Engineer >>>>>>>> WSO2 Inc, >>>>>>>> Mobile: +94 (77) 1690552 >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Chalitha Maheshwari* >>>>>>> Software Engineer-QA, >>>>>>> WSO2 Inc. >>>>>>> >>>>>>> *E-mail:* [email protected] >>>>>>> *Mobile: *+94710 411 112 >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Dev mailing list >>>>>>> [email protected] >>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Thanks >>>>>> Abimaran Kugathasan >>>>>> Senior Software Engineer >>>>>> >>>>>> Email : [email protected] >>>>>> Mobile : +94 773922820 >>>>>> >>>>>> <http://stackoverflow.com/users/515034> >>>>>> <http://lk.linkedin.com/in/abimaran> >>>>>> <http://www.lkabimaran.blogspot.com/> <https://github.com/abimarank> >>>>>> <https://twitter.com/abimaran> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Chalitha Maheshwari* >>>>> Software Engineer-QA, >>>>> WSO2 Inc. >>>>> >>>>> *E-mail:* [email protected] >>>>> *Mobile: *+94710 411 112 >>>>> >>>>> _______________________________________________ >>>>> Dev mailing list >>>>> [email protected] >>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>> >>>>> >>>> >>>> >>>> -- >>>> *Chandana Napagoda* >>>> Senior Software Engineer >>>> WSO2 Inc. - http://wso2.org >>>> >>>> *Email : [email protected] <[email protected]>**Mobile : >>>> +94718169299 <%2B94718169299>* >>>> >>>> *Blog : http://cnapagoda.blogspot.com >>>> <http://cnapagoda.blogspot.com>* >>>> >>>> >>>> _______________________________________________ >>>> Dev mailing list >>>> [email protected] >>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>> >>>> >>> >>> >>> -- >>> Nuwan Dias >>> >>> Technical Lead - WSO2, Inc. http://wso2.com >>> email : [email protected] >>> Phone : +94 777 775 729 >>> >> >> >> >> -- >> *Chalitha Maheshwari* >> Software Engineer-QA, >> WSO2 Inc. >> >> *E-mail:* [email protected] >> *Mobile: *+94710 411 112 >> > > > > -- > *Chalitha Maheshwari* > Software Engineer-QA, > WSO2 Inc. > > *E-mail:* [email protected] > *Mobile: *+94710 411 112 > -- Nuwan Dias Technical Lead - WSO2, Inc. http://wso2.com email : [email protected] Phone : +94 777 775 729
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
