Carbon db is on local H2 all the other databases has pointed to the oracle (APIM,GOV,Conf,UM,MB.THROTTLE)
On Thu, Jun 30, 2016 at 4:35 PM, Nuwan Dias <[email protected]> wrote: > You're getting it for the H2 db right? How can it have so many > connections? Have you validated the datasources file to ensure the other > datasources (APIM, Gov, UM) point to different DBs? > > On Thu, Jun 30, 2016 at 4:32 PM, Chalitha Waldeniyage <[email protected]> > wrote: > >> Hi All, >> >> I have set the <startIndexing> property as false in Gateway and Key >> manager components. Now i am unable to see the above mentioned error in key >> manager and gateway nodes. >> But it's continuously printing in publisher and Store nodes >> >> On Thu, Jun 30, 2016 at 3:16 PM, Chalitha Waldeniyage <[email protected]> >> wrote: >> >>> Hi All, >>> >>> As i mentioned above, our cluster setup is running on 15th Beta pack. In >>> that pack, we were unable find "startIndexing" property in registry.xml. >>> But it's available on 17th pack. >>> As per the offline chat with chandana, It will be o.k put that property >>> and check. We will let you know whether issue is resolving with this >>> remedy. >>> >>> On Thu, Jun 30, 2016 at 3:02 PM, Nuwan Dias <[email protected]> wrote: >>> >>>> I updated the 2.0.0 tuning doc and mentioned to disable indexing on >>>> Gateways and Key Managers. >>>> >>>> On Thu, Jun 30, 2016 at 2:42 PM, Chandana Napagoda <[email protected]> >>>> wrote: >>>> >>>>> Hi Chalitha, >>>>> >>>>> You can disable the indexing in the APIM GW and KM nodes by setting >>>>> "startIndexing" parameter as false. This "startIndexing" parameter should >>>>> be configured in the registry.xml file under "indexingConfiguration" >>>>> section. >>>>> >>>>> Anyway, it's better to check the DB query execution time by using >>>>> log4jdbc driver. >>>>> >>>>> Regards, >>>>> Chandana >>>>> >>>>> On Thu, Jun 30, 2016 at 2:29 PM, Chalitha Waldeniyage < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Abimaran, >>>>>> >>>>>> As per Danesh comment, I have increase the max active connection till >>>>>> 120. Still i am seeing this error printed continuously. Do we need to >>>>>> increase the value further >>>>>> >>>>>> >>>>>> [2016-06-30 08:58:28,802] ERROR - TransactionManager Failed to start >>>>>> new registry transaction. >>>>>> org.apache.tomcat.jdbc.pool.PoolExhaustedException: >>>>>> [pool-234-thread-10] Timeout: Pool empty. Unable to fetch a connection in >>>>>> 60 seconds, none available[size:120; busy:120; idle:0; lastwait:60000]. >>>>>> at >>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674) >>>>>> at >>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) >>>>>> at >>>>>> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) >>>>>> at >>>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80) >>>>>> at >>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443) >>>>>> at >>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:505) >>>>>> at >>>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:177) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:616) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:60) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:596) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:593) >>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:593) >>>>>> at >>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:209) >>>>>> at >>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196) >>>>>> at >>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>> [2016-06-30 08:58:28,803] ERROR - AsyncIndexer Error while indexing. >>>>>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to >>>>>> start new registry transaction. >>>>>> at >>>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:97) >>>>>> at >>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443) >>>>>> at >>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:505) >>>>>> at >>>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:177) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:616) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:60) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:596) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:593) >>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:593) >>>>>> at >>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:209) >>>>>> at >>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196) >>>>>> at >>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>> Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: >>>>>> [pool-234-thread-10] Timeout: Pool empty. Unable to fetch a connection in >>>>>> 60 seconds, none available[size:120; busy:120; idle:0; lastwait:60000]. >>>>>> at >>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674) >>>>>> at >>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) >>>>>> at >>>>>> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) >>>>>> at >>>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80) >>>>>> ... 16 more >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Jun 30, 2016 at 2:06 PM, Abimaran Kugathasan < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi Chalitha, >>>>>>> >>>>>>> Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: >>>>>>> [pool-216-thread-7] Timeout: Pool empty. Unable to fetch a connection >>>>>>> in 60 >>>>>>> seconds, none available[size:50; busy:50; idle:0; lastwait:60000]. >>>>>>> >>>>>>> From the above logs, the problem lies with the database connection >>>>>>> which has pool size of 50. So it should be carbon db, because for other >>>>>>> dbsm you are having 100 as the pool size. You need to increase it as >>>>>>> well. >>>>>>> >>>>>>> On Thu, Jun 30, 2016 at 1:46 PM, Chalitha Waldeniyage < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi Danesh, >>>>>>>> >>>>>>>> Yes, We have done that on Registry,AM,UM,Conf,Gov databases up to >>>>>>>> 100 which are pointed to oracle database. But for the carbon db we >>>>>>>> didn't >>>>>>>> change it since it's local h2. >>>>>>>> Do we need to change it also? >>>>>>>> >>>>>>>> On Thu, Jun 30, 2016 at 1:34 PM, Danesh Kuruppu <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Chalitha, >>>>>>>>> >>>>>>>>> Can you increase the maxActive in Registry DB configuration. Seems >>>>>>>>> like there are no available connections in tomcat jdbc pool. Please >>>>>>>>> set >>>>>>>>> this to 80 at least. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> Danesh >>>>>>>>> >>>>>>>>> On Thu, Jun 30, 2016 at 8:29 AM, Chalitha Waldeniyage < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi All, >>>>>>>>>> >>>>>>>>>> We have setup an API Manager cluster (beta 15th pack) fronted by >>>>>>>>>> nginx with following details. >>>>>>>>>> >>>>>>>>>> Publisher >>>>>>>>>> Two store nodes >>>>>>>>>> Gateway Cluster(one manager with two workers) >>>>>>>>>> IS as key manager(two node cluster) >>>>>>>>>> DB-Oracle 12c >>>>>>>>>> Userstore-Read/write ldap >>>>>>>>>> >>>>>>>>>> We have ran a long running test for some days and observed out of >>>>>>>>>> memory on one of the IS nodes. Once we have cleaned all created heap >>>>>>>>>> dumps,logs and try to restart the cluster again and found the >>>>>>>>>> following >>>>>>>>>> error continuously on publisher, gateway and IS nodes. >>>>>>>>>> >>>>>>>>>> I have attached the current database running process details with >>>>>>>>>> [1]. Further we have used the [2] for fine tunning the servers. >>>>>>>>>> >>>>>>>>>> Appreciate your feedback on resolving this. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ERROR - TransactionManager Failed to start new registry >>>>>>>>>> transaction. >>>>>>>>>> org.apache.tomcat.jdbc.pool.PoolExhaustedException: >>>>>>>>>> [pool-216-thread-7] Timeout: Pool empty. Unable to fetch a >>>>>>>>>> connection in 60 >>>>>>>>>> seconds, none available[size:50; busy:50; idle:0; lastwait:60000]. >>>>>>>>>> at >>>>>>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674) >>>>>>>>>> at >>>>>>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) >>>>>>>>>> at >>>>>>>>>> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.getTags(EmbeddedRegistry.java:1532) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.getTags(CacheBackedRegistry.java:403) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getTagsInternal(UserRegistry.java:1398) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.access$2500(UserRegistry.java:60) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1381) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1378) >>>>>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getTags(UserRegistry.java:1378) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.addTags(IndexDocumentCreator.java:201) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.createIndexDocument(IndexDocumentCreator.java:117) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:212) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196) >>>>>>>>>> at >>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>>>>>>> at >>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>>>>>>>> at >>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>>> [2016-06-30 06:28:48,855] ERROR - AsyncIndexer Error while >>>>>>>>>> indexing. >>>>>>>>>> org.wso2.carbon.registry.core.exceptions.RegistryException: >>>>>>>>>> Failed to start new registry transaction. >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:97) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.getTags(EmbeddedRegistry.java:1532) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.getTags(CacheBackedRegistry.java:403) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getTagsInternal(UserRegistry.java:1398) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.access$2500(UserRegistry.java:60) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1381) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1378) >>>>>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getTags(UserRegistry.java:1378) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.addTags(IndexDocumentCreator.java:201) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.createIndexDocument(IndexDocumentCreator.java:117) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:212) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196) >>>>>>>>>> at >>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>>>>>>> at >>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>>>>>>>> at >>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>>> Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: >>>>>>>>>> [pool-216-thread-7] Timeout: Pool empty. Unable to fetch a >>>>>>>>>> connection in 60 >>>>>>>>>> seconds, none available[size:50; busy:50; idle:0; lastwait:60000]. >>>>>>>>>> at >>>>>>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674) >>>>>>>>>> at >>>>>>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) >>>>>>>>>> at >>>>>>>>>> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80) >>>>>>>>>> ... 18 more >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [1] >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [2] https://docs.wso2.com/display/AM1100/Tuning+Performance >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thank you, >>>>>>>>>> -- >>>>>>>>>> *Chalitha Maheshwari* >>>>>>>>>> Software Engineer-QA, >>>>>>>>>> WSO2 Inc. >>>>>>>>>> >>>>>>>>>> *E-mail:* [email protected] >>>>>>>>>> *Mobile: *+94710 411 112 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Dev mailing list >>>>>>>>>> [email protected] >>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> Danesh Kuruppu >>>>>>>>> Software Engineer >>>>>>>>> WSO2 Inc, >>>>>>>>> Mobile: +94 (77) 1690552 >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *Chalitha Maheshwari* >>>>>>>> Software Engineer-QA, >>>>>>>> WSO2 Inc. >>>>>>>> >>>>>>>> *E-mail:* [email protected] >>>>>>>> *Mobile: *+94710 411 112 >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Dev mailing list >>>>>>>> [email protected] >>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Thanks >>>>>>> Abimaran Kugathasan >>>>>>> Senior Software Engineer >>>>>>> >>>>>>> Email : [email protected] >>>>>>> Mobile : +94 773922820 >>>>>>> >>>>>>> <http://stackoverflow.com/users/515034> >>>>>>> <http://lk.linkedin.com/in/abimaran> >>>>>>> <http://www.lkabimaran.blogspot.com/> >>>>>>> <https://github.com/abimarank> <https://twitter.com/abimaran> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Chalitha Maheshwari* >>>>>> Software Engineer-QA, >>>>>> WSO2 Inc. >>>>>> >>>>>> *E-mail:* [email protected] >>>>>> *Mobile: *+94710 411 112 >>>>>> >>>>>> _______________________________________________ >>>>>> Dev mailing list >>>>>> [email protected] >>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Chandana Napagoda* >>>>> Senior Software Engineer >>>>> WSO2 Inc. - http://wso2.org >>>>> >>>>> *Email : [email protected] <[email protected]>**Mobile : >>>>> +94718169299 <%2B94718169299>* >>>>> >>>>> *Blog : http://cnapagoda.blogspot.com >>>>> <http://cnapagoda.blogspot.com>* >>>>> >>>>> >>>>> _______________________________________________ >>>>> Dev mailing list >>>>> [email protected] >>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>> >>>>> >>>> >>>> >>>> -- >>>> Nuwan Dias >>>> >>>> Technical Lead - WSO2, Inc. http://wso2.com >>>> email : [email protected] >>>> Phone : +94 777 775 729 >>>> >>> >>> >>> >>> -- >>> *Chalitha Maheshwari* >>> Software Engineer-QA, >>> WSO2 Inc. >>> >>> *E-mail:* [email protected] >>> *Mobile: *+94710 411 112 >>> >> >> >> >> -- >> *Chalitha Maheshwari* >> Software Engineer-QA, >> WSO2 Inc. >> >> *E-mail:* [email protected] >> *Mobile: *+94710 411 112 >> > > > > -- > Nuwan Dias > > Technical Lead - WSO2, Inc. http://wso2.com > email : [email protected] > Phone : +94 777 775 729 > -- *Chalitha Maheshwari* Software Engineer-QA, WSO2 Inc. *E-mail:* [email protected] *Mobile: *+94710 411 112
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
