Hi All, I have set the <startIndexing> property as false in Gateway and Key manager components. Now i am unable to see the above mentioned error in key manager and gateway nodes. But it's continuously printing in publisher and Store nodes
On Thu, Jun 30, 2016 at 3:16 PM, Chalitha Waldeniyage <[email protected]> wrote: > Hi All, > > As i mentioned above, our cluster setup is running on 15th Beta pack. In > that pack, we were unable find "startIndexing" property in registry.xml. > But it's available on 17th pack. > As per the offline chat with chandana, It will be o.k put that property > and check. We will let you know whether issue is resolving with this > remedy. > > On Thu, Jun 30, 2016 at 3:02 PM, Nuwan Dias <[email protected]> wrote: > >> I updated the 2.0.0 tuning doc and mentioned to disable indexing on >> Gateways and Key Managers. >> >> On Thu, Jun 30, 2016 at 2:42 PM, Chandana Napagoda <[email protected]> >> wrote: >> >>> Hi Chalitha, >>> >>> You can disable the indexing in the APIM GW and KM nodes by setting >>> "startIndexing" parameter as false. This "startIndexing" parameter should >>> be configured in the registry.xml file under "indexingConfiguration" >>> section. >>> >>> Anyway, it's better to check the DB query execution time by using >>> log4jdbc driver. >>> >>> Regards, >>> Chandana >>> >>> On Thu, Jun 30, 2016 at 2:29 PM, Chalitha Waldeniyage <[email protected] >>> > wrote: >>> >>>> Hi Abimaran, >>>> >>>> As per Danesh comment, I have increase the max active connection till >>>> 120. Still i am seeing this error printed continuously. Do we need to >>>> increase the value further >>>> >>>> >>>> [2016-06-30 08:58:28,802] ERROR - TransactionManager Failed to start >>>> new registry transaction. >>>> org.apache.tomcat.jdbc.pool.PoolExhaustedException: >>>> [pool-234-thread-10] Timeout: Pool empty. Unable to fetch a connection in >>>> 60 seconds, none available[size:120; busy:120; idle:0; lastwait:60000]. >>>> at >>>> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674) >>>> at >>>> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) >>>> at >>>> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) >>>> at >>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80) >>>> at >>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443) >>>> at >>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:505) >>>> at >>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:177) >>>> at >>>> org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:616) >>>> at >>>> org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:60) >>>> at >>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:596) >>>> at >>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:593) >>>> at java.security.AccessController.doPrivileged(Native Method) >>>> at >>>> org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:593) >>>> at >>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:209) >>>> at >>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196) >>>> at >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>> at java.lang.Thread.run(Thread.java:745) >>>> [2016-06-30 08:58:28,803] ERROR - AsyncIndexer Error while indexing. >>>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to >>>> start new registry transaction. >>>> at >>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:97) >>>> at >>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443) >>>> at >>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:505) >>>> at >>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:177) >>>> at >>>> org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:616) >>>> at >>>> org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:60) >>>> at >>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:596) >>>> at >>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:593) >>>> at java.security.AccessController.doPrivileged(Native Method) >>>> at >>>> org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:593) >>>> at >>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:209) >>>> at >>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196) >>>> at >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>> at java.lang.Thread.run(Thread.java:745) >>>> Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: >>>> [pool-234-thread-10] Timeout: Pool empty. Unable to fetch a connection in >>>> 60 seconds, none available[size:120; busy:120; idle:0; lastwait:60000]. >>>> at >>>> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674) >>>> at >>>> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) >>>> at >>>> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) >>>> at >>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80) >>>> ... 16 more >>>> >>>> >>>> >>>> >>>> On Thu, Jun 30, 2016 at 2:06 PM, Abimaran Kugathasan <[email protected] >>>> > wrote: >>>> >>>>> Hi Chalitha, >>>>> >>>>> Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: >>>>> [pool-216-thread-7] Timeout: Pool empty. Unable to fetch a connection in >>>>> 60 >>>>> seconds, none available[size:50; busy:50; idle:0; lastwait:60000]. >>>>> >>>>> From the above logs, the problem lies with the database connection >>>>> which has pool size of 50. So it should be carbon db, because for other >>>>> dbsm you are having 100 as the pool size. You need to increase it as well. >>>>> >>>>> On Thu, Jun 30, 2016 at 1:46 PM, Chalitha Waldeniyage < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Danesh, >>>>>> >>>>>> Yes, We have done that on Registry,AM,UM,Conf,Gov databases up to 100 >>>>>> which are pointed to oracle database. But for the carbon db we didn't >>>>>> change it since it's local h2. >>>>>> Do we need to change it also? >>>>>> >>>>>> On Thu, Jun 30, 2016 at 1:34 PM, Danesh Kuruppu <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi Chalitha, >>>>>>> >>>>>>> Can you increase the maxActive in Registry DB configuration. Seems >>>>>>> like there are no available connections in tomcat jdbc pool. Please set >>>>>>> this to 80 at least. >>>>>>> >>>>>>> Thanks >>>>>>> Danesh >>>>>>> >>>>>>> On Thu, Jun 30, 2016 at 8:29 AM, Chalitha Waldeniyage < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> We have setup an API Manager cluster (beta 15th pack) fronted by >>>>>>>> nginx with following details. >>>>>>>> >>>>>>>> Publisher >>>>>>>> Two store nodes >>>>>>>> Gateway Cluster(one manager with two workers) >>>>>>>> IS as key manager(two node cluster) >>>>>>>> DB-Oracle 12c >>>>>>>> Userstore-Read/write ldap >>>>>>>> >>>>>>>> We have ran a long running test for some days and observed out of >>>>>>>> memory on one of the IS nodes. Once we have cleaned all created heap >>>>>>>> dumps,logs and try to restart the cluster again and found the following >>>>>>>> error continuously on publisher, gateway and IS nodes. >>>>>>>> >>>>>>>> I have attached the current database running process details with >>>>>>>> [1]. Further we have used the [2] for fine tunning the servers. >>>>>>>> >>>>>>>> Appreciate your feedback on resolving this. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ERROR - TransactionManager Failed to start new registry transaction. >>>>>>>> org.apache.tomcat.jdbc.pool.PoolExhaustedException: >>>>>>>> [pool-216-thread-7] Timeout: Pool empty. Unable to fetch a connection >>>>>>>> in 60 >>>>>>>> seconds, none available[size:50; busy:50; idle:0; lastwait:60000]. >>>>>>>> at >>>>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674) >>>>>>>> at >>>>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) >>>>>>>> at >>>>>>>> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) >>>>>>>> at >>>>>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80) >>>>>>>> at >>>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443) >>>>>>>> at >>>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.getTags(EmbeddedRegistry.java:1532) >>>>>>>> at >>>>>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.getTags(CacheBackedRegistry.java:403) >>>>>>>> at >>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getTagsInternal(UserRegistry.java:1398) >>>>>>>> at >>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.access$2500(UserRegistry.java:60) >>>>>>>> at >>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1381) >>>>>>>> at >>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1378) >>>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>>> at >>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getTags(UserRegistry.java:1378) >>>>>>>> at >>>>>>>> org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.addTags(IndexDocumentCreator.java:201) >>>>>>>> at >>>>>>>> org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.createIndexDocument(IndexDocumentCreator.java:117) >>>>>>>> at >>>>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:212) >>>>>>>> at >>>>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196) >>>>>>>> at >>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>>>>> at >>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>>>>>> at >>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>> [2016-06-30 06:28:48,855] ERROR - AsyncIndexer Error while indexing. >>>>>>>> org.wso2.carbon.registry.core.exceptions.RegistryException: Failed >>>>>>>> to start new registry transaction. >>>>>>>> at >>>>>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:97) >>>>>>>> at >>>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443) >>>>>>>> at >>>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.getTags(EmbeddedRegistry.java:1532) >>>>>>>> at >>>>>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.getTags(CacheBackedRegistry.java:403) >>>>>>>> at >>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getTagsInternal(UserRegistry.java:1398) >>>>>>>> at >>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.access$2500(UserRegistry.java:60) >>>>>>>> at >>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1381) >>>>>>>> at >>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry$26.run(UserRegistry.java:1378) >>>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>>> at >>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getTags(UserRegistry.java:1378) >>>>>>>> at >>>>>>>> org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.addTags(IndexDocumentCreator.java:201) >>>>>>>> at >>>>>>>> org.wso2.carbon.registry.indexing.indexer.IndexDocumentCreator.createIndexDocument(IndexDocumentCreator.java:117) >>>>>>>> at >>>>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:212) >>>>>>>> at >>>>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:196) >>>>>>>> at >>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>>>>> at >>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>>>>>> at >>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>> Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: >>>>>>>> [pool-216-thread-7] Timeout: Pool empty. Unable to fetch a connection >>>>>>>> in 60 >>>>>>>> seconds, none available[size:50; busy:50; idle:0; lastwait:60000]. >>>>>>>> at >>>>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674) >>>>>>>> at >>>>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) >>>>>>>> at >>>>>>>> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) >>>>>>>> at >>>>>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80) >>>>>>>> ... 18 more >>>>>>>> >>>>>>>> >>>>>>>> [1] >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> [2] https://docs.wso2.com/display/AM1100/Tuning+Performance >>>>>>>> >>>>>>>> >>>>>>>> Thank you, >>>>>>>> -- >>>>>>>> *Chalitha Maheshwari* >>>>>>>> Software Engineer-QA, >>>>>>>> WSO2 Inc. >>>>>>>> >>>>>>>> *E-mail:* [email protected] >>>>>>>> *Mobile: *+94710 411 112 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Dev mailing list >>>>>>>> [email protected] >>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Danesh Kuruppu >>>>>>> Software Engineer >>>>>>> WSO2 Inc, >>>>>>> Mobile: +94 (77) 1690552 >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Chalitha Maheshwari* >>>>>> Software Engineer-QA, >>>>>> WSO2 Inc. >>>>>> >>>>>> *E-mail:* [email protected] >>>>>> *Mobile: *+94710 411 112 >>>>>> >>>>>> _______________________________________________ >>>>>> Dev mailing list >>>>>> [email protected] >>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks >>>>> Abimaran Kugathasan >>>>> Senior Software Engineer >>>>> >>>>> Email : [email protected] >>>>> Mobile : +94 773922820 >>>>> >>>>> <http://stackoverflow.com/users/515034> >>>>> <http://lk.linkedin.com/in/abimaran> >>>>> <http://www.lkabimaran.blogspot.com/> <https://github.com/abimarank> >>>>> <https://twitter.com/abimaran> >>>>> >>>>> >>>> >>>> >>>> -- >>>> *Chalitha Maheshwari* >>>> Software Engineer-QA, >>>> WSO2 Inc. >>>> >>>> *E-mail:* [email protected] >>>> *Mobile: *+94710 411 112 >>>> >>>> _______________________________________________ >>>> Dev mailing list >>>> [email protected] >>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>> >>>> >>> >>> >>> -- >>> *Chandana Napagoda* >>> Senior Software Engineer >>> WSO2 Inc. - http://wso2.org >>> >>> *Email : [email protected] <[email protected]>**Mobile : +94718169299 >>> <%2B94718169299>* >>> >>> *Blog : http://cnapagoda.blogspot.com >>> <http://cnapagoda.blogspot.com>* >>> >>> >>> _______________________________________________ >>> Dev mailing list >>> [email protected] >>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>> >>> >> >> >> -- >> Nuwan Dias >> >> Technical Lead - WSO2, Inc. http://wso2.com >> email : [email protected] >> Phone : +94 777 775 729 >> > > > > -- > *Chalitha Maheshwari* > Software Engineer-QA, > WSO2 Inc. > > *E-mail:* [email protected] > *Mobile: *+94710 411 112 > -- *Chalitha Maheshwari* Software Engineer-QA, WSO2 Inc. *E-mail:* [email protected] *Mobile: *+94710 411 112
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
