Hi Dilini, To overcome the $subject issue you can do 2 things,
1. Increase the database max connections[1] 2. Increase the solr indexing frequency(indexingFrequencyInSeconds)[2]. Note: After the first startup you can decrease above values. This much indexing load will happen in first startup. @Rajith, AFAIU many of the connections are created to carry out the solr indexing task. Since ESB is not using Governance aspect you can simply disable[3] solr indexing task. [1] - http://stackoverflow.com/questions/22297773/increase-mysql-connections [2] - registry.xml <indexingFrequencyInSeconds>{add value here}</indexingFrequencyInSeconds> [3] - registry.xml <startIndexing>false</startIndexing> Thanks, Thushara. On Thu, Jul 14, 2016 at 12:04 PM, Rajith Vitharana <[email protected]> wrote: > Hi, > > We faced the same issue in ESB latest when clustering enabled, with 4 node > cluster, number of connections goes to around 395 and gradually comes down > to 20 or less, even with the exception happened(mysql having default max > connections 151), it seems it gets stable to 20 or less connections after > some time. Created a public jira to track this issue at [1] > > [1] - https://wso2.org/jira/browse/REGISTRY-3771 > > Thanks, > > On Wed, Jul 13, 2016 at 4:42 PM, Dilini Gunatilake <[email protected]> > wrote: > >> Hi All, >> >> I have set up a 4 node cluster for ES 2.1.0 Alpha2 pack with mysql 5.7. >> When I started one store node I got an error [3]. >> >> Then, I checked the connections in the mysql server and found that the >> default number of max_connections (151) have been exceeded while the store >> node is starting. But initially without starting any ES instances there >> were only around 20 connections. So, what we observed was when starting ES >> instances substantially high amount of connections will be created and they >> will be gradually reduced after sometime. Is this an acceptable behavior? >> >> Also, we could find the recommended max_connections to configure in some >> products [1][2], but couldn't find anything for ES. So, what is the >> recommended value for ES? >> For the time being we will increase the max connections to 300 and >> proceed with testing. >> >> Your earliest response is highly appreciated. >> >> [1] https://docs.wso2.com/display/AM200/Tuning+Performance >> [2] >> https://docs.wso2.com/display/Governance510/Performance+Tuning+Recommendations >> >> [3] >> >> [2016-07-13 07:50:40,014] ERROR >> {org.wso2.carbon.registry.core.dataaccess.TransactionManager} - Failed to >> start new registry transaction. >> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: >> Could not create connection to database server. Attempted reconnect 3 >> times. Giving up. >> at sun.reflect.GeneratedConstructorAccessor107.newInstance(Unknown Source) >> at >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >> at java.lang.reflect.Constructor.newInstance(Constructor.java:422) >> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404) >> at com.mysql.jdbc.Util.getInstance(Util.java:387) >> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:917) >> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:896) >> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:885) >> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:860) >> at >> com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2165) >> at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2090) >> at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:795) >> at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:44) >> at sun.reflect.GeneratedConstructorAccessor76.newInstance(Unknown Source) >> at >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >> at java.lang.reflect.Constructor.newInstance(Constructor.java:422) >> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404) >> at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:400) >> at >> com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:327) >> at >> org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:278) >> at >> org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:182) >> at >> org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:701) >> at >> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:635) >> at >> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) >> at >> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) >> at >> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80) >> at >> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443) >> at >> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:505) >> at >> org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:617) >> at >> org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:61) >> at >> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:597) >> at >> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:594) >> at java.security.AccessController.doPrivileged(Native Method) >> at >> org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:594) >> at >> org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler.get(MountHandler.java:399) >> at >> org.wso2.carbon.registry.core.jdbc.handlers.HandlerManager.get(HandlerManager.java:2444) >> at >> org.wso2.carbon.registry.core.jdbc.handlers.UserDefinedHandlerManager.get(UserDefinedHandlerManager.java:200) >> at >> org.wso2.carbon.registry.core.jdbc.handlers.HandlerLifecycleManager.get(HandlerLifecycleManager.java:914) >> at >> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:512) >> at >> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:203) >> at >> org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:617) >> at >> org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:61) >> at >> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:597) >> at >> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:594) >> at java.security.AccessController.doPrivileged(Native Method) >> at >> org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:594) >> at >> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:242) >> at >> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:210) >> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> at java.lang.Thread.run(Thread.java:745) >> Caused by: >> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Too >> many connections >> at sun.reflect.GeneratedConstructorAccessor107.newInstance(Unknown Source) >> at >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >> at java.lang.reflect.Constructor.newInstance(Constructor.java:422) >> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404) >> at com.mysql.jdbc.Util.getInstance(Util.java:387) >> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:917) >> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3878) >> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3814) >> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:871) >> at >> com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1694) >> at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1215) >> at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2255) >> at >> com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2106) >> ... 43 more >> [2016-07-13 07:50:40,077] ERROR >> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while indexing. >> Resource at path >> "/_system/governance/store/asset_resources/gadget/8c3eda76-ec7f-43e3-bbe8-efa14f85d780/images_thumbnail"could >> not be >> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >> Resource does not exist at path >> /_system/governance/store/asset_resources/gadget/8c3eda76-ec7f-43e3-bbe8-efa14f85d780/images_thumbnail >> [2016-07-13 07:50:40,076] ERROR >> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while indexing. >> Resource at path "/_system/governance/gadgets/admin/Stacked Line >> Chart/1.0.0"could not be >> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >> Resource does not exist at path /_system/governance/gadgets/admin/Stacked >> Line Chart/1.0.0 >> [2016-07-13 07:50:40,074] ERROR >> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while indexing. >> Resource at path >> "/_system/governance/store/asset_resources/gadget/57235acc-ad87-41db-84e7-dadc0bcff8d5/images_banner"could >> not be >> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >> Resource does not exist at path >> /_system/governance/store/asset_resources/gadget/57235acc-ad87-41db-84e7-dadc0bcff8d5/images_banner >> [2016-07-13 07:50:40,067] ERROR >> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while indexing. >> Resource at path >> "/_system/governance/store/asset_resources/gadget/8c3eda76-ec7f-43e3-bbe8-efa14f85d780/images_banner"could >> not be >> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >> Resource does not exist at path >> /_system/governance/store/asset_resources/gadget/8c3eda76-ec7f-43e3-bbe8-efa14f85d780/images_banner >> [2016-07-13 07:50:40,064] ERROR >> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while indexing. >> Resource at path "/_system/governance/gadgets/admin/WSO2 Carbon Commits >> List Discussion/1.0.0"could not be >> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >> Resource does not exist at path /_system/governance/gadgets/admin/WSO2 >> Carbon Commits List Discussion/1.0.0 >> [2016-07-13 07:50:40,062] ERROR >> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while indexing. >> Resource at path >> "/_system/governance/store/asset_resources/site/6b89dd21-0a98-4489-9f0d-fc327ff46bc0/images_thumbnail"could >> not be >> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >> Resource does not exist at path >> /_system/governance/store/asset_resources/site/6b89dd21-0a98-4489-9f0d-fc327ff46bc0/images_thumbnail >> [2016-07-13 07:50:40,056] ERROR >> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while indexing. >> Resource at path >> "/_system/governance/store/asset_resources/gadget/4c4f7877-ff39-4379-9ee4-6c1827732ec6/images_thumbnail"could >> not be >> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >> Resource does not exist at path >> /_system/governance/store/asset_resources/gadget/4c4f7877-ff39-4379-9ee4-6c1827732ec6/images_thumbnail >> [2016-07-13 07:50:40,079] ERROR >> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while indexing. >> Resource at path >> "/_system/governance/store/asset_resources/gadget/57235acc-ad87-41db-84e7-dadc0bcff8d5/images_thumbnail"could >> not be >> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >> Resource does not exist at path >> /_system/governance/store/asset_resources/gadget/57235acc-ad87-41db-84e7-dadc0bcff8d5/images_thumbnail >> >> >> Thank you. >> Regards, >> >> -- >> >> *Dilini GunatilakeSoftware Engineer - QA Team* >> Mobile : +94 (0) 771 162518 >> [email protected] >> >> _______________________________________________ >> Dev mailing list >> [email protected] >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > Rajith Vitharana > > Software Engineer, > WSO2 Inc. : wso2.com > Mobile : +94715883223 > Blog : http://lankavitharana.blogspot.com/ > > _______________________________________________ > Dev mailing list > [email protected] > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- Thanks, Thushara Kasun Ranawaka Software Engineer WSO2 Inc.; <http://www.wso2.com> lean.enterprise.middleware Mobile : *+94 (0) 773438949 <%2B94%20%280%29%20773438949>* *[email protected] <[email protected]>*
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
