Hi, On Thu, Jul 14, 2016 at 12:44 PM, Sumedha Rubasinghe <sume...@wso2.com> wrote:
> Guys, > Some food for thought. > > 1. A single MySQL Server has a globally defined max_connections figure > (151 as Dilini mentioned) > > So on a clustered setup, when you point to a single MySQL Server from > various WSO2 products, depending on no of keep alive connections specified, > you will at some point will hit too many connections from server side @ > some point. > Yeah totally agree, but that much(395) is bit high AFAIU when just starting up a server(without any load in 4 servers, and when it stabilize, it only consumes less than 20 connections without load) @Thushara, Thanks for the info, will check what you have mentioned. Thanks, > > So you have to rethink the deployment and use a different database server. > > 2. If ES has a tendency to perform too many database calls @ startup, > - You can temporally eliminate that by controlling node start up order > > > On Thu, Jul 14, 2016 at 12:04 PM, Rajith Vitharana <raji...@wso2.com> > wrote: > >> Hi, >> >> We faced the same issue in ESB latest when clustering enabled, with 4 >> node cluster, number of connections goes to around 395 and gradually comes >> down to 20 or less, even with the exception happened(mysql having default >> max connections 151), it seems it gets stable to 20 or less connections >> after some time. Created a public jira to track this issue at [1] >> >> [1] - https://wso2.org/jira/browse/REGISTRY-3771 >> >> Thanks, >> >> On Wed, Jul 13, 2016 at 4:42 PM, Dilini Gunatilake <dili...@wso2.com> >> wrote: >> >>> Hi All, >>> >>> I have set up a 4 node cluster for ES 2.1.0 Alpha2 pack with mysql 5.7. >>> When I started one store node I got an error [3]. >>> >>> Then, I checked the connections in the mysql server and found that the >>> default number of max_connections (151) have been exceeded while the store >>> node is starting. But initially without starting any ES instances there >>> were only around 20 connections. So, what we observed was when starting ES >>> instances substantially high amount of connections will be created and they >>> will be gradually reduced after sometime. Is this an acceptable behavior? >>> >>> Also, we could find the recommended max_connections to configure in some >>> products [1][2], but couldn't find anything for ES. So, what is the >>> recommended value for ES? >>> For the time being we will increase the max connections to 300 and >>> proceed with testing. >>> >>> Your earliest response is highly appreciated. >>> >>> [1] https://docs.wso2.com/display/AM200/Tuning+Performance >>> [2] >>> https://docs.wso2.com/display/Governance510/Performance+Tuning+Recommendations >>> >>> [3] >>> >>> [2016-07-13 07:50:40,014] ERROR >>> {org.wso2.carbon.registry.core.dataaccess.TransactionManager} - Failed to >>> start new registry transaction. >>> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: >>> Could not create connection to database server. Attempted reconnect 3 >>> times. Giving up. >>> at sun.reflect.GeneratedConstructorAccessor107.newInstance(Unknown >>> Source) >>> at >>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>> at java.lang.reflect.Constructor.newInstance(Constructor.java:422) >>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404) >>> at com.mysql.jdbc.Util.getInstance(Util.java:387) >>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:917) >>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:896) >>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:885) >>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:860) >>> at >>> com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2165) >>> at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2090) >>> at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:795) >>> at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:44) >>> at sun.reflect.GeneratedConstructorAccessor76.newInstance(Unknown Source) >>> at >>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>> at java.lang.reflect.Constructor.newInstance(Constructor.java:422) >>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404) >>> at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:400) >>> at >>> com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:327) >>> at >>> org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:278) >>> at >>> org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:182) >>> at >>> org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:701) >>> at >>> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:635) >>> at >>> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) >>> at >>> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) >>> at >>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80) >>> at >>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443) >>> at >>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:505) >>> at >>> org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:617) >>> at >>> org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:61) >>> at >>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:597) >>> at >>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:594) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at >>> org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:594) >>> at >>> org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler.get(MountHandler.java:399) >>> at >>> org.wso2.carbon.registry.core.jdbc.handlers.HandlerManager.get(HandlerManager.java:2444) >>> at >>> org.wso2.carbon.registry.core.jdbc.handlers.UserDefinedHandlerManager.get(UserDefinedHandlerManager.java:200) >>> at >>> org.wso2.carbon.registry.core.jdbc.handlers.HandlerLifecycleManager.get(HandlerLifecycleManager.java:914) >>> at >>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:512) >>> at >>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:203) >>> at >>> org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:617) >>> at >>> org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:61) >>> at >>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:597) >>> at >>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:594) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at >>> org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:594) >>> at >>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:242) >>> at >>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:210) >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>> at java.lang.Thread.run(Thread.java:745) >>> Caused by: >>> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Too >>> many connections >>> at sun.reflect.GeneratedConstructorAccessor107.newInstance(Unknown >>> Source) >>> at >>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>> at java.lang.reflect.Constructor.newInstance(Constructor.java:422) >>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404) >>> at com.mysql.jdbc.Util.getInstance(Util.java:387) >>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:917) >>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3878) >>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3814) >>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:871) >>> at >>> com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1694) >>> at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1215) >>> at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2255) >>> at >>> com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2106) >>> ... 43 more >>> [2016-07-13 07:50:40,077] ERROR >>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while indexing. >>> Resource at path >>> "/_system/governance/store/asset_resources/gadget/8c3eda76-ec7f-43e3-bbe8-efa14f85d780/images_thumbnail"could >>> not be >>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >>> Resource does not exist at path >>> /_system/governance/store/asset_resources/gadget/8c3eda76-ec7f-43e3-bbe8-efa14f85d780/images_thumbnail >>> [2016-07-13 07:50:40,076] ERROR >>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while indexing. >>> Resource at path "/_system/governance/gadgets/admin/Stacked Line >>> Chart/1.0.0"could not be >>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >>> Resource does not exist at path /_system/governance/gadgets/admin/Stacked >>> Line Chart/1.0.0 >>> [2016-07-13 07:50:40,074] ERROR >>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while indexing. >>> Resource at path >>> "/_system/governance/store/asset_resources/gadget/57235acc-ad87-41db-84e7-dadc0bcff8d5/images_banner"could >>> not be >>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >>> Resource does not exist at path >>> /_system/governance/store/asset_resources/gadget/57235acc-ad87-41db-84e7-dadc0bcff8d5/images_banner >>> [2016-07-13 07:50:40,067] ERROR >>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while indexing. >>> Resource at path >>> "/_system/governance/store/asset_resources/gadget/8c3eda76-ec7f-43e3-bbe8-efa14f85d780/images_banner"could >>> not be >>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >>> Resource does not exist at path >>> /_system/governance/store/asset_resources/gadget/8c3eda76-ec7f-43e3-bbe8-efa14f85d780/images_banner >>> [2016-07-13 07:50:40,064] ERROR >>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while indexing. >>> Resource at path "/_system/governance/gadgets/admin/WSO2 Carbon Commits >>> List Discussion/1.0.0"could not be >>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >>> Resource does not exist at path /_system/governance/gadgets/admin/WSO2 >>> Carbon Commits List Discussion/1.0.0 >>> [2016-07-13 07:50:40,062] ERROR >>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while indexing. >>> Resource at path >>> "/_system/governance/store/asset_resources/site/6b89dd21-0a98-4489-9f0d-fc327ff46bc0/images_thumbnail"could >>> not be >>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >>> Resource does not exist at path >>> /_system/governance/store/asset_resources/site/6b89dd21-0a98-4489-9f0d-fc327ff46bc0/images_thumbnail >>> [2016-07-13 07:50:40,056] ERROR >>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while indexing. >>> Resource at path >>> "/_system/governance/store/asset_resources/gadget/4c4f7877-ff39-4379-9ee4-6c1827732ec6/images_thumbnail"could >>> not be >>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >>> Resource does not exist at path >>> /_system/governance/store/asset_resources/gadget/4c4f7877-ff39-4379-9ee4-6c1827732ec6/images_thumbnail >>> [2016-07-13 07:50:40,079] ERROR >>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while indexing. >>> Resource at path >>> "/_system/governance/store/asset_resources/gadget/57235acc-ad87-41db-84e7-dadc0bcff8d5/images_thumbnail"could >>> not be >>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >>> Resource does not exist at path >>> /_system/governance/store/asset_resources/gadget/57235acc-ad87-41db-84e7-dadc0bcff8d5/images_thumbnail >>> >>> >>> Thank you. >>> Regards, >>> >>> -- >>> >>> *Dilini GunatilakeSoftware Engineer - QA Team* >>> Mobile : +94 (0) 771 162518 >>> dili...@wso2.com >>> >>> _______________________________________________ >>> Dev mailing list >>> Dev@wso2.org >>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>> >>> >> >> >> -- >> Rajith Vitharana >> >> Software Engineer, >> WSO2 Inc. : wso2.com >> Mobile : +94715883223 >> Blog : http://lankavitharana.blogspot.com/ >> >> _______________________________________________ >> Dev mailing list >> Dev@wso2.org >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > /sumedha > m: +94 773017743 > b : bit.ly/sumedha > -- Rajith Vitharana Software Engineer, WSO2 Inc. : wso2.com Mobile : +94715883223 Blog : http://lankavitharana.blogspot.com/
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
