Hi Thushara, Increase the mysql active connections to 350. Please note that this is bounded to server specs. Now start 2 servers at a time. Wait 10 mins before starting the other 2 WSO2 servers. Or else you can always start 1 server at a time every 10 mins. Therefore it will take 30 mins to start all 4 nodes.
Do you think we can tell something like this in front of a customer? Normally customers want to start the servers within seconds and they want to start an entire cluster within minutes (maximum). Please provide a recommendation which makes sense in a real world. We need a proper way to handle this. On Fri, Jul 15, 2016 at 1:25 PM, Thushara Ranawaka <[email protected]> wrote: > Hi Dilini, > > > On Thu, Jul 14, 2016 at 2:31 PM, Dilini Gunatilake <[email protected]> > wrote: > >> Hi, >> >> @Thushara, >> Thanks for the info. Any recommended number of max_connections to >> configure considering indexing with a high load etc? Then we can add those >> details in the docs to avoid future issues. >> > > My personal recommendation, > Increase the mysql active connections to 350. Please note that this is > bounded to server specs. > Now start 2 servers at a time. Wait 10 mins before starting the other 2 > WSO2 servers. > Or else you can always start 1 server at a time every 10 mins. Therefore > it will take 30 mins to start all 4 nodes. > > Thanks, > Thushara. > > >> >> @Sumedha, >> I was able to start all the nodes because the number of connections >> reduces after some time. >> >> As Rajith mentioned, the concern is having that number of connections >> created just for a server startup having less than 25 assets. >> >> Thanks, >> Dilini >> >> >> On Thu, Jul 14, 2016 at 12:53 PM, Rajith Vitharana <[email protected]> >> wrote: >> >>> Hi, >>> >>> On Thu, Jul 14, 2016 at 12:44 PM, Sumedha Rubasinghe <[email protected]> >>> wrote: >>> >>>> Guys, >>>> Some food for thought. >>>> >>>> 1. A single MySQL Server has a globally defined max_connections figure >>>> (151 as Dilini mentioned) >>>> >>>> So on a clustered setup, when you point to a single MySQL Server from >>>> various WSO2 products, depending on no of keep alive connections specified, >>>> you will at some point will hit too many connections from server side @ >>>> some point. >>>> >>> Yeah totally agree, but that much(395) is bit high AFAIU when just >>> starting up a server(without any load in 4 servers, and when it stabilize, >>> it only consumes less than 20 connections without load) >>> >>> @Thushara, Thanks for the info, will check what you have mentioned. >>> >>> Thanks, >>> >>>> >>>> So you have to rethink the deployment and use a different database >>>> server. >>>> >>>> 2. If ES has a tendency to perform too many database calls @ startup, >>>> - You can temporally eliminate that by controlling node start up order >>>> >>>> >>>> On Thu, Jul 14, 2016 at 12:04 PM, Rajith Vitharana <[email protected]> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> We faced the same issue in ESB latest when clustering enabled, with 4 >>>>> node cluster, number of connections goes to around 395 and gradually comes >>>>> down to 20 or less, even with the exception happened(mysql having default >>>>> max connections 151), it seems it gets stable to 20 or less connections >>>>> after some time. Created a public jira to track this issue at [1] >>>>> >>>>> [1] - https://wso2.org/jira/browse/REGISTRY-3771 >>>>> >>>>> Thanks, >>>>> >>>>> On Wed, Jul 13, 2016 at 4:42 PM, Dilini Gunatilake <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi All, >>>>>> >>>>>> I have set up a 4 node cluster for ES 2.1.0 Alpha2 pack with mysql >>>>>> 5.7. When I started one store node I got an error [3]. >>>>>> >>>>>> Then, I checked the connections in the mysql server and found that >>>>>> the default number of max_connections (151) have been exceeded while the >>>>>> store node is starting. But initially without starting any ES instances >>>>>> there were only around 20 connections. So, what we observed was when >>>>>> starting ES instances substantially high amount of connections will be >>>>>> created and they will be gradually reduced after sometime. Is this an >>>>>> acceptable behavior? >>>>>> >>>>>> Also, we could find the recommended max_connections to configure in >>>>>> some products [1][2], but couldn't find anything for ES. So, what is the >>>>>> recommended value for ES? >>>>>> For the time being we will increase the max connections to 300 and >>>>>> proceed with testing. >>>>>> >>>>>> Your earliest response is highly appreciated. >>>>>> >>>>>> [1] https://docs.wso2.com/display/AM200/Tuning+Performance >>>>>> [2] >>>>>> https://docs.wso2.com/display/Governance510/Performance+Tuning+Recommendations >>>>>> >>>>>> [3] >>>>>> >>>>>> [2016-07-13 07:50:40,014] ERROR >>>>>> {org.wso2.carbon.registry.core.dataaccess.TransactionManager} - Failed >>>>>> to >>>>>> start new registry transaction. >>>>>> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: >>>>>> Could not create connection to database server. Attempted reconnect 3 >>>>>> times. Giving up. >>>>>> at sun.reflect.GeneratedConstructorAccessor107.newInstance(Unknown >>>>>> Source) >>>>>> at >>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:422) >>>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404) >>>>>> at com.mysql.jdbc.Util.getInstance(Util.java:387) >>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:917) >>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:896) >>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:885) >>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:860) >>>>>> at >>>>>> com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2165) >>>>>> at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2090) >>>>>> at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:795) >>>>>> at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:44) >>>>>> at sun.reflect.GeneratedConstructorAccessor76.newInstance(Unknown >>>>>> Source) >>>>>> at >>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:422) >>>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404) >>>>>> at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:400) >>>>>> at >>>>>> com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:327) >>>>>> at >>>>>> org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:278) >>>>>> at >>>>>> org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:182) >>>>>> at >>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:701) >>>>>> at >>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:635) >>>>>> at >>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) >>>>>> at >>>>>> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) >>>>>> at >>>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80) >>>>>> at >>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443) >>>>>> at >>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:505) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:617) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:61) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:597) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:594) >>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:594) >>>>>> at >>>>>> org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler.get(MountHandler.java:399) >>>>>> at >>>>>> org.wso2.carbon.registry.core.jdbc.handlers.HandlerManager.get(HandlerManager.java:2444) >>>>>> at >>>>>> org.wso2.carbon.registry.core.jdbc.handlers.UserDefinedHandlerManager.get(UserDefinedHandlerManager.java:200) >>>>>> at >>>>>> org.wso2.carbon.registry.core.jdbc.handlers.HandlerLifecycleManager.get(HandlerLifecycleManager.java:914) >>>>>> at >>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:512) >>>>>> at >>>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:203) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:617) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:61) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:597) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:594) >>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>> at >>>>>> org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:594) >>>>>> at >>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:242) >>>>>> at >>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:210) >>>>>> at >>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>> Caused by: >>>>>> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Too >>>>>> many connections >>>>>> at sun.reflect.GeneratedConstructorAccessor107.newInstance(Unknown >>>>>> Source) >>>>>> at >>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:422) >>>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404) >>>>>> at com.mysql.jdbc.Util.getInstance(Util.java:387) >>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:917) >>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3878) >>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3814) >>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:871) >>>>>> at >>>>>> com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1694) >>>>>> at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1215) >>>>>> at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2255) >>>>>> at >>>>>> com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2106) >>>>>> ... 43 more >>>>>> [2016-07-13 07:50:40,077] ERROR >>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while indexing. >>>>>> Resource at path >>>>>> "/_system/governance/store/asset_resources/gadget/8c3eda76-ec7f-43e3-bbe8-efa14f85d780/images_thumbnail"could >>>>>> not be >>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >>>>>> Resource does not exist at path >>>>>> /_system/governance/store/asset_resources/gadget/8c3eda76-ec7f-43e3-bbe8-efa14f85d780/images_thumbnail >>>>>> [2016-07-13 07:50:40,076] ERROR >>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while indexing. >>>>>> Resource at path "/_system/governance/gadgets/admin/Stacked Line >>>>>> Chart/1.0.0"could not be >>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >>>>>> Resource does not exist at path /_system/governance/gadgets/admin/Stacked >>>>>> Line Chart/1.0.0 >>>>>> [2016-07-13 07:50:40,074] ERROR >>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while indexing. >>>>>> Resource at path >>>>>> "/_system/governance/store/asset_resources/gadget/57235acc-ad87-41db-84e7-dadc0bcff8d5/images_banner"could >>>>>> not be >>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >>>>>> Resource does not exist at path >>>>>> /_system/governance/store/asset_resources/gadget/57235acc-ad87-41db-84e7-dadc0bcff8d5/images_banner >>>>>> [2016-07-13 07:50:40,067] ERROR >>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while indexing. >>>>>> Resource at path >>>>>> "/_system/governance/store/asset_resources/gadget/8c3eda76-ec7f-43e3-bbe8-efa14f85d780/images_banner"could >>>>>> not be >>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >>>>>> Resource does not exist at path >>>>>> /_system/governance/store/asset_resources/gadget/8c3eda76-ec7f-43e3-bbe8-efa14f85d780/images_banner >>>>>> [2016-07-13 07:50:40,064] ERROR >>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while indexing. >>>>>> Resource at path "/_system/governance/gadgets/admin/WSO2 Carbon Commits >>>>>> List Discussion/1.0.0"could not be >>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >>>>>> Resource does not exist at path /_system/governance/gadgets/admin/WSO2 >>>>>> Carbon Commits List Discussion/1.0.0 >>>>>> [2016-07-13 07:50:40,062] ERROR >>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while indexing. >>>>>> Resource at path >>>>>> "/_system/governance/store/asset_resources/site/6b89dd21-0a98-4489-9f0d-fc327ff46bc0/images_thumbnail"could >>>>>> not be >>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >>>>>> Resource does not exist at path >>>>>> /_system/governance/store/asset_resources/site/6b89dd21-0a98-4489-9f0d-fc327ff46bc0/images_thumbnail >>>>>> [2016-07-13 07:50:40,056] ERROR >>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while indexing. >>>>>> Resource at path >>>>>> "/_system/governance/store/asset_resources/gadget/4c4f7877-ff39-4379-9ee4-6c1827732ec6/images_thumbnail"could >>>>>> not be >>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >>>>>> Resource does not exist at path >>>>>> /_system/governance/store/asset_resources/gadget/4c4f7877-ff39-4379-9ee4-6c1827732ec6/images_thumbnail >>>>>> [2016-07-13 07:50:40,079] ERROR >>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while indexing. >>>>>> Resource at path >>>>>> "/_system/governance/store/asset_resources/gadget/57235acc-ad87-41db-84e7-dadc0bcff8d5/images_thumbnail"could >>>>>> not be >>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >>>>>> Resource does not exist at path >>>>>> /_system/governance/store/asset_resources/gadget/57235acc-ad87-41db-84e7-dadc0bcff8d5/images_thumbnail >>>>>> >>>>>> >>>>>> Thank you. >>>>>> Regards, >>>>>> >>>>>> -- >>>>>> >>>>>> *Dilini GunatilakeSoftware Engineer - QA Team* >>>>>> Mobile : +94 (0) 771 162518 >>>>>> [email protected] >>>>>> >>>>>> _______________________________________________ >>>>>> Dev mailing list >>>>>> [email protected] >>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Rajith Vitharana >>>>> >>>>> Software Engineer, >>>>> WSO2 Inc. : wso2.com >>>>> Mobile : +94715883223 >>>>> Blog : http://lankavitharana.blogspot.com/ >>>>> >>>>> _______________________________________________ >>>>> Dev mailing list >>>>> [email protected] >>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>> >>>>> >>>> >>>> >>>> -- >>>> /sumedha >>>> m: +94 773017743 >>>> b : bit.ly/sumedha >>>> >>> >>> >>> >>> -- >>> Rajith Vitharana >>> >>> Software Engineer, >>> WSO2 Inc. : wso2.com >>> Mobile : +94715883223 >>> Blog : http://lankavitharana.blogspot.com/ >>> >> >> >> >> -- >> >> *Dilini GunatilakeSoftware Engineer - QA Team* >> Mobile : +94771162518 >> [email protected] >> >> >> _______________________________________________ >> Dev mailing list >> [email protected] >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > Thanks, > Thushara Kasun Ranawaka > Software Engineer > WSO2 Inc.; <http://www.wso2.com> > lean.enterprise.middleware > Mobile : *+94 (0) 773438949 <%2B94%20%280%29%20773438949>* > *[email protected] <[email protected]>* > > _______________________________________________ > Dev mailing list > [email protected] > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- Thank you and Best Regards, Chanaka Fernando Senior Technical Lead m: +94 773337238 https://wso2.com <https://wso2.com/signature>
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
