Hi Chanaka,
On Fri, Jul 15, 2016 at 2:42 PM, Chanaka Fernando <[email protected]> wrote:
> Hi Thushara,
>
> Increase the mysql active connections to 350. Please note that this is
> bounded to server specs.
> Now start 2 servers at a time. Wait 10 mins before starting the other 2
> WSO2 servers.
> Or else you can always start 1 server at a time every 10 mins. Therefore
> it will take 30 mins to start all 4 nodes.
>
> Do you think we can tell something like this in front of a customer?
> Normally customers want to start the servers within seconds and they want
> to start an entire cluster within minutes (maximum). Please provide a
> recommendation which makes sense in a real world. We need a proper way to
> handle this.
>
This recommendation is for ES not for ESB and specially not for customers
since we can automate this using config. ES uses governance features
therefore ES need to have solr enabled. Therefore please disable[3] solr in
all ESB nodes. Further more I understand they will start all the servers
straightaway but for ES initial start(database with data) there is a heavy
indexing process is happening in the background to have a smooth ride
afterward. Since there is 4 nodes either user have to increase database
connections from DB side or start servers one by one with a delay. We can
automate this using indexingConfiguration[4] in registry.xml. This is just
a one time thing, you can change it back to default values after the first
startup.
[4] - <startingDelayInSeconds>{Different values in each nodes with 10 mins
different.}</startingDelayInSeconds>
<indexingFrequencyInSeconds>3</indexingFrequencyInSeconds>
<!--number of resources submit for given indexing thread -->
<batchSize>50</batchSize>
<!--number of worker threads for indexing -->
<indexerPoolSize>10</indexerPoolSize>
Thanks,
Thushara.
> On Fri, Jul 15, 2016 at 1:25 PM, Thushara Ranawaka <[email protected]>
> wrote:
>
>> Hi Dilini,
>>
>>
>> On Thu, Jul 14, 2016 at 2:31 PM, Dilini Gunatilake <[email protected]>
>> wrote:
>>
>>> Hi,
>>>
>>> @Thushara,
>>> Thanks for the info. Any recommended number of max_connections to
>>> configure considering indexing with a high load etc? Then we can add those
>>> details in the docs to avoid future issues.
>>>
>>
>> My personal recommendation,
>> Increase the mysql active connections to 350. Please note that this is
>> bounded to server specs.
>> Now start 2 servers at a time. Wait 10 mins before starting the other 2
>> WSO2 servers.
>> Or else you can always start 1 server at a time every 10 mins. Therefore
>> it will take 30 mins to start all 4 nodes.
>>
>> Thanks,
>> Thushara.
>>
>>
>>>
>>> @Sumedha,
>>> I was able to start all the nodes because the number of connections
>>> reduces after some time.
>>>
>>> As Rajith mentioned, the concern is having that number of connections
>>> created just for a server startup having less than 25 assets.
>>>
>>> Thanks,
>>> Dilini
>>>
>>>
>>> On Thu, Jul 14, 2016 at 12:53 PM, Rajith Vitharana <[email protected]>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> On Thu, Jul 14, 2016 at 12:44 PM, Sumedha Rubasinghe <[email protected]>
>>>> wrote:
>>>>
>>>>> Guys,
>>>>> Some food for thought.
>>>>>
>>>>> 1. A single MySQL Server has a globally defined max_connections figure
>>>>> (151 as Dilini mentioned)
>>>>>
>>>>> So on a clustered setup, when you point to a single MySQL Server from
>>>>> various WSO2 products, depending on no of keep alive connections
>>>>> specified,
>>>>> you will at some point will hit too many connections from server side @
>>>>> some point.
>>>>>
>>>> Yeah totally agree, but that much(395) is bit high AFAIU when just
>>>> starting up a server(without any load in 4 servers, and when it stabilize,
>>>> it only consumes less than 20 connections without load)
>>>>
>>>> @Thushara, Thanks for the info, will check what you have mentioned.
>>>>
>>>> Thanks,
>>>>
>>>>>
>>>>> So you have to rethink the deployment and use a different database
>>>>> server.
>>>>>
>>>>> 2. If ES has a tendency to perform too many database calls @ startup,
>>>>> - You can temporally eliminate that by controlling node start up order
>>>>>
>>>>>
>>>>> On Thu, Jul 14, 2016 at 12:04 PM, Rajith Vitharana <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> We faced the same issue in ESB latest when clustering enabled, with 4
>>>>>> node cluster, number of connections goes to around 395 and gradually
>>>>>> comes
>>>>>> down to 20 or less, even with the exception happened(mysql having default
>>>>>> max connections 151), it seems it gets stable to 20 or less connections
>>>>>> after some time. Created a public jira to track this issue at [1]
>>>>>>
>>>>>> [1] - https://wso2.org/jira/browse/REGISTRY-3771
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> On Wed, Jul 13, 2016 at 4:42 PM, Dilini Gunatilake <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> I have set up a 4 node cluster for ES 2.1.0 Alpha2 pack with mysql
>>>>>>> 5.7. When I started one store node I got an error [3].
>>>>>>>
>>>>>>> Then, I checked the connections in the mysql server and found that
>>>>>>> the default number of max_connections (151) have been exceeded while the
>>>>>>> store node is starting. But initially without starting any ES instances
>>>>>>> there were only around 20 connections. So, what we observed was when
>>>>>>> starting ES instances substantially high amount of connections will be
>>>>>>> created and they will be gradually reduced after sometime. Is this an
>>>>>>> acceptable behavior?
>>>>>>>
>>>>>>> Also, we could find the recommended max_connections to configure in
>>>>>>> some products [1][2], but couldn't find anything for ES. So, what is the
>>>>>>> recommended value for ES?
>>>>>>> For the time being we will increase the max connections to 300 and
>>>>>>> proceed with testing.
>>>>>>>
>>>>>>> Your earliest response is highly appreciated.
>>>>>>>
>>>>>>> [1] https://docs.wso2.com/display/AM200/Tuning+Performance
>>>>>>> [2]
>>>>>>> https://docs.wso2.com/display/Governance510/Performance+Tuning+Recommendations
>>>>>>>
>>>>>>> [3]
>>>>>>>
>>>>>>> [2016-07-13 07:50:40,014] ERROR
>>>>>>> {org.wso2.carbon.registry.core.dataaccess.TransactionManager} - Failed
>>>>>>> to
>>>>>>> start new registry transaction.
>>>>>>> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
>>>>>>> Could not create connection to database server. Attempted reconnect 3
>>>>>>> times. Giving up.
>>>>>>> at sun.reflect.GeneratedConstructorAccessor107.newInstance(Unknown
>>>>>>> Source)
>>>>>>> at
>>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
>>>>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
>>>>>>> at com.mysql.jdbc.Util.getInstance(Util.java:387)
>>>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:917)
>>>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:896)
>>>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:885)
>>>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:860)
>>>>>>> at
>>>>>>> com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2165)
>>>>>>> at
>>>>>>> com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2090)
>>>>>>> at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:795)
>>>>>>> at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:44)
>>>>>>> at sun.reflect.GeneratedConstructorAccessor76.newInstance(Unknown
>>>>>>> Source)
>>>>>>> at
>>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
>>>>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
>>>>>>> at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:400)
>>>>>>> at
>>>>>>> com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:327)
>>>>>>> at
>>>>>>> org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:278)
>>>>>>> at
>>>>>>> org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:182)
>>>>>>> at
>>>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:701)
>>>>>>> at
>>>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:635)
>>>>>>> at
>>>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188)
>>>>>>> at
>>>>>>> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:505)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:617)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:61)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:597)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:594)
>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:594)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler.get(MountHandler.java:399)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.jdbc.handlers.HandlerManager.get(HandlerManager.java:2444)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.jdbc.handlers.UserDefinedHandlerManager.get(UserDefinedHandlerManager.java:200)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.jdbc.handlers.HandlerLifecycleManager.get(HandlerLifecycleManager.java:914)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:512)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:203)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:617)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:61)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:597)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:594)
>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:594)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:242)
>>>>>>> at
>>>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:210)
>>>>>>> at
>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>> Caused by:
>>>>>>> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
>>>>>>> Too
>>>>>>> many connections
>>>>>>> at sun.reflect.GeneratedConstructorAccessor107.newInstance(Unknown
>>>>>>> Source)
>>>>>>> at
>>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
>>>>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
>>>>>>> at com.mysql.jdbc.Util.getInstance(Util.java:387)
>>>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:917)
>>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3878)
>>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3814)
>>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:871)
>>>>>>> at
>>>>>>> com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1694)
>>>>>>> at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1215)
>>>>>>> at
>>>>>>> com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2255)
>>>>>>> at
>>>>>>> com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2106)
>>>>>>> ... 43 more
>>>>>>> [2016-07-13 07:50:40,077] ERROR
>>>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while
>>>>>>> indexing.
>>>>>>> Resource at path
>>>>>>> "/_system/governance/store/asset_resources/gadget/8c3eda76-ec7f-43e3-bbe8-efa14f85d780/images_thumbnail"could
>>>>>>> not be
>>>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException:
>>>>>>> Resource does not exist at path
>>>>>>> /_system/governance/store/asset_resources/gadget/8c3eda76-ec7f-43e3-bbe8-efa14f85d780/images_thumbnail
>>>>>>> [2016-07-13 07:50:40,076] ERROR
>>>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while
>>>>>>> indexing.
>>>>>>> Resource at path "/_system/governance/gadgets/admin/Stacked Line
>>>>>>> Chart/1.0.0"could not be
>>>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException:
>>>>>>> Resource does not exist at path
>>>>>>> /_system/governance/gadgets/admin/Stacked
>>>>>>> Line Chart/1.0.0
>>>>>>> [2016-07-13 07:50:40,074] ERROR
>>>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while
>>>>>>> indexing.
>>>>>>> Resource at path
>>>>>>> "/_system/governance/store/asset_resources/gadget/57235acc-ad87-41db-84e7-dadc0bcff8d5/images_banner"could
>>>>>>> not be
>>>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException:
>>>>>>> Resource does not exist at path
>>>>>>> /_system/governance/store/asset_resources/gadget/57235acc-ad87-41db-84e7-dadc0bcff8d5/images_banner
>>>>>>> [2016-07-13 07:50:40,067] ERROR
>>>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while
>>>>>>> indexing.
>>>>>>> Resource at path
>>>>>>> "/_system/governance/store/asset_resources/gadget/8c3eda76-ec7f-43e3-bbe8-efa14f85d780/images_banner"could
>>>>>>> not be
>>>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException:
>>>>>>> Resource does not exist at path
>>>>>>> /_system/governance/store/asset_resources/gadget/8c3eda76-ec7f-43e3-bbe8-efa14f85d780/images_banner
>>>>>>> [2016-07-13 07:50:40,064] ERROR
>>>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while
>>>>>>> indexing.
>>>>>>> Resource at path "/_system/governance/gadgets/admin/WSO2 Carbon Commits
>>>>>>> List Discussion/1.0.0"could not be
>>>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException:
>>>>>>> Resource does not exist at path /_system/governance/gadgets/admin/WSO2
>>>>>>> Carbon Commits List Discussion/1.0.0
>>>>>>> [2016-07-13 07:50:40,062] ERROR
>>>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while
>>>>>>> indexing.
>>>>>>> Resource at path
>>>>>>> "/_system/governance/store/asset_resources/site/6b89dd21-0a98-4489-9f0d-fc327ff46bc0/images_thumbnail"could
>>>>>>> not be
>>>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException:
>>>>>>> Resource does not exist at path
>>>>>>> /_system/governance/store/asset_resources/site/6b89dd21-0a98-4489-9f0d-fc327ff46bc0/images_thumbnail
>>>>>>> [2016-07-13 07:50:40,056] ERROR
>>>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while
>>>>>>> indexing.
>>>>>>> Resource at path
>>>>>>> "/_system/governance/store/asset_resources/gadget/4c4f7877-ff39-4379-9ee4-6c1827732ec6/images_thumbnail"could
>>>>>>> not be
>>>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException:
>>>>>>> Resource does not exist at path
>>>>>>> /_system/governance/store/asset_resources/gadget/4c4f7877-ff39-4379-9ee4-6c1827732ec6/images_thumbnail
>>>>>>> [2016-07-13 07:50:40,079] ERROR
>>>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while
>>>>>>> indexing.
>>>>>>> Resource at path
>>>>>>> "/_system/governance/store/asset_resources/gadget/57235acc-ad87-41db-84e7-dadc0bcff8d5/images_thumbnail"could
>>>>>>> not be
>>>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException:
>>>>>>> Resource does not exist at path
>>>>>>> /_system/governance/store/asset_resources/gadget/57235acc-ad87-41db-84e7-dadc0bcff8d5/images_thumbnail
>>>>>>>
>>>>>>>
>>>>>>> Thank you.
>>>>>>> Regards,
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> *Dilini GunatilakeSoftware Engineer - QA Team*
>>>>>>> Mobile : +94 (0) 771 162518
>>>>>>> [email protected]
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Dev mailing list
>>>>>>> [email protected]
>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Rajith Vitharana
>>>>>>
>>>>>> Software Engineer,
>>>>>> WSO2 Inc. : wso2.com
>>>>>> Mobile : +94715883223
>>>>>> Blog : http://lankavitharana.blogspot.com/
>>>>>>
>>>>>> _______________________________________________
>>>>>> Dev mailing list
>>>>>> [email protected]
>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> /sumedha
>>>>> m: +94 773017743
>>>>> b : bit.ly/sumedha
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Rajith Vitharana
>>>>
>>>> Software Engineer,
>>>> WSO2 Inc. : wso2.com
>>>> Mobile : +94715883223
>>>> Blog : http://lankavitharana.blogspot.com/
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> *Dilini GunatilakeSoftware Engineer - QA Team*
>>> Mobile : +94771162518
>>> [email protected]
>>>
>>>
>>> _______________________________________________
>>> Dev mailing list
>>> [email protected]
>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>
>>>
>>
>>
>> --
>> Thanks,
>> Thushara Kasun Ranawaka
>> Software Engineer
>> WSO2 Inc.; <http://www.wso2.com>
>> lean.enterprise.middleware
>> Mobile : *+94 (0) 773438949 <%2B94%20%280%29%20773438949>*
>> *[email protected] <[email protected]>*
>>
>> _______________________________________________
>> Dev mailing list
>> [email protected]
>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>
>>
>
>
> --
> Thank you and Best Regards,
> Chanaka Fernando
> Senior Technical Lead
> m: +94 773337238
> https://wso2.com <https://wso2.com/signature>
>
>
>
>
>
>
>
--
Thanks,
Thushara Kasun Ranawaka
Software Engineer
WSO2 Inc.; <http://www.wso2.com>
lean.enterprise.middleware
Mobile : *+94 (0) 773438949*
*[email protected] <[email protected]>*
_______________________________________________
Dev mailing list
[email protected]
http://wso2.org/cgi-bin/mailman/listinfo/dev
