On Fri, Jul 15, 2016 at 5:39 AM, Thushara Ranawaka <thusha...@wso2.com>
wrote:
> Hi Chanaka,
>
> On Fri, Jul 15, 2016 at 2:42 PM, Chanaka Fernando <chana...@wso2.com>
> wrote:
>
>> Hi Thushara,
>>
>> Increase the mysql active connections to 350. Please note that this is
>> bounded to server specs.
>> Now start 2 servers at a time. Wait 10 mins before starting the other 2
>> WSO2 servers.
>> Or else you can always start 1 server at a time every 10 mins. Therefore
>> it will take 30 mins to start all 4 nodes.
>>
>> Do you think we can tell something like this in front of a customer?
>> Normally customers want to start the servers within seconds and they want
>> to start an entire cluster within minutes (maximum). Please provide a
>> recommendation which makes sense in a real world. We need a proper way to
>> handle this.
>>
>
> This recommendation is for ES not for ESB and specially not for customers
> since we can automate this using config. ES uses governance features
> therefore ES need to have solr enabled. Therefore please disable[3] solr in
> all ESB nodes. Further more I understand they will start all the servers
> straightaway but for ES initial start(database with data) there is a heavy
> indexing process is happening in the background to have a smooth ride
> afterward. Since there is 4 nodes either user have to increase database
> connections from DB side or start servers one by one with a delay. We can
> automate this using indexingConfiguration[4] in registry.xml. This is just
> a one time thing, you can change it back to default values after the first
> startup.
>
> [4] - <startingDelayInSeconds>{Different values in each nodes with 10 mins
> different.}</startingDelayInSeconds>
> <indexingFrequencyInSeconds>3</indexingFrequencyInSeconds>
> <!--number of resources submit for given indexing thread -->
> <batchSize>50</batchSize>
> <!--number of worker threads for indexing -->
> <indexerPoolSize>10</indexerPoolSize>
>
> Thanks,
> Thushara.
>
>
In a different thought, this kind of work arounds will cause products to
hard to be run in a cloud environment where
you can not guarantee product startup order since containers/vm come
and go due to various reasons
Manually changing configurations also not possible in a automated
deployment environment.
So it is better if we can have a solid solution.
>
>
On Fri, Jul 15, 2016 at 1:25 PM, Thushara Ranawaka <thusha...@wso2.com>
>> wrote:
>>
>>> Hi Dilini,
>>>
>>>
>>> On Thu, Jul 14, 2016 at 2:31 PM, Dilini Gunatilake <dili...@wso2.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> @Thushara,
>>>> Thanks for the info. Any recommended number of max_connections to
>>>> configure considering indexing with a high load etc? Then we can add those
>>>> details in the docs to avoid future issues.
>>>>
>>>
>>> My personal recommendation,
>>> Increase the mysql active connections to 350. Please note that this is
>>> bounded to server specs.
>>> Now start 2 servers at a time. Wait 10 mins before starting the other 2
>>> WSO2 servers.
>>> Or else you can always start 1 server at a time every 10 mins. Therefore
>>> it will take 30 mins to start all 4 nodes.
>>>
>>> Thanks,
>>> Thushara.
>>>
>>>
>>>>
>>>> @Sumedha,
>>>> I was able to start all the nodes because the number of connections
>>>> reduces after some time.
>>>>
>>>> As Rajith mentioned, the concern is having that number of connections
>>>> created just for a server startup having less than 25 assets.
>>>>
>>>> Thanks,
>>>> Dilini
>>>>
>>>>
>>>> On Thu, Jul 14, 2016 at 12:53 PM, Rajith Vitharana <raji...@wso2.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> On Thu, Jul 14, 2016 at 12:44 PM, Sumedha Rubasinghe <sume...@wso2.com
>>>>> > wrote:
>>>>>
>>>>>> Guys,
>>>>>> Some food for thought.
>>>>>>
>>>>>> 1. A single MySQL Server has a globally defined max_connections
>>>>>> figure (151 as Dilini mentioned)
>>>>>>
>>>>>> So on a clustered setup, when you point to a single MySQL Server from
>>>>>> various WSO2 products, depending on no of keep alive connections
>>>>>> specified,
>>>>>> you will at some point will hit too many connections from server side @
>>>>>> some point.
>>>>>>
>>>>> Yeah totally agree, but that much(395) is bit high AFAIU when just
>>>>> starting up a server(without any load in 4 servers, and when it stabilize,
>>>>> it only consumes less than 20 connections without load)
>>>>>
>>>>> @Thushara, Thanks for the info, will check what you have mentioned.
>>>>>
>>>>> Thanks,
>>>>>
>>>>>>
>>>>>> So you have to rethink the deployment and use a different database
>>>>>> server.
>>>>>>
>>>>>> 2. If ES has a tendency to perform too many database calls @ startup,
>>>>>> - You can temporally eliminate that by controlling node start up order
>>>>>>
>>>>>>
>>>>>> On Thu, Jul 14, 2016 at 12:04 PM, Rajith Vitharana <raji...@wso2.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> We faced the same issue in ESB latest when clustering enabled, with
>>>>>>> 4 node cluster, number of connections goes to around 395 and gradually
>>>>>>> comes down to 20 or less, even with the exception happened(mysql having
>>>>>>> default max connections 151), it seems it gets stable to 20 or less
>>>>>>> connections after some time. Created a public jira to track this issue
>>>>>>> at
>>>>>>> [1]
>>>>>>>
>>>>>>> [1] - https://wso2.org/jira/browse/REGISTRY-3771
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> On Wed, Jul 13, 2016 at 4:42 PM, Dilini Gunatilake <dili...@wso2.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> I have set up a 4 node cluster for ES 2.1.0 Alpha2 pack with mysql
>>>>>>>> 5.7. When I started one store node I got an error [3].
>>>>>>>>
>>>>>>>> Then, I checked the connections in the mysql server and found that
>>>>>>>> the default number of max_connections (151) have been exceeded while
>>>>>>>> the
>>>>>>>> store node is starting. But initially without starting any ES instances
>>>>>>>> there were only around 20 connections. So, what we observed was when
>>>>>>>> starting ES instances substantially high amount of connections will be
>>>>>>>> created and they will be gradually reduced after sometime. Is this an
>>>>>>>> acceptable behavior?
>>>>>>>>
>>>>>>>> Also, we could find the recommended max_connections to configure in
>>>>>>>> some products [1][2], but couldn't find anything for ES. So, what is
>>>>>>>> the
>>>>>>>> recommended value for ES?
>>>>>>>> For the time being we will increase the max connections to 300 and
>>>>>>>> proceed with testing.
>>>>>>>>
>>>>>>>> Your earliest response is highly appreciated.
>>>>>>>>
>>>>>>>> [1] https://docs.wso2.com/display/AM200/Tuning+Performance
>>>>>>>> [2]
>>>>>>>> https://docs.wso2.com/display/Governance510/Performance+Tuning+Recommendations
>>>>>>>>
>>>>>>>> [3]
>>>>>>>>
>>>>>>>> [2016-07-13 07:50:40,014] ERROR
>>>>>>>> {org.wso2.carbon.registry.core.dataaccess.TransactionManager} -
>>>>>>>> Failed to
>>>>>>>> start new registry transaction.
>>>>>>>> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
>>>>>>>> Could not create connection to database server. Attempted reconnect 3
>>>>>>>> times. Giving up.
>>>>>>>> at sun.reflect.GeneratedConstructorAccessor107.newInstance(Unknown
>>>>>>>> Source)
>>>>>>>> at
>>>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
>>>>>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
>>>>>>>> at com.mysql.jdbc.Util.getInstance(Util.java:387)
>>>>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:917)
>>>>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:896)
>>>>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:885)
>>>>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:860)
>>>>>>>> at
>>>>>>>> com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2165)
>>>>>>>> at
>>>>>>>> com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2090)
>>>>>>>> at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:795)
>>>>>>>> at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:44)
>>>>>>>> at sun.reflect.GeneratedConstructorAccessor76.newInstance(Unknown
>>>>>>>> Source)
>>>>>>>> at
>>>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
>>>>>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
>>>>>>>> at
>>>>>>>> com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:400)
>>>>>>>> at
>>>>>>>> com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:327)
>>>>>>>> at
>>>>>>>> org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:278)
>>>>>>>> at
>>>>>>>> org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:182)
>>>>>>>> at
>>>>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:701)
>>>>>>>> at
>>>>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:635)
>>>>>>>> at
>>>>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188)
>>>>>>>> at
>>>>>>>> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:505)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:617)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:61)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:597)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:594)
>>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:594)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler.get(MountHandler.java:399)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.jdbc.handlers.HandlerManager.get(HandlerManager.java:2444)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.jdbc.handlers.UserDefinedHandlerManager.get(UserDefinedHandlerManager.java:200)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.jdbc.handlers.HandlerLifecycleManager.get(HandlerLifecycleManager.java:914)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:512)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:203)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:617)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:61)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:597)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:594)
>>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:594)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:242)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:210)
>>>>>>>> at
>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>> Caused by:
>>>>>>>> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
>>>>>>>> Too
>>>>>>>> many connections
>>>>>>>> at sun.reflect.GeneratedConstructorAccessor107.newInstance(Unknown
>>>>>>>> Source)
>>>>>>>> at
>>>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
>>>>>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
>>>>>>>> at com.mysql.jdbc.Util.getInstance(Util.java:387)
>>>>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:917)
>>>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3878)
>>>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3814)
>>>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:871)
>>>>>>>> at
>>>>>>>> com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1694)
>>>>>>>> at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1215)
>>>>>>>> at
>>>>>>>> com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2255)
>>>>>>>> at
>>>>>>>> com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2106)
>>>>>>>> ... 43 more
>>>>>>>> [2016-07-13 07:50:40,077] ERROR
>>>>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while
>>>>>>>> indexing.
>>>>>>>> Resource at path
>>>>>>>> "/_system/governance/store/asset_resources/gadget/8c3eda76-ec7f-43e3-bbe8-efa14f85d780/images_thumbnail"could
>>>>>>>> not be
>>>>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException:
>>>>>>>> Resource does not exist at path
>>>>>>>> /_system/governance/store/asset_resources/gadget/8c3eda76-ec7f-43e3-bbe8-efa14f85d780/images_thumbnail
>>>>>>>> [2016-07-13 07:50:40,076] ERROR
>>>>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while
>>>>>>>> indexing.
>>>>>>>> Resource at path "/_system/governance/gadgets/admin/Stacked Line
>>>>>>>> Chart/1.0.0"could not be
>>>>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException:
>>>>>>>> Resource does not exist at path
>>>>>>>> /_system/governance/gadgets/admin/Stacked
>>>>>>>> Line Chart/1.0.0
>>>>>>>> [2016-07-13 07:50:40,074] ERROR
>>>>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while
>>>>>>>> indexing.
>>>>>>>> Resource at path
>>>>>>>> "/_system/governance/store/asset_resources/gadget/57235acc-ad87-41db-84e7-dadc0bcff8d5/images_banner"could
>>>>>>>> not be
>>>>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException:
>>>>>>>> Resource does not exist at path
>>>>>>>> /_system/governance/store/asset_resources/gadget/57235acc-ad87-41db-84e7-dadc0bcff8d5/images_banner
>>>>>>>> [2016-07-13 07:50:40,067] ERROR
>>>>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while
>>>>>>>> indexing.
>>>>>>>> Resource at path
>>>>>>>> "/_system/governance/store/asset_resources/gadget/8c3eda76-ec7f-43e3-bbe8-efa14f85d780/images_banner"could
>>>>>>>> not be
>>>>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException:
>>>>>>>> Resource does not exist at path
>>>>>>>> /_system/governance/store/asset_resources/gadget/8c3eda76-ec7f-43e3-bbe8-efa14f85d780/images_banner
>>>>>>>> [2016-07-13 07:50:40,064] ERROR
>>>>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while
>>>>>>>> indexing.
>>>>>>>> Resource at path "/_system/governance/gadgets/admin/WSO2 Carbon Commits
>>>>>>>> List Discussion/1.0.0"could not be
>>>>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException:
>>>>>>>> Resource does not exist at path /_system/governance/gadgets/admin/WSO2
>>>>>>>> Carbon Commits List Discussion/1.0.0
>>>>>>>> [2016-07-13 07:50:40,062] ERROR
>>>>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while
>>>>>>>> indexing.
>>>>>>>> Resource at path
>>>>>>>> "/_system/governance/store/asset_resources/site/6b89dd21-0a98-4489-9f0d-fc327ff46bc0/images_thumbnail"could
>>>>>>>> not be
>>>>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException:
>>>>>>>> Resource does not exist at path
>>>>>>>> /_system/governance/store/asset_resources/site/6b89dd21-0a98-4489-9f0d-fc327ff46bc0/images_thumbnail
>>>>>>>> [2016-07-13 07:50:40,056] ERROR
>>>>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while
>>>>>>>> indexing.
>>>>>>>> Resource at path
>>>>>>>> "/_system/governance/store/asset_resources/gadget/4c4f7877-ff39-4379-9ee4-6c1827732ec6/images_thumbnail"could
>>>>>>>> not be
>>>>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException:
>>>>>>>> Resource does not exist at path
>>>>>>>> /_system/governance/store/asset_resources/gadget/4c4f7877-ff39-4379-9ee4-6c1827732ec6/images_thumbnail
>>>>>>>> [2016-07-13 07:50:40,079] ERROR
>>>>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while
>>>>>>>> indexing.
>>>>>>>> Resource at path
>>>>>>>> "/_system/governance/store/asset_resources/gadget/57235acc-ad87-41db-84e7-dadc0bcff8d5/images_thumbnail"could
>>>>>>>> not be
>>>>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException:
>>>>>>>> Resource does not exist at path
>>>>>>>> /_system/governance/store/asset_resources/gadget/57235acc-ad87-41db-84e7-dadc0bcff8d5/images_thumbnail
>>>>>>>>
>>>>>>>>
>>>>>>>> Thank you.
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> *Dilini GunatilakeSoftware Engineer - QA Team*
>>>>>>>> Mobile : +94 (0) 771 162518
>>>>>>>> dili...@wso2.com
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Dev mailing list
>>>>>>>> Dev@wso2.org
>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Rajith Vitharana
>>>>>>>
>>>>>>> Software Engineer,
>>>>>>> WSO2 Inc. : wso2.com
>>>>>>> Mobile : +94715883223
>>>>>>> Blog : http://lankavitharana.blogspot.com/
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Dev mailing list
>>>>>>> Dev@wso2.org
>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> /sumedha
>>>>>> m: +94 773017743
>>>>>> b : bit.ly/sumedha
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Rajith Vitharana
>>>>>
>>>>> Software Engineer,
>>>>> WSO2 Inc. : wso2.com
>>>>> Mobile : +94715883223
>>>>> Blog : http://lankavitharana.blogspot.com/
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> *Dilini GunatilakeSoftware Engineer - QA Team*
>>>> Mobile : +94771162518
>>>> dili...@wso2.com
>>>>
>>>>
>>>> _______________________________________________
>>>> Dev mailing list
>>>> Dev@wso2.org
>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Thushara Kasun Ranawaka
>>> Software Engineer
>>> WSO2 Inc.; <http://www.wso2.com>
>>> lean.enterprise.middleware
>>> Mobile : *+94 (0) 773438949 <%2B94%20%280%29%20773438949>*
>>> *thusha...@wso2.com <thusha...@wso2.com>*
>>>
>>> _______________________________________________
>>> Dev mailing list
>>> Dev@wso2.org
>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>
>>>
>>
>>
>> --
>> Thank you and Best Regards,
>> Chanaka Fernando
>> Senior Technical Lead
>> m: +94 773337238
>> https://wso2.com <https://wso2.com/signature>
>>
>>
>>
>>
>>
>>
>>
>
>
> --
> Thanks,
> Thushara Kasun Ranawaka
> Software Engineer
> WSO2 Inc.; <http://www.wso2.com>
> lean.enterprise.middleware
> Mobile : *+94 (0) 773438949 <%2B94%20%280%29%20773438949>*
> *thusha...@wso2.com <thusha...@wso2.com>*
>
> _______________________________________________
> Dev mailing list
> Dev@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>
--
Udara Liyanage
Software Engineer
WSO2, Inc.: http://wso2.com
lean. enterprise. middleware
Blog: http://udaraliyanage.wordpress.com
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
