Hi Rajith, Are you using governance aspects of registry? Kindly share the pack with reproducing steps, Will have a look.
Thanks, Thushara. On Fri, Jul 15, 2016 at 3:41 PM, Rajith Vitharana <[email protected]> wrote: > Hi All, > > As advised by Thushara, tried removing solr indexing in ESB using [1], but > in that case searching the registry with newly added resources fails(it > doesn't show any results). So is it ok to do this change to ESB? > > [1] - registry.xml <startIndexing>false</startIndexing> > > Thanks, > > On Fri, Jul 15, 2016 at 3:09 PM, Thushara Ranawaka <[email protected]> > wrote: > >> Hi Chanaka, >> >> On Fri, Jul 15, 2016 at 2:42 PM, Chanaka Fernando <[email protected]> >> wrote: >> >>> Hi Thushara, >>> >>> Increase the mysql active connections to 350. Please note that this is >>> bounded to server specs. >>> Now start 2 servers at a time. Wait 10 mins before starting the other 2 >>> WSO2 servers. >>> Or else you can always start 1 server at a time every 10 mins. Therefore >>> it will take 30 mins to start all 4 nodes. >>> >>> Do you think we can tell something like this in front of a customer? >>> Normally customers want to start the servers within seconds and they want >>> to start an entire cluster within minutes (maximum). Please provide a >>> recommendation which makes sense in a real world. We need a proper way to >>> handle this. >>> >> >> This recommendation is for ES not for ESB and specially not for customers >> since we can automate this using config. ES uses governance features >> therefore ES need to have solr enabled. Therefore please disable[3] solr in >> all ESB nodes. Further more I understand they will start all the servers >> straightaway but for ES initial start(database with data) there is a heavy >> indexing process is happening in the background to have a smooth ride >> afterward. Since there is 4 nodes either user have to increase database >> connections from DB side or start servers one by one with a delay. We can >> automate this using indexingConfiguration[4] in registry.xml. This is just >> a one time thing, you can change it back to default values after the first >> startup. >> >> [4] - <startingDelayInSeconds>{Different values in each nodes with 10 >> mins different.}</startingDelayInSeconds> >> <indexingFrequencyInSeconds>3</indexingFrequencyInSeconds> >> <!--number of resources submit for given indexing thread --> >> <batchSize>50</batchSize> >> <!--number of worker threads for indexing --> >> <indexerPoolSize>10</indexerPoolSize> >> >> Thanks, >> Thushara. >> >> >>> On Fri, Jul 15, 2016 at 1:25 PM, Thushara Ranawaka <[email protected]> >>> wrote: >>> >>>> Hi Dilini, >>>> >>>> >>>> On Thu, Jul 14, 2016 at 2:31 PM, Dilini Gunatilake <[email protected]> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> @Thushara, >>>>> Thanks for the info. Any recommended number of max_connections to >>>>> configure considering indexing with a high load etc? Then we can add those >>>>> details in the docs to avoid future issues. >>>>> >>>> >>>> My personal recommendation, >>>> Increase the mysql active connections to 350. Please note that this is >>>> bounded to server specs. >>>> Now start 2 servers at a time. Wait 10 mins before starting the other 2 >>>> WSO2 servers. >>>> Or else you can always start 1 server at a time every 10 mins. >>>> Therefore it will take 30 mins to start all 4 nodes. >>>> >>>> Thanks, >>>> Thushara. >>>> >>>> >>>>> >>>>> @Sumedha, >>>>> I was able to start all the nodes because the number of connections >>>>> reduces after some time. >>>>> >>>>> As Rajith mentioned, the concern is having that number of connections >>>>> created just for a server startup having less than 25 assets. >>>>> >>>>> Thanks, >>>>> Dilini >>>>> >>>>> >>>>> On Thu, Jul 14, 2016 at 12:53 PM, Rajith Vitharana <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> On Thu, Jul 14, 2016 at 12:44 PM, Sumedha Rubasinghe < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Guys, >>>>>>> Some food for thought. >>>>>>> >>>>>>> 1. A single MySQL Server has a globally defined max_connections >>>>>>> figure (151 as Dilini mentioned) >>>>>>> >>>>>>> So on a clustered setup, when you point to a single MySQL Server >>>>>>> from various WSO2 products, depending on no of keep alive connections >>>>>>> specified, you will at some point will hit too many connections from >>>>>>> server >>>>>>> side @ some point. >>>>>>> >>>>>> Yeah totally agree, but that much(395) is bit high AFAIU when just >>>>>> starting up a server(without any load in 4 servers, and when it >>>>>> stabilize, >>>>>> it only consumes less than 20 connections without load) >>>>>> >>>>>> @Thushara, Thanks for the info, will check what you have mentioned. >>>>>> >>>>>> Thanks, >>>>>> >>>>>>> >>>>>>> So you have to rethink the deployment and use a different database >>>>>>> server. >>>>>>> >>>>>>> 2. If ES has a tendency to perform too many database calls @ >>>>>>> startup, >>>>>>> - You can temporally eliminate that by controlling node start up >>>>>>> order >>>>>>> >>>>>>> >>>>>>> On Thu, Jul 14, 2016 at 12:04 PM, Rajith Vitharana <[email protected] >>>>>>> > wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> We faced the same issue in ESB latest when clustering enabled, with >>>>>>>> 4 node cluster, number of connections goes to around 395 and gradually >>>>>>>> comes down to 20 or less, even with the exception happened(mysql having >>>>>>>> default max connections 151), it seems it gets stable to 20 or less >>>>>>>> connections after some time. Created a public jira to track this issue >>>>>>>> at >>>>>>>> [1] >>>>>>>> >>>>>>>> [1] - https://wso2.org/jira/browse/REGISTRY-3771 >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> On Wed, Jul 13, 2016 at 4:42 PM, Dilini Gunatilake < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> I have set up a 4 node cluster for ES 2.1.0 Alpha2 pack with mysql >>>>>>>>> 5.7. When I started one store node I got an error [3]. >>>>>>>>> >>>>>>>>> Then, I checked the connections in the mysql server and found that >>>>>>>>> the default number of max_connections (151) have been exceeded while >>>>>>>>> the >>>>>>>>> store node is starting. But initially without starting any ES >>>>>>>>> instances >>>>>>>>> there were only around 20 connections. So, what we observed was when >>>>>>>>> starting ES instances substantially high amount of connections will be >>>>>>>>> created and they will be gradually reduced after sometime. Is this an >>>>>>>>> acceptable behavior? >>>>>>>>> >>>>>>>>> Also, we could find the recommended max_connections to configure >>>>>>>>> in some products [1][2], but couldn't find anything for ES. So, what >>>>>>>>> is the >>>>>>>>> recommended value for ES? >>>>>>>>> For the time being we will increase the max connections to 300 and >>>>>>>>> proceed with testing. >>>>>>>>> >>>>>>>>> Your earliest response is highly appreciated. >>>>>>>>> >>>>>>>>> [1] https://docs.wso2.com/display/AM200/Tuning+Performance >>>>>>>>> [2] >>>>>>>>> https://docs.wso2.com/display/Governance510/Performance+Tuning+Recommendations >>>>>>>>> >>>>>>>>> [3] >>>>>>>>> >>>>>>>>> [2016-07-13 07:50:40,014] ERROR >>>>>>>>> {org.wso2.carbon.registry.core.dataaccess.TransactionManager} - >>>>>>>>> Failed to >>>>>>>>> start new registry transaction. >>>>>>>>> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: >>>>>>>>> Could not create connection to database server. Attempted reconnect 3 >>>>>>>>> times. Giving up. >>>>>>>>> at sun.reflect.GeneratedConstructorAccessor107.newInstance(Unknown >>>>>>>>> Source) >>>>>>>>> at >>>>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>>>>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:422) >>>>>>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404) >>>>>>>>> at com.mysql.jdbc.Util.getInstance(Util.java:387) >>>>>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:917) >>>>>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:896) >>>>>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:885) >>>>>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:860) >>>>>>>>> at >>>>>>>>> com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2165) >>>>>>>>> at >>>>>>>>> com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2090) >>>>>>>>> at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:795) >>>>>>>>> at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:44) >>>>>>>>> at sun.reflect.GeneratedConstructorAccessor76.newInstance(Unknown >>>>>>>>> Source) >>>>>>>>> at >>>>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>>>>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:422) >>>>>>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404) >>>>>>>>> at >>>>>>>>> com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:400) >>>>>>>>> at >>>>>>>>> com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:327) >>>>>>>>> at >>>>>>>>> org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:278) >>>>>>>>> at >>>>>>>>> org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:182) >>>>>>>>> at >>>>>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:701) >>>>>>>>> at >>>>>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:635) >>>>>>>>> at >>>>>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188) >>>>>>>>> at >>>>>>>>> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:505) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:617) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:61) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:597) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:594) >>>>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:594) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler.get(MountHandler.java:399) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.jdbc.handlers.HandlerManager.get(HandlerManager.java:2444) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.jdbc.handlers.UserDefinedHandlerManager.get(UserDefinedHandlerManager.java:200) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.jdbc.handlers.HandlerLifecycleManager.get(HandlerLifecycleManager.java:914) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:512) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:203) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:617) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:61) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:597) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:594) >>>>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:594) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:242) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:210) >>>>>>>>> at >>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>>>>>> at >>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>>>>>>> at >>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>> Caused by: >>>>>>>>> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: >>>>>>>>> Too >>>>>>>>> many connections >>>>>>>>> at sun.reflect.GeneratedConstructorAccessor107.newInstance(Unknown >>>>>>>>> Source) >>>>>>>>> at >>>>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>>>>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:422) >>>>>>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404) >>>>>>>>> at com.mysql.jdbc.Util.getInstance(Util.java:387) >>>>>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:917) >>>>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3878) >>>>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3814) >>>>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:871) >>>>>>>>> at >>>>>>>>> com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1694) >>>>>>>>> at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1215) >>>>>>>>> at >>>>>>>>> com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2255) >>>>>>>>> at >>>>>>>>> com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2106) >>>>>>>>> ... 43 more >>>>>>>>> [2016-07-13 07:50:40,077] ERROR >>>>>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while >>>>>>>>> indexing. >>>>>>>>> Resource at path >>>>>>>>> "/_system/governance/store/asset_resources/gadget/8c3eda76-ec7f-43e3-bbe8-efa14f85d780/images_thumbnail"could >>>>>>>>> not be >>>>>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >>>>>>>>> Resource does not exist at path >>>>>>>>> /_system/governance/store/asset_resources/gadget/8c3eda76-ec7f-43e3-bbe8-efa14f85d780/images_thumbnail >>>>>>>>> [2016-07-13 07:50:40,076] ERROR >>>>>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while >>>>>>>>> indexing. >>>>>>>>> Resource at path "/_system/governance/gadgets/admin/Stacked Line >>>>>>>>> Chart/1.0.0"could not be >>>>>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >>>>>>>>> Resource does not exist at path >>>>>>>>> /_system/governance/gadgets/admin/Stacked >>>>>>>>> Line Chart/1.0.0 >>>>>>>>> [2016-07-13 07:50:40,074] ERROR >>>>>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while >>>>>>>>> indexing. >>>>>>>>> Resource at path >>>>>>>>> "/_system/governance/store/asset_resources/gadget/57235acc-ad87-41db-84e7-dadc0bcff8d5/images_banner"could >>>>>>>>> not be >>>>>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >>>>>>>>> Resource does not exist at path >>>>>>>>> /_system/governance/store/asset_resources/gadget/57235acc-ad87-41db-84e7-dadc0bcff8d5/images_banner >>>>>>>>> [2016-07-13 07:50:40,067] ERROR >>>>>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while >>>>>>>>> indexing. >>>>>>>>> Resource at path >>>>>>>>> "/_system/governance/store/asset_resources/gadget/8c3eda76-ec7f-43e3-bbe8-efa14f85d780/images_banner"could >>>>>>>>> not be >>>>>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >>>>>>>>> Resource does not exist at path >>>>>>>>> /_system/governance/store/asset_resources/gadget/8c3eda76-ec7f-43e3-bbe8-efa14f85d780/images_banner >>>>>>>>> [2016-07-13 07:50:40,064] ERROR >>>>>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while >>>>>>>>> indexing. >>>>>>>>> Resource at path "/_system/governance/gadgets/admin/WSO2 Carbon >>>>>>>>> Commits >>>>>>>>> List Discussion/1.0.0"could not be >>>>>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >>>>>>>>> Resource does not exist at path /_system/governance/gadgets/admin/WSO2 >>>>>>>>> Carbon Commits List Discussion/1.0.0 >>>>>>>>> [2016-07-13 07:50:40,062] ERROR >>>>>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while >>>>>>>>> indexing. >>>>>>>>> Resource at path >>>>>>>>> "/_system/governance/store/asset_resources/site/6b89dd21-0a98-4489-9f0d-fc327ff46bc0/images_thumbnail"could >>>>>>>>> not be >>>>>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >>>>>>>>> Resource does not exist at path >>>>>>>>> /_system/governance/store/asset_resources/site/6b89dd21-0a98-4489-9f0d-fc327ff46bc0/images_thumbnail >>>>>>>>> [2016-07-13 07:50:40,056] ERROR >>>>>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while >>>>>>>>> indexing. >>>>>>>>> Resource at path >>>>>>>>> "/_system/governance/store/asset_resources/gadget/4c4f7877-ff39-4379-9ee4-6c1827732ec6/images_thumbnail"could >>>>>>>>> not be >>>>>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >>>>>>>>> Resource does not exist at path >>>>>>>>> /_system/governance/store/asset_resources/gadget/4c4f7877-ff39-4379-9ee4-6c1827732ec6/images_thumbnail >>>>>>>>> [2016-07-13 07:50:40,079] ERROR >>>>>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} - Error while >>>>>>>>> indexing. >>>>>>>>> Resource at path >>>>>>>>> "/_system/governance/store/asset_resources/gadget/57235acc-ad87-41db-84e7-dadc0bcff8d5/images_thumbnail"could >>>>>>>>> not be >>>>>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException: >>>>>>>>> Resource does not exist at path >>>>>>>>> /_system/governance/store/asset_resources/gadget/57235acc-ad87-41db-84e7-dadc0bcff8d5/images_thumbnail >>>>>>>>> >>>>>>>>> >>>>>>>>> Thank you. >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> *Dilini GunatilakeSoftware Engineer - QA Team* >>>>>>>>> Mobile : +94 (0) 771 162518 >>>>>>>>> [email protected] >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Dev mailing list >>>>>>>>> [email protected] >>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Rajith Vitharana >>>>>>>> >>>>>>>> Software Engineer, >>>>>>>> WSO2 Inc. : wso2.com >>>>>>>> Mobile : +94715883223 >>>>>>>> Blog : http://lankavitharana.blogspot.com/ >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Dev mailing list >>>>>>>> [email protected] >>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> /sumedha >>>>>>> m: +94 773017743 >>>>>>> b : bit.ly/sumedha >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Rajith Vitharana >>>>>> >>>>>> Software Engineer, >>>>>> WSO2 Inc. : wso2.com >>>>>> Mobile : +94715883223 >>>>>> Blog : http://lankavitharana.blogspot.com/ >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> *Dilini GunatilakeSoftware Engineer - QA Team* >>>>> Mobile : +94771162518 >>>>> [email protected] >>>>> >>>>> >>>>> _______________________________________________ >>>>> Dev mailing list >>>>> [email protected] >>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>> >>>>> >>>> >>>> >>>> -- >>>> Thanks, >>>> Thushara Kasun Ranawaka >>>> Software Engineer >>>> WSO2 Inc.; <http://www.wso2.com> >>>> lean.enterprise.middleware >>>> Mobile : *+94 (0) 773438949 <%2B94%20%280%29%20773438949>* >>>> *[email protected] <[email protected]>* >>>> >>>> _______________________________________________ >>>> Dev mailing list >>>> [email protected] >>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>> >>>> >>> >>> >>> -- >>> Thank you and Best Regards, >>> Chanaka Fernando >>> Senior Technical Lead >>> m: +94 773337238 >>> https://wso2.com <https://wso2.com/signature> >>> >>> >>> >>> >>> >>> >>> >> >> >> -- >> Thanks, >> Thushara Kasun Ranawaka >> Software Engineer >> WSO2 Inc.; <http://www.wso2.com> >> lean.enterprise.middleware >> Mobile : *+94 (0) 773438949 <%2B94%20%280%29%20773438949>* >> *[email protected] <[email protected]>* >> >> _______________________________________________ >> Dev mailing list >> [email protected] >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > Rajith Vitharana > > Software Engineer, > WSO2 Inc. : wso2.com > Mobile : +94715883223 > Blog : http://lankavitharana.blogspot.com/ > -- Thanks, Thushara Kasun Ranawaka Software Engineer WSO2 Inc.; <http://www.wso2.com> lean.enterprise.middleware Mobile : *+94 (0) 773438949* *[email protected] <[email protected]>*
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
