Also AliveMemberCount in LB is always 0. Does that mean although there are successful join messages, in actuality no app server never joined?
Damitha On Thu, Jul 19, 2012 at 10:06 AM, Damitha Kumarage <[email protected]> wrote: > Hi Sajith, > During these weird join/disappear cases can you verify that lxc instances > are continuously accessible. Verifying that you can log into them and stay > logged is enough to verify that. > > Damitha > > > On Thu, Jul 19, 2012 at 8:48 AM, Nirmal Fernando <[email protected]> wrote: > >> I see abnormal behaviours of nodes started up. >> >> [2012-07-19 02:49:46,351] INFO - ServiceRequestsInFlightAutoscaler >> *********** isPrimaryLoadBalancer: true >> [2012-07-19 02:49:46,513] INFO - ServiceRequestsInFlightAutoscaler >> ************ AliveMemberCount: 0 >> [2012-07-19 02:49:46,599] WARN - ServiceRequestsInFlightAutoscaler App >> domain Sanity check failed for [wso2.as.domain] . Current instances: 0. >> Required instances: 1 >> [2012-07-19 02:49:46,621] INFO - ServiceRequestsInFlightAutoscaler >> Launching 1 App instances for domain wso2.as.domain >> [2012-07-19 02:50:16,331] INFO - ServiceRequestsInFlightAutoscaler >> *********** isPrimaryLoadBalancer: true >> [2012-07-19 02:50:17,041] INFO - ServiceRequestsInFlightAutoscaler >> ************ AliveMemberCount: 0 >> >> >> >> [2012-07-19 02:50:46,329] INFO - ServiceRequestsInFlightAutoscaler >> *********** isPrimaryLoadBalancer: true >> [2012-07-19 02:50:46,917] INFO - ServiceRequestsInFlightAutoscaler >> ************ AliveMemberCount: 0 >> [2012-07-19 02:51:16,330] INFO - ServiceRequestsInFlightAutoscaler >> *********** isPrimaryLoadBalancer: true >> [2012-07-19 02:51:17,108] INFO - ServiceRequestsInFlightAutoscaler >> ************ AliveMemberCount: 0 >> [2012-07-19 02:51:46,330] INFO - ServiceRequestsInFlightAutoscaler >> *********** isPrimaryLoadBalancer: true >> [2012-07-19 02:53:29,819] INFO - RpcMembershipRequestHandler Received >> JOIN message from 172.16.1.2:4000(wso2.as.domain) >> [2012-07-19 02:53:29,820] INFO - MembershipManager Application member >> 172.16.1.2:4000(wso2.as.domain) joined group wso2.as.domain >> [2012-07-19 02:53:40,824] INFO - DefaultGroupManagementAgent Application >> member Host:172.16.1.2, Port: 4000, HTTP:9763, HTTPS:9443, ACTIVE:true >> joined application cluster >> >> *It seems to got joined, but .... scroll down till the above exact colour >> * >> >> >> [2012-07-19 03:03:27,179] INFO - ServiceRequestsInFlightAutoscaler >> ************ AliveMemberCount: 0 >> [2012-07-19 03:03:27,818] WARN - ServiceRequestsInFlightAutoscaler App >> domain Sanity check failed for [wso2.as.domain] . Current instances: 0. >> Required instances: 1 >> [2012-07-19 03:03:27,831] INFO - ServiceRequestsInFlightAutoscaler >> Launching 1 App instances for domain wso2.as.domain >> [2012-07-19 03:03:46,329] INFO - ServiceRequestsInFlightAutoscaler >> *********** isPrimaryLoadBalancer: true >> [2012-07-19 03:04:16,181] INFO - ServiceRequestsInFlightAutoscaler >> ************ AliveMemberCount: 0 >> [2012-07-19 03:04:46,330] INFO - ServiceRequestsInFlightAutoscaler >> *********** isPrimaryLoadBalancer: true >> [2012-07-19 03:04:52,975] INFO - ServiceRequestsInFlightAutoscaler >> ************ AliveMemberCount: 0 >> [2012-07-19 03:05:16,329] INFO - ServiceRequestsInFlightAutoscaler >> *********** isPrimaryLoadBalancer: true >> >> [2012-07-19 03:05:25,719] INFO - ServiceRequestsInFlightAutoscaler >> ************ AliveMemberCount: 0 >> [2012-07-19 03:05:46,330] INFO - ServiceRequestsInFlightAutoscaler >> *********** isPrimaryLoadBalancer: true >> >> [2012-07-19 03:06:23,848] INFO - RpcMembershipRequestHandler Received >> JOIN message from 172.15.0.254:4006(wso2.as.domain) >> [2012-07-19 03:06:23,848] INFO - MembershipManager Application member >> 172.15.0.254:4006(wso2.as.domain) joined group wso2.as.domain >> [2012-07-19 03:06:33,988] INFO - DefaultGroupManagementAgent Application >> member Host:172.15.0.254, Port: 4006, HTTP:9763, HTTPS:9443, ACTIVE:true >> joined application cluster >> [2012-07-19 03:06:41,761] INFO - DefaultGroupManagementAgent Application >> member Host:172.15.0.254, Port: 4006, HTTP:9763, HTTPS:9443, ACTIVE:true >> left cluster. >> [2012-07-19 03:06:41,761] INFO - TribesMembershipListener Member >> 172.15.0.254:4006(wso2.as.domain) left cluster >> >> *And who is this 172.15.0.254?? This is not a node that we started, I >> think. oh ... this is the node that u started manually? Then why it left so >> soon?* >> >> >> >> >> [2012-07-19 03:07:36,874] INFO - MembershipManager Application member >> 172.15.0.254:4006(wso2.as.domain) joined group wso2.as.domain >> [2012-07-19 03:07:46,874] INFO - DefaultGroupManagementAgent Application >> member Host:172.15.0.254, Port: 4006, HTTP:9763, HTTPS:9443, ACTIVE:true >> joined application cluster >> [2012-07-19 03:07:50,728] INFO - RpcMembershipRequestHandler Received >> JOIN message from 172.15.0.254:4006(wso2.as.domain) >> >> >> >> [2012-07-19 03:09:25,312] INFO - RpcMembershipRequestHandler Received >> JOIN message from 172.16.1.3:4000(wso2.as.domain) >> [2012-07-19 03:09:25,313] INFO - MembershipManager Application member >> 172.16.1.3:4000(wso2.as.domain) joined group wso2.as.domain >> [2012-07-19 03:09:32,325] INFO - RpcMembershipRequestHandler Received >> MEMBER_LIST message from 172.16.1.3:4000(wso2.as.domain) >> >> >> [2012-07-19 03:09:32,326] INFO - MemberListCommand Trying to add member >> 172.16.1.2:4000(wso2.as.domain)... >> [2012-07-19 03:09:32,326] INFO - MemberListCommand Added member >> 172.16.1.2:4000(wso2.as.domain) >> >> *I suspect it got dropped, some point of time (network issues?). I could >> see the same issue in 172.16.1.3 instance too. >> >> * >> [2012-07-19 03:09:32,326] INFO - MemberListCommand Trying to add member >> 172.15.0.254:4006(wso2.as.domain)... >> [2012-07-19 03:09:32,326] INFO - MemberListCommand Added member >> 172.15.0.254:4006(wso2.as.domain) >> [2012-07-19 03:09:32,326] INFO - MemberListCommand Trying to add member >> 172.16.1.3:4000(wso2.as.domain)... >> [2012-07-19 03:09:32,327] INFO - MemberListCommand Added member >> 172.16.1.3:4000(wso2.as.domain) >> [2012-07-19 03:09:36,320] INFO - DefaultGroupManagementAgent Application >> member Host:172.16.1.3, Port: 4000, HTTP:9763, HTTPS:9443, ACTIVE:true >> joined application cluster >> >> >> >> >> >> >> [2012-07-19 03:11:34,771] INFO - RpcMembershipRequestHandler Received >> MEMBER_LIST message from 172.16.1.3:4000(wso2.as.domain) >> >> [2012-07-19 03:11:34,771] INFO - MemberListCommand Trying to add member >> 172.16.1.2:4000(wso2.as.domain)... >> [2012-07-19 03:11:34,772] INFO - MemberListCommand Added member >> 172.16.1.2:4000(wso2.as.domain) >> >> *And again... and so on...* >> >> [2012-07-19 03:11:34,772] INFO - MemberListCommand Trying to add member >> 172.15.0.254:4006(wso2.as.domain)... >> [2012-07-19 03:11:34,772] INFO - MemberListCommand Added member >> 172.15.0.254:4006(wso2.as.domain) >> [2012-07-19 03:11:34,772] INFO - MemberListCommand Trying to add member >> 172.16.1.3:4000(wso2.as.domain)... >> [2012-07-19 03:11:34,772] INFO - MemberListCommand Added member >> 172.16.1.3:4000(wso2.as.domain) >> [2012-07-19 03:12:06,604] INFO - RpcMembershipRequestHandler Received >> MEMBER_LIST message from 172.16.1.2:4000(wso2.as.domain) >> [2012-07-19 03:12:06,604] INFO - MemberListCommand Trying to add member >> 172.16.1.3:4000(wso2.as.domain)... >> [2012-07-19 03:12:06,604] INFO - MemberListCommand Added member >> 172.16.1.3:4000(wso2.as.domain) >> [2012-07-19 03:12:06,605] INFO - MemberListCommand Trying to add member >> 172.16.1.2:4000(wso2.as.domain)... >> [2012-07-19 03:12:06,605] INFO - MemberListCommand Added member >> 172.16.1.2:4000(wso2.as.domain) >> >> >> >> >> >> >> >> [2012-07-19 03:13:07,058] INFO - DefaultGroupManagementAgent Application >> member Host:172.15.0.254, Port: 4006, HTTP:9763, HTTPS:9443, ACTIVE:true >> left cluster. >> [2012-07-19 03:13:07,059] INFO - TribesMembershipListener Member >> 172.15.0.254:4006(wso2.as.domain) left cluster >> >> >> [2012-07-19 03:15:13,591] INFO - RpcMembershipRequestHandler Received >> MEMBER_LIST message from 172.16.1.3:4000(wso2.as.domain) >> [2012-07-19 03:15:13,591] INFO - MemberListCommand Trying to add member >> 172.16.1.2:4000(wso2.as.domain)... >> [2012-07-19 03:15:13,591] INFO - MemberListCommand Added member >> 172.16.1.2:4000(wso2.as.domain) >> [2012-07-19 03:15:13,591] INFO - MemberListCommand Trying to add member >> 172.16.1.3:4000(wso2.as.domain)... >> [2012-07-19 03:15:13,592] INFO - MemberListCommand Added member >> 172.16.1.3:4000(wso2.as.domain) >> [2012-07-19 03:15:46,794] INFO - ServiceRequestsInFlightAutoscaler >> ************ AliveMemberCount: 0 >> >> [2012-07-19 03:16:15,999] INFO - RpcMembershipRequestHandler Received >> MEMBER_LIST message from 172.16.1.2:4000(wso2.as.domain) >> [2012-07-19 03:16:16,000] INFO - MemberListCommand Trying to add member >> 172.16.1.3:4000(wso2.as.domain)... >> [2012-07-19 03:16:16,000] INFO - MemberListCommand Added member >> 172.16.1.3:4000(wso2.as.domain) >> [2012-07-19 03:16:16,000] INFO - MemberListCommand Trying to add member >> 172.16.1.2:4000(wso2.as.domain)... >> [2012-07-19 03:16:16,000] INFO - MemberListCommand Added member >> 172.16.1.2:4000(wso2.as.domain) >> [2012-07-19 03:16:16,329] INFO - ServiceRequestsInFlightAutoscaler >> *********** isPrimaryLoadBalancer: true >> [2012-07-19 03:16:16,511] INFO - ServiceRequestsInFlightAutoscaler >> ************ AliveMemberCount: 0 >> >> ^[[D[2012-07-19 03:16:24,885] INFO - DefaultGroupManagementAgent >> Application member Host:172.16.1.2, Port: 4000, HTTP:9763, HTTPS:9443, >> ACTIVE:true left cluster. >> [2012-07-19 03:16:24,885] INFO - TribesMembershipListener Member >> 172.16.1.2:4000(wso2.as.domain) left cluster >> >> >> [2012-07-19 03:16:46,330] INFO - ServiceRequestsInFlightAutoscaler >> *********** isPrimaryLoadBalancer: true >> [2012-07-19 03:16:46,485] INFO - ServiceRequestsInFlightAutoscaler >> ************ AliveMemberCount: 0 >> [2012-07-19 03:17:16,329] INFO - ServiceRequestsInFlightAutoscaler >> *********** isPrimaryLoadBalancer: true >> [2012-07-19 03:17:16,644] INFO - ServiceRequestsInFlightAutoscaler >> ************ AliveMemberCount: 0 >> [2012-07-19 03:23:19,262] INFO - ServiceRequestsInFlightAutoscaler >> *********** isPrimaryLoadBalancer: true >> [2012-07-19 03:23:19,778] INFO - ServiceRequestsInFlightAutoscaler >> ************ AliveMemberCount: 0 >> [2012-07-19 03:23:46,330] INFO - ServiceRequestsInFlightAutoscaler >> *********** isPrimaryLoadBalancer: true >> [2012-07-19 03:23:46,565] INFO - ServiceRequestsInFlightAutoscaler >> ************ AliveMemberCount: 0 >> [2012-07-19 03:24:16,329] INFO - ServiceRequestsInFlightAutoscaler >> *********** isPrimaryLoadBalancer: true >> [2012-07-19 03:24:16,826] INFO - ServiceRequestsInFlightAutoscaler >> ************ AliveMemberCount: 0 >> [2012-07-19 03:24:46,330] INFO - ServiceRequestsInFlightAutoscaler >> *********** isPrimaryLoadBalancer: true >> [2012-07-19 03:24:46,488] INFO - ServiceRequestsInFlightAutoscaler >> ************ AliveMemberCount: 0 >> [2012-07-19 03:25:16,330] INFO - ServiceRequestsInFlightAutoscaler >> *********** isPrimaryLoadBalancer: true >> [2012-07-19 03:25:16,497] INFO - ServiceRequestsInFlightAutoscaler >> ************ AliveMemberCount: 0 >> [2012-07-19 03:25:46,329] INFO - ServiceRequestsInFlightAutoscaler >> *********** isPrimaryLoadBalancer: true >> [2012-07-19 03:25:46,792] INFO - ServiceRequestsInFlightAutoscaler >> ************ AliveMemberCount: 0 >> [2012-07-19 03:26:00,610] INFO - MembershipManager Application member >> 172.15.0.254:4006(wso2.as.domain) joined group wso2.as.domain >> >> [2012-07-19 03:26:10,613] INFO - DefaultGroupManagementAgent Application >> member Host:172.15.0.254, Port: 4006, HTTP:9763, HTTPS:9443, ACTIVE:true >> joined application cluster >> ----[2012-07-19 03:26:13,929] INFO - RpcMembershipRequestHandler >> Received JOIN message from 172.15.0.254:4006(wso2.as.domain) >> [2012-07-19 03:26:16,329] INFO - ServiceRequestsInFlightAutoscaler >> *********** isPrimaryLoadBalancer: true >> [2012-07-19 03:26:16,505] INFO - ServiceRequestsInFlightAutoscaler >> ************ AliveMemberCount: 0 >> [2012-07-19 03:26:37,916] INFO - DefaultGroupManagementAgent Application >> member Host:172.16.1.3, Port: 4000, HTTP:9763, HTTPS:9443, ACTIVE:true left >> cluster. >> [2012-07-19 03:26:37,916] INFO - TribesMembershipListener Member >> 172.16.1.3:4000(wso2.as.domain) left cluster >> [2012-07-19 03:26:46,330] INFO - ServiceRequestsInFlightAutoscaler >> *********** isPrimaryLoadBalancer: true >> [2012-07-19 03:26:46,429] INFO - ServiceRequestsInFlightAutoscaler >> ************ AliveMemberCount: 0 >> >> >> >> [2012-07-19 03:27:16,330] INFO - ServiceRequestsInFlightAutoscaler >> *********** isPrimaryLoadBalancer: true >> [2012-07-19 03:27:16,431] INFO - ServiceRequestsInFlightAutoscaler >> ************ AliveMemberCount: 0 >> [2012-07-19 03:27:46,330] INFO - ServiceRequestsInFlightAutoscaler >> *********** isPrimaryLoadBalancer: true >> [2012-07-19 03:27:46,424] INFO - ServiceRequestsInFlightAutoscaler >> ************ AliveMemberCount: 0 >> [2012-07-19 03:27:47,018] INFO - DefaultGroupManagementAgent Application >> member Host:172.15.0.254, Port: 4006, HTTP:9763, HTTPS:9443, ACTIVE:true >> left cluster. >> [2012-07-19 03:27:47,018] INFO - TribesMembershipListener Member >> 172.15.0.254:4006(wso2.as.domain) left cluster >> [2012-07-19 03:28:16,329] INFO - ServiceRequestsInFlightAutoscaler >> *********** isPrimaryLoadBalancer: true >> [2012-07-19 03:28:16,434] INFO - ServiceRequestsInFlightAutoscaler >> ************ AliveMemberCount: 0 >> [2012-07-19 03:28:16,518] WARN - ServiceRequestsInFlightAutoscaler App >> domain Sanity check failed for [wso2.as.domain] . Current instances: 0. >> Required instances: 1 >> [2012-07-19 03:28:16,602] INFO - ServiceRequestsInFlightAutoscaler >> Launching 1 App instances for domain wso2.as.domain >> >> On Thu, Jul 19, 2012 at 4:09 AM, Sajith Kariyawasam <[email protected]>wrote: >> >>> Here s what i have observed in the last try , few anomalies (like at one >>> instance I could see 2 instances are up and running in Openstack) , but >>> didn;t get that timeout error as we got while the setup was in >>> Virtualboxes, and was able to scale up as well as scale down, but im not >>> too sure abt the conditions LB has been using, >>> will do another a test in the morning , till then Nirmal can u verify ? >>> >>> LB and Autoscaler were in the host machine, while only the Openstack was >>> in VirtualBox.. >>> >>> First I let Autoscaler to start an instance in Openstack,.relevant logs >>> are as follows.. >>> >>> LB Log >>> ====== >>> >>> [2012-07-19 02:49:46,351] INFO - ServiceRequestsInFlightAutoscaler >>> *********** isPrimaryLoadBalancer: true >>> [2012-07-19 02:49:46,513] INFO - ServiceRequestsInFlightAutoscaler >>> ************ AliveMemberCount: 0 >>> [2012-07-19 02:49:46,599] WARN - ServiceRequestsInFlightAutoscaler App >>> domain Sanity check failed for [wso2.as.domain] . Current instances: 0. >>> Required instances: 1 >>> [2012-07-19 02:49:46,621] INFO - ServiceRequestsInFlightAutoscaler >>> Launching 1 App instances for domain wso2.as.domain >>> [2012-07-19 02:50:16,331] INFO - ServiceRequestsInFlightAutoscaler >>> *********** isPrimaryLoadBalancer: true >>> [2012-07-19 02:50:17,041] INFO - ServiceRequestsInFlightAutoscaler >>> ************ AliveMemberCount: 0 >>> >>> [2012-07-19 02:50:46,329] INFO - ServiceRequestsInFlightAutoscaler >>> *********** isPrimaryLoadBalancer: true >>> [2012-07-19 02:50:46,917] INFO - ServiceRequestsInFlightAutoscaler >>> ************ AliveMemberCount: 0 >>> [2012-07-19 02:51:16,330] INFO - ServiceRequestsInFlightAutoscaler >>> *********** isPrimaryLoadBalancer: true >>> [2012-07-19 02:51:17,108] INFO - ServiceRequestsInFlightAutoscaler >>> ************ AliveMemberCount: 0 >>> [2012-07-19 02:51:46,330] INFO - ServiceRequestsInFlightAutoscaler >>> *********** isPrimaryLoadBalancer: true >>> [2012-07-19 02:53:29,819] INFO - RpcMembershipRequestHandler Received >>> JOIN message from 172.16.1.2:4000(wso2.as.domain) >>> [2012-07-19 02:53:29,820] INFO - MembershipManager Application member >>> 172.16.1.2:4000(wso2.as.domain) joined group wso2.as.domain >>> [2012-07-19 02:53:40,824] INFO - DefaultGroupManagementAgent >>> Application member Host:172.16.1.2, Port: 4000, HTTP:9763, HTTPS:9443, >>> ACTIVE:true joined application cluster >>> >>> [2012-07-19 03:03:27,179] INFO - ServiceRequestsInFlightAutoscaler >>> ************ AliveMemberCount: 0 >>> [2012-07-19 03:03:27,818] WARN - ServiceRequestsInFlightAutoscaler App >>> domain Sanity check failed for [wso2.as.domain] . Current instances: 0. >>> Required instances: 1 >>> [2012-07-19 03:03:27,831] INFO - ServiceRequestsInFlightAutoscaler >>> Launching 1 App instances for domain wso2.as.domain >>> [2012-07-19 03:03:46,329] INFO - ServiceRequestsInFlightAutoscaler >>> *********** isPrimaryLoadBalancer: true >>> [2012-07-19 03:04:16,181] INFO - ServiceRequestsInFlightAutoscaler >>> ************ AliveMemberCount: 0 >>> [2012-07-19 03:04:46,330] INFO - ServiceRequestsInFlightAutoscaler >>> *********** isPrimaryLoadBalancer: true >>> [2012-07-19 03:04:52,975] INFO - ServiceRequestsInFlightAutoscaler >>> ************ AliveMemberCount: 0 >>> [2012-07-19 03:05:16,329] INFO - ServiceRequestsInFlightAutoscaler >>> *********** isPrimaryLoadBalancer: true >>> >>> >>> Here, at [2012-07-19 03:03:27,818] the sanity check has been failed.. >>> can it be expected? >>> >> >> Yes, If the node that got joined moment ago, disappears suddenly. >> >> *NOTE:* There can and possibly be issues with the autoscaler task, which >> is not yet tested extensively and will be tested while I get a chance to do >> perf tests. But with these kind of weird behaviours with instances starting >> up in openstack, I don't think we can come to conclusions on the behaviour >> of Autoscaler task. >> >> Further I can't see this weird behaviours in EC2 case, and there I'll try >> to test and fix issues of autoscaler task, till we get openstack issues >> sorted out. >> >> >> i mean, by that time node 172.16.1.2:4000 has joined the LB, >>> >>> LB config is as follows... >>> >>> defaults { >>> >>> min_app_instances 1; >>> >>> max_app_instances 5; >>> >>> queue_length_per_node 3; >>> >>> rounds_to_average 2; >>> >>> instances_per_scale_up 1; >>> >>> message_expiry_time 60000; >>> >>> } >>> >>> >>> Thereafter, I started an app server outside Openstack and aftert some >>> time a termination request was sent and an instance was successfully >>> terminated .. >>> >>> logs are attached for reference .. >>> >>> On Thu, Jul 19, 2012 at 2:47 AM, Damitha Kumarage <[email protected]>wrote: >>> >>>> >>>> >>>> ---------- Forwarded message ---------- >>>> From: Damitha Kumarage <[email protected]> >>>> Date: Thu, Jul 19, 2012 at 2:45 AM >>>> Subject: lxc demo >>>> To: wso2 <[email protected]> >>>> >>>> >>>> Hi Azeez, >>>> >>>> The lxc demo seems to be still lagging. Although container join the LB >>>> there are some failures in some of the clustering messages due to timeouts >>>> specific to openstack. >>>> We suspect, this is due to memory constraints in virtualbox instances >>>> we use. We are trying running LB and autoscaler in laptop instead of >>>> virtualboxes. >>>> >>>> Nirmal has a working setup where LB and autoscaler run in a wso2 public >>>> machine and burst into EC2 when load increase. Shall we do this demo as >>>> backup? >>>> >>>> Thanks, >>>> Damitha >>>> >>>> -- >>>> __________________________________________________________________ >>>> Damitha Kumarage >>>> Senior Technical Lead; WSO2 Inc. >>>> Oxygenating the Web Service Platform >>>> http://www.wso2.com/ >>>> blog: >>>> http://damithakumarage.wordpress.com/ >>>> __________________________________________________________________ >>>> >>>> >>>> >>>> -- >>>> __________________________________________________________________ >>>> Damitha Kumarage >>>> Senior Technical Lead; WSO2 Inc. >>>> Oxygenating the Web Service Platform >>>> http://www.wso2.com/ >>>> blog: >>>> http://damithakumarage.wordpress.com/ >>>> __________________________________________________________________ >>>> >>> >>> >>> >>> -- >>> Regards, >>> * >>> * >>> *Sajith Kariyawasam* >>> *Senior Software Engineer; WSO2, Inc.; http://wso2.com* >>> >>> >> >> >> -- >> >> Thanks & regards, >> Nirmal >> >> Software Engineer- Platform Technologies Team, WSO2 Inc. >> Mobile: +94715779733 >> Blog: http://nirmalfdo.blogspot.com/ >> > > > > -- > __________________________________________________________________ > Damitha Kumarage > Senior Technical Lead; WSO2 Inc. > Oxygenating the Web Service Platform > http://www.wso2.com/ > blog: > http://damithakumarage.wordpress.com/ > __________________________________________________________________ > -- __________________________________________________________________ Damitha Kumarage Senior Technical Lead; WSO2 Inc. Oxygenating the Web Service Platform http://www.wso2.com/ blog: http://damithakumarage.wordpress.com/ __________________________________________________________________
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
