Hi All, Did this fix included to last release? I'm getting exact same issue in API Manager 1.10 with high concurrency. Getting exact same NPE mentioned in below JIRA[1]
[1]https://wso2.org/jira/browse/ESBJAVA-4142 Thanks, sanjeewa. On Wed, Oct 14, 2015 at 11:09 AM, Jagath Sisirakumara Ariyarathne < [email protected]> wrote: > Hi Nuwan, > > Our plan is to do the first milestone release end of next week. With this > we hope to incorporate latest carbon-deployment and kernel releases (if it > released during this week). > > Thanks. > > On Wed, Oct 14, 2015 at 11:07 AM, Chanaka Fernando <[email protected]> > wrote: > >> [Adding Jagath] >> >> On Wed, Oct 14, 2015 at 10:17 AM, Nuwan Dias <[email protected]> wrote: >> >>> Hi ESB folks, >>> >>> Can we get a milestone release of synapse and carbon-mediation including >>> this fix and another PR I sent to synapse? We have to do a milestone >>> release of API Manager this week. >>> >>> Thanks, >>> NuwanD. >>> >>> On Mon, Oct 12, 2015 at 11:43 AM, Nuwan Dias <[email protected]> wrote: >>> >>>> Thanks a lot Chanaka. >>>> >>>> On Mon, Oct 12, 2015 at 11:39 AM, Chanaka Fernando <[email protected]> >>>> wrote: >>>> >>>>> Hi All, >>>>> >>>>> This issue is fixed with the following PR [1]. Thanks for reporting >>>>> and helping us to figure out the issue. >>>>> >>>>> [1] https://github.com/wso2/carbon-mediation/pull/467 >>>>> >>>>> On Fri, Oct 9, 2015 at 3:41 PM, Chanaka Fernando <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Sameera, >>>>>> >>>>>> Your judgment is correct and it helped us to find the root cause :). >>>>>> I'll fix the code from carbon-mediation side. Thanks for your input. >>>>>> >>>>>> On Fri, Oct 9, 2015 at 3:19 PM, Sameera Jayasoma <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi Chanaka, >>>>>>> >>>>>>> You don't need to destroy the current context. This part is handle >>>>>>> by the Kernel code. Users of the CarbonContext API do not need to worry >>>>>>> about destroying the context. >>>>>>> >>>>>>> Who has done this change? So my initial judgment is correct. :) >>>>>>> Stack cannot become empty. >>>>>>> >>>>>>> Thanks, >>>>>>> Sameera. >>>>>>> >>>>>>> On Thu, Oct 8, 2015 at 6:14 PM, Chanaka Fernando <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> At last, I was able to find the root cause for this behavior. This >>>>>>>> is actually coming from carbon-mediation code. What actually happens is >>>>>>>> that when ESB starts with a tenant, it will not load the tenant until >>>>>>>> it >>>>>>>> receives the first request. When It receives the first request, it will >>>>>>>> call the MultitenantMessageReceiver.processRequest() method. Within >>>>>>>> this >>>>>>>> method, it will call the following method. >>>>>>>> >>>>>>>> PrivilegedCarbonContext.startTenantFlow(); >>>>>>>> >>>>>>>> >>>>>>>> This will get the ThreadLocal variable *parentContextHolderStack *and >>>>>>>> get the stack and push the carbonContextDataHolder object to the >>>>>>>> stack. After this, tenant loading happens and within the >>>>>>>> initialization of >>>>>>>> the carbon mediation registry we have the following code segment. >>>>>>>> >>>>>>>> *org.wso2.carbon.mediation.registry.WSO2Registry.java* >>>>>>>> >>>>>>>> /** >>>>>>>> * Carbon Kernel mandates to set Threadlocal before calling anything >>>>>>>> in kernel >>>>>>>> */ >>>>>>>> private void setTenantInfo() { >>>>>>>> // Preserve user name >>>>>>>> String username = >>>>>>>> PrivilegedCarbonContext.getThreadLocalCarbonContext().getUsername(); >>>>>>>> *PrivilegedCarbonContext.destroyCurrentContext();* >>>>>>>> PrivilegedCarbonContext cc = >>>>>>>> PrivilegedCarbonContext.getThreadLocalCarbonContext(); >>>>>>>> cc.setTenantDomain(domain); >>>>>>>> cc.setTenantId(tenantId); >>>>>>>> if (username != null) { // Set back the user name >>>>>>>> >>>>>>>> PrivilegedCarbonContext.getThreadLocalCarbonContext().setUsername(username); >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>>>> Within the above method following line causes the issue. >>>>>>>> >>>>>>>> PrivilegedCarbonContext.destroyCurrentContext(); >>>>>>>> >>>>>>>> When this method is called, it will reset the *parentContextHolderStack >>>>>>>> *and the initial object push into the stack is destroyed. Then >>>>>>>> after tenant loading, within the >>>>>>>> MultitenantMessageReceiver.processRequest() method, it tries to end the >>>>>>>> tenant flow within finally block and try to pop the object which it >>>>>>>> pushes >>>>>>>> early. But right now, we have a new context stack and it will throw the >>>>>>>> emptyStackException due to that. >>>>>>>> >>>>>>>> @Sameera/KasunG: Do we really need to use the following code >>>>>>>> segment within the above method? >>>>>>>> >>>>>>>> >>>>>>>> *PrivilegedCarbonContext.destroyCurrentContext();* >>>>>>>> >>>>>>>> I saw this method called within carbon-mediation components in 3 >>>>>>>> different locations. AFAIU, we don't need to destroy the current >>>>>>>> context >>>>>>>> when we are accessing the ThreadLocalContext. Please share your >>>>>>>> thoughts >>>>>>>> such that we can fix this issue. Issue is fixed when I comment out the >>>>>>>> above line :) >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Chanaka >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Oct 7, 2015 at 5:34 PM, Chanaka Fernando <[email protected] >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> Hi Sameera/KasunG, >>>>>>>>> >>>>>>>>> I have debugged the code to find the root cause for this empty >>>>>>>>> stack exception. This is causing several other issues at the ESB >>>>>>>>> layer. >>>>>>>>> What actually happens is that inside the >>>>>>>>> MultitenantMessageReceiver.processRequest() method, we have the >>>>>>>>> following >>>>>>>>> code segment. >>>>>>>>> >>>>>>>>> try { >>>>>>>>> PrivilegedCarbonContext.startTenantFlow(); >>>>>>>>> PrivilegedCarbonContext privilegedCarbonContext = >>>>>>>>> PrivilegedCarbonContext.getThreadLocalCarbonContext(); >>>>>>>>> privilegedCarbonContext.setTenantDomain(tenantDomain, true); >>>>>>>>> // this is to prevent non-blocking transports from sending 202 >>>>>>>>> >>>>>>>>> mainInMsgContext.getOperationContext().setProperty(Constants.RESPONSE_WRITTEN, >>>>>>>>> "SKIP"); >>>>>>>>> >>>>>>>>> ConfigurationContext tenantConfigCtx = >>>>>>>>> >>>>>>>>> TenantAxisUtils.getTenantConfigurationContext(tenantDomain, >>>>>>>>> >>>>>>>>> mainConfigCtx); >>>>>>>>> if (tenantConfigCtx == null) { >>>>>>>>> // Throw AxisFault: Tenant does not exist >>>>>>>>> handleException(mainInMsgContext, new AxisFault("Tenant " + >>>>>>>>> tenantDomain + >>>>>>>>> " not >>>>>>>>> found")); >>>>>>>>> return; >>>>>>>>> } >>>>>>>>> >>>>>>>>> if (mainInMsgContext.isDoingREST()) { // Handle REST requests >>>>>>>>> doREST(mainInMsgContext, to, tenantDomain, tenantConfigCtx, >>>>>>>>> serviceAndOperation); >>>>>>>>> } else { >>>>>>>>> doSOAP(mainInMsgContext, tenantDomain, tenantConfigCtx, >>>>>>>>> serviceAndOperation); >>>>>>>>> } >>>>>>>>> } finally { >>>>>>>>> PrivilegedCarbonContext.endTenantFlow(); >>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> When we are calling the endTenantFlow() method, it will go inside >>>>>>>>> the following method within the CarbonContextDataHolder class. >>>>>>>>> >>>>>>>>> /** >>>>>>>>> * This will end the tenant flow and restore the previous >>>>>>>>> CarbonContext. >>>>>>>>> */ >>>>>>>>> public void endTenantFlow() { >>>>>>>>> Stack<CarbonContextDataHolder> carbonContextDataHolders = >>>>>>>>> parentContextHolderStack.get(); >>>>>>>>> if (carbonContextDataHolders != null) { >>>>>>>>> currentContextHolder.set(carbonContextDataHolders.pop()); >>>>>>>>> } >>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> At the time of calling this method carbonContextDataHolders stack >>>>>>>>> has become empty and causing the EmptyStackException. When I debug the >>>>>>>>> code, I found that there is a scheduled task running in a different >>>>>>>>> thread >>>>>>>>> and accessing the same method frequently. This is coming from >>>>>>>>> AbstractQuartzTaskManager class method. >>>>>>>>> >>>>>>>>> public void triggerComplete(Trigger trigger, JobExecutionContext >>>>>>>>> jobExecutionContext, >>>>>>>>> Trigger.CompletedExecutionInstruction >>>>>>>>> completedExecutionInstruction) { >>>>>>>>> PrivilegedCarbonContext.startTenantFlow(); >>>>>>>>> >>>>>>>>> PrivilegedCarbonContext.getThreadLocalCarbonContext().setTenantId(getTenantId(), >>>>>>>>> true); >>>>>>>>> if(trigger.getNextFireTime() == null) { >>>>>>>>> try { >>>>>>>>> TaskUtils.setTaskFinished(getTaskRepository(), >>>>>>>>> trigger.getJobKey().getName(), true); >>>>>>>>> } catch (TaskException e) { >>>>>>>>> log.error("Error in Finishing Task [" + >>>>>>>>> trigger.getJobKey().getName() + >>>>>>>>> "]: " + e.getMessage(), e); >>>>>>>>> } >>>>>>>>> } >>>>>>>>> PrivilegedCarbonContext.endTenantFlow(); >>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> What I could not figure out is the thread which is emptying the >>>>>>>>> stack. Is it possible that a different thread can access this >>>>>>>>> carbonContextDataHolders stack and popping out the element before the >>>>>>>>> PassThroughMessageProcessor thread access the same? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Jul 8, 2015 at 11:12 AM, Jagath Sisirakumara Ariyarathne < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi Sameera, >>>>>>>>>> >>>>>>>>>> I will check it and update. >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>> >>>>>>>>>> On Wed, Jul 8, 2015 at 11:10 AM, Sameera Jayasoma < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Jagath, >>>>>>>>>>> >>>>>>>>>>> Can you debug and see whey the stack becomes empty? Thats a >>>>>>>>>>> serious problem. Stack should be become empty here. >>>>>>>>>>> >>>>>>>>>>> Checking whether the stack is empty will stop the error log, but >>>>>>>>>>> it doesn't fix the actual problem here. >>>>>>>>>>> >>>>>>>>>>> On Wed, Jul 8, 2015 at 10:50 AM, Sameera Jayasoma < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> I understand, but we need to understand why that stack becomes >>>>>>>>>>>> empty. AFAIK, if we follow the proper APIs, stack should not >>>>>>>>>>>> become empty >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Jul 7, 2015 at 5:47 PM, Kasun Indrasiri <[email protected] >>>>>>>>>>>> > wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Yeah, we should always check for an empty stack. >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Jul 7, 2015 at 5:17 PM, Malaka Silva <[email protected]> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> I think we need to check isEmpty as well. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, Jul 7, 2015 at 3:41 PM, Jagath Sisirakumara >>>>>>>>>>>>>> Ariyarathne <[email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I am working on [1] and found that the cause of the >>>>>>>>>>>>>>> exception mentioned below is in the code segment in org >>>>>>>>>>>>>>> .wso2.carbon.context.internal.CarbonContextDataHolder in >>>>>>>>>>>>>>> carbon.utils. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> public void endTenantFlow() { >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Stack<CarbonContextDataHolder> carbonContextDataHolders = >>>>>>>>>>>>>>> parentContextHolderStack.get(); >>>>>>>>>>>>>>> if (carbonContextDataHolders != null) { >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> currentContextHolder.set(carbonContextDataHolders.pop()); >>>>>>>>>>>>>>> } >>>>>>>>>>>>>>> } >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> *Exception :* >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> java.util.EmptyStackException >>>>>>>>>>>>>>> at java.util.Stack.peek(Stack.java:102) >>>>>>>>>>>>>>> at java.util.Stack.pop(Stack.java:84) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.wso2.carbon.context.internal.CarbonContextDataHolder.endTenantFlow(CarbonContextDataHolder.java:1291) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.wso2.carbon.context.PrivilegedCarbonContext.endTenantFlow(PrivilegedCarbonContext.java:75) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute(TaskQuartzJobAdapter.java:69) >>>>>>>>>>>>>>> at org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Issue occurs when it tries to pop elements from >>>>>>>>>>>>>>> carbonContextDataHolders stack when it is empty. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Is it a possible scenario that this stack being empty and >>>>>>>>>>>>>>> shouldn't it be handled at CarbonContextDataHolder (check >>>>>>>>>>>>>>> isEmpty in stack)? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1] - https://wso2.org/jira/browse/ESBJAVA-3832 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Jagath Ariyarathne >>>>>>>>>>>>>>> Technical Lead >>>>>>>>>>>>>>> WSO2 Inc. http://wso2.com/ >>>>>>>>>>>>>>> Email: [email protected] >>>>>>>>>>>>>>> Mob : +94 77 386 7048 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> >>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Malaka Silva >>>>>>>>>>>>>> Senior Tech Lead >>>>>>>>>>>>>> M: +94 777 219 791 >>>>>>>>>>>>>> Tel : 94 11 214 5345 >>>>>>>>>>>>>> Fax :94 11 2145300 >>>>>>>>>>>>>> Skype : malaka.sampath.silva >>>>>>>>>>>>>> LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77 >>>>>>>>>>>>>> Blog : http://mrmalakasilva.blogspot.com/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> WSO2, Inc. >>>>>>>>>>>>>> lean . enterprise . middleware >>>>>>>>>>>>>> http://www.wso2.com/ >>>>>>>>>>>>>> http://www.wso2.com/about/team/malaka-silva/ >>>>>>>>>>>>>> <http://wso2.com/about/team/malaka-silva/> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Save a tree -Conserve nature & Save the world for your >>>>>>>>>>>>>> future. Print this email only if it is absolutely necessary. >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> Dev mailing list >>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Kasun Indrasiri >>>>>>>>>>>>> Software Architect >>>>>>>>>>>>> WSO2, Inc.; http://wso2.com >>>>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>>>> >>>>>>>>>>>>> cell: +94 77 556 5206 >>>>>>>>>>>>> Blog : http://kasunpanorama.blogspot.com/ >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Dev mailing list >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Sameera Jayasoma, >>>>>>>>>>>> Software Architect, >>>>>>>>>>>> >>>>>>>>>>>> WSO2, Inc. (http://wso2.com) >>>>>>>>>>>> email: [email protected] >>>>>>>>>>>> blog: http://blog.sameera.org >>>>>>>>>>>> twitter: https://twitter.com/sameerajayasoma >>>>>>>>>>>> flickr: >>>>>>>>>>>> http://www.flickr.com/photos/sameera-jayasoma/collections >>>>>>>>>>>> Mobile: 0094776364456 >>>>>>>>>>>> >>>>>>>>>>>> Lean . Enterprise . Middleware >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Sameera Jayasoma, >>>>>>>>>>> Software Architect, >>>>>>>>>>> >>>>>>>>>>> WSO2, Inc. (http://wso2.com) >>>>>>>>>>> email: [email protected] >>>>>>>>>>> blog: http://blog.sameera.org >>>>>>>>>>> twitter: https://twitter.com/sameerajayasoma >>>>>>>>>>> flickr: >>>>>>>>>>> http://www.flickr.com/photos/sameera-jayasoma/collections >>>>>>>>>>> Mobile: 0094776364456 >>>>>>>>>>> >>>>>>>>>>> Lean . Enterprise . Middleware >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Dev mailing list >>>>>>>>>>> [email protected] >>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Jagath Ariyarathne >>>>>>>>>> Technical Lead >>>>>>>>>> WSO2 Inc. http://wso2.com/ >>>>>>>>>> Email: [email protected] >>>>>>>>>> Mob : +94 77 386 7048 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Dev mailing list >>>>>>>>>> [email protected] >>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> -- >>>>>>>>> Chanaka Fernando >>>>>>>>> Senior Technical Lead >>>>>>>>> WSO2, Inc.; http://wso2.com >>>>>>>>> lean.enterprise.middleware >>>>>>>>> >>>>>>>>> mobile: +94 773337238 >>>>>>>>> Blog : http://soatutorials.blogspot.com >>>>>>>>> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0 >>>>>>>>> Twitter:https://twitter.com/chanakaudaya >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> -- >>>>>>>> Chanaka Fernando >>>>>>>> Senior Technical Lead >>>>>>>> WSO2, Inc.; http://wso2.com >>>>>>>> lean.enterprise.middleware >>>>>>>> >>>>>>>> mobile: +94 773337238 >>>>>>>> Blog : http://soatutorials.blogspot.com >>>>>>>> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0 >>>>>>>> Twitter:https://twitter.com/chanakaudaya >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Sameera Jayasoma, >>>>>>> Software Architect, >>>>>>> >>>>>>> WSO2, Inc. (http://wso2.com) >>>>>>> email: [email protected] >>>>>>> blog: http://blog.sameera.org >>>>>>> twitter: https://twitter.com/sameerajayasoma >>>>>>> flickr: http://www.flickr.com/photos/sameera-jayasoma/collections >>>>>>> Mobile: 0094776364456 >>>>>>> >>>>>>> Lean . Enterprise . Middleware >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> -- >>>>>> Chanaka Fernando >>>>>> Senior Technical Lead >>>>>> WSO2, Inc.; http://wso2.com >>>>>> lean.enterprise.middleware >>>>>> >>>>>> mobile: +94 773337238 >>>>>> Blog : http://soatutorials.blogspot.com >>>>>> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0 >>>>>> Twitter:https://twitter.com/chanakaudaya >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> -- >>>>> Chanaka Fernando >>>>> Senior Technical Lead >>>>> WSO2, Inc.; http://wso2.com >>>>> lean.enterprise.middleware >>>>> >>>>> mobile: +94 773337238 >>>>> Blog : http://soatutorials.blogspot.com >>>>> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0 >>>>> Twitter:https://twitter.com/chanakaudaya >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Dev mailing list >>>>> [email protected] >>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>> >>>>> >>>> >>>> >>>> -- >>>> Nuwan Dias >>>> >>>> Technical Lead - WSO2, Inc. http://wso2.com >>>> email : [email protected] >>>> Phone : +94 777 775 729 >>>> >>> >>> >>> >>> -- >>> Nuwan Dias >>> >>> Technical Lead - WSO2, Inc. http://wso2.com >>> email : [email protected] >>> Phone : +94 777 775 729 >>> >> >> >> >> -- >> -- >> Chanaka Fernando >> Senior Technical Lead >> WSO2, Inc.; http://wso2.com >> lean.enterprise.middleware >> >> mobile: +94 773337238 >> Blog : http://soatutorials.blogspot.com >> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0 >> Twitter:https://twitter.com/chanakaudaya >> >> >> >> >> > > > -- > Jagath Ariyarathne > Technical Lead > WSO2 Inc. http://wso2.com/ > Email: [email protected] > Mob : +94 77 386 7048 > > > _______________________________________________ > Dev mailing list > [email protected] > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- *Sanjeewa Malalgoda* WSO2 Inc. Mobile : +94713068779 <http://sanjeewamalalgoda.blogspot.com/>blog :http://sanjeewamalalgoda.blogspot.com/ <http://sanjeewamalalgoda.blogspot.com/>
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
