[Adding Jagath] On Wed, Oct 14, 2015 at 10:17 AM, Nuwan Dias <[email protected]> wrote:
> Hi ESB folks, > > Can we get a milestone release of synapse and carbon-mediation including > this fix and another PR I sent to synapse? We have to do a milestone > release of API Manager this week. > > Thanks, > NuwanD. > > On Mon, Oct 12, 2015 at 11:43 AM, Nuwan Dias <[email protected]> wrote: > >> Thanks a lot Chanaka. >> >> On Mon, Oct 12, 2015 at 11:39 AM, Chanaka Fernando <[email protected]> >> wrote: >> >>> Hi All, >>> >>> This issue is fixed with the following PR [1]. Thanks for reporting and >>> helping us to figure out the issue. >>> >>> [1] https://github.com/wso2/carbon-mediation/pull/467 >>> >>> On Fri, Oct 9, 2015 at 3:41 PM, Chanaka Fernando <[email protected]> >>> wrote: >>> >>>> Hi Sameera, >>>> >>>> Your judgment is correct and it helped us to find the root cause :). >>>> I'll fix the code from carbon-mediation side. Thanks for your input. >>>> >>>> On Fri, Oct 9, 2015 at 3:19 PM, Sameera Jayasoma <[email protected]> >>>> wrote: >>>> >>>>> Hi Chanaka, >>>>> >>>>> You don't need to destroy the current context. This part is handle by >>>>> the Kernel code. Users of the CarbonContext API do not need to worry about >>>>> destroying the context. >>>>> >>>>> Who has done this change? So my initial judgment is correct. :) Stack >>>>> cannot become empty. >>>>> >>>>> Thanks, >>>>> Sameera. >>>>> >>>>> On Thu, Oct 8, 2015 at 6:14 PM, Chanaka Fernando <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi All, >>>>>> >>>>>> At last, I was able to find the root cause for this behavior. This is >>>>>> actually coming from carbon-mediation code. What actually happens is that >>>>>> when ESB starts with a tenant, it will not load the tenant until it >>>>>> receives the first request. When It receives the first request, it will >>>>>> call the MultitenantMessageReceiver.processRequest() method. Within this >>>>>> method, it will call the following method. >>>>>> >>>>>> PrivilegedCarbonContext.startTenantFlow(); >>>>>> >>>>>> >>>>>> This will get the ThreadLocal variable *parentContextHolderStack *and >>>>>> get the stack and push the carbonContextDataHolder object to the >>>>>> stack. After this, tenant loading happens and within the initialization >>>>>> of >>>>>> the carbon mediation registry we have the following code segment. >>>>>> >>>>>> *org.wso2.carbon.mediation.registry.WSO2Registry.java* >>>>>> >>>>>> /** >>>>>> * Carbon Kernel mandates to set Threadlocal before calling anything in >>>>>> kernel >>>>>> */ >>>>>> private void setTenantInfo() { >>>>>> // Preserve user name >>>>>> String username = >>>>>> PrivilegedCarbonContext.getThreadLocalCarbonContext().getUsername(); >>>>>> *PrivilegedCarbonContext.destroyCurrentContext();* >>>>>> PrivilegedCarbonContext cc = >>>>>> PrivilegedCarbonContext.getThreadLocalCarbonContext(); >>>>>> cc.setTenantDomain(domain); >>>>>> cc.setTenantId(tenantId); >>>>>> if (username != null) { // Set back the user name >>>>>> >>>>>> PrivilegedCarbonContext.getThreadLocalCarbonContext().setUsername(username); >>>>>> } >>>>>> } >>>>>> >>>>>> Within the above method following line causes the issue. >>>>>> >>>>>> PrivilegedCarbonContext.destroyCurrentContext(); >>>>>> >>>>>> When this method is called, it will reset the *parentContextHolderStack >>>>>> *and the initial object push into the stack is destroyed. Then after >>>>>> tenant loading, within the MultitenantMessageReceiver.processRequest() >>>>>> method, it tries to end the tenant flow within finally block and try to >>>>>> pop >>>>>> the object which it pushes early. But right now, we have a new context >>>>>> stack and it will throw the emptyStackException due to that. >>>>>> >>>>>> @Sameera/KasunG: Do we really need to use the following code segment >>>>>> within the above method? >>>>>> >>>>>> >>>>>> *PrivilegedCarbonContext.destroyCurrentContext();* >>>>>> >>>>>> I saw this method called within carbon-mediation components in 3 >>>>>> different locations. AFAIU, we don't need to destroy the current context >>>>>> when we are accessing the ThreadLocalContext. Please share your thoughts >>>>>> such that we can fix this issue. Issue is fixed when I comment out the >>>>>> above line :) >>>>>> >>>>>> >>>>>> Thanks, >>>>>> Chanaka >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Oct 7, 2015 at 5:34 PM, Chanaka Fernando <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi Sameera/KasunG, >>>>>>> >>>>>>> I have debugged the code to find the root cause for this empty stack >>>>>>> exception. This is causing several other issues at the ESB layer. What >>>>>>> actually happens is that inside the >>>>>>> MultitenantMessageReceiver.processRequest() method, we have the >>>>>>> following >>>>>>> code segment. >>>>>>> >>>>>>> try { >>>>>>> PrivilegedCarbonContext.startTenantFlow(); >>>>>>> PrivilegedCarbonContext privilegedCarbonContext = >>>>>>> PrivilegedCarbonContext.getThreadLocalCarbonContext(); >>>>>>> privilegedCarbonContext.setTenantDomain(tenantDomain, true); >>>>>>> // this is to prevent non-blocking transports from sending 202 >>>>>>> >>>>>>> mainInMsgContext.getOperationContext().setProperty(Constants.RESPONSE_WRITTEN, >>>>>>> "SKIP"); >>>>>>> >>>>>>> ConfigurationContext tenantConfigCtx = >>>>>>> >>>>>>> TenantAxisUtils.getTenantConfigurationContext(tenantDomain, >>>>>>> >>>>>>> mainConfigCtx); >>>>>>> if (tenantConfigCtx == null) { >>>>>>> // Throw AxisFault: Tenant does not exist >>>>>>> handleException(mainInMsgContext, new AxisFault("Tenant " + >>>>>>> tenantDomain + >>>>>>> " not found")); >>>>>>> return; >>>>>>> } >>>>>>> >>>>>>> if (mainInMsgContext.isDoingREST()) { // Handle REST requests >>>>>>> doREST(mainInMsgContext, to, tenantDomain, tenantConfigCtx, >>>>>>> serviceAndOperation); >>>>>>> } else { >>>>>>> doSOAP(mainInMsgContext, tenantDomain, tenantConfigCtx, >>>>>>> serviceAndOperation); >>>>>>> } >>>>>>> } finally { >>>>>>> PrivilegedCarbonContext.endTenantFlow(); >>>>>>> } >>>>>>> >>>>>>> >>>>>>> When we are calling the endTenantFlow() method, it will go inside >>>>>>> the following method within the CarbonContextDataHolder class. >>>>>>> >>>>>>> /** >>>>>>> * This will end the tenant flow and restore the previous CarbonContext. >>>>>>> */ >>>>>>> public void endTenantFlow() { >>>>>>> Stack<CarbonContextDataHolder> carbonContextDataHolders = >>>>>>> parentContextHolderStack.get(); >>>>>>> if (carbonContextDataHolders != null) { >>>>>>> currentContextHolder.set(carbonContextDataHolders.pop()); >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> >>>>>>> At the time of calling this method carbonContextDataHolders stack >>>>>>> has become empty and causing the EmptyStackException. When I debug the >>>>>>> code, I found that there is a scheduled task running in a different >>>>>>> thread >>>>>>> and accessing the same method frequently. This is coming from >>>>>>> AbstractQuartzTaskManager class method. >>>>>>> >>>>>>> public void triggerComplete(Trigger trigger, JobExecutionContext >>>>>>> jobExecutionContext, >>>>>>> Trigger.CompletedExecutionInstruction >>>>>>> completedExecutionInstruction) { >>>>>>> PrivilegedCarbonContext.startTenantFlow(); >>>>>>> >>>>>>> PrivilegedCarbonContext.getThreadLocalCarbonContext().setTenantId(getTenantId(), >>>>>>> true); >>>>>>> if(trigger.getNextFireTime() == null) { >>>>>>> try { >>>>>>> TaskUtils.setTaskFinished(getTaskRepository(), >>>>>>> trigger.getJobKey().getName(), true); >>>>>>> } catch (TaskException e) { >>>>>>> log.error("Error in Finishing Task [" + >>>>>>> trigger.getJobKey().getName() + >>>>>>> "]: " + e.getMessage(), e); >>>>>>> } >>>>>>> } >>>>>>> PrivilegedCarbonContext.endTenantFlow(); >>>>>>> } >>>>>>> >>>>>>> >>>>>>> What I could not figure out is the thread which is emptying the >>>>>>> stack. Is it possible that a different thread can access this >>>>>>> carbonContextDataHolders stack and popping out the element before the >>>>>>> PassThroughMessageProcessor thread access the same? >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Jul 8, 2015 at 11:12 AM, Jagath Sisirakumara Ariyarathne < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi Sameera, >>>>>>>> >>>>>>>> I will check it and update. >>>>>>>> >>>>>>>> Thanks. >>>>>>>> >>>>>>>> On Wed, Jul 8, 2015 at 11:10 AM, Sameera Jayasoma <[email protected] >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> Hi Jagath, >>>>>>>>> >>>>>>>>> Can you debug and see whey the stack becomes empty? Thats a >>>>>>>>> serious problem. Stack should be become empty here. >>>>>>>>> >>>>>>>>> Checking whether the stack is empty will stop the error log, but >>>>>>>>> it doesn't fix the actual problem here. >>>>>>>>> >>>>>>>>> On Wed, Jul 8, 2015 at 10:50 AM, Sameera Jayasoma < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> I understand, but we need to understand why that stack becomes >>>>>>>>>> empty. AFAIK, if we follow the proper APIs, stack should not become >>>>>>>>>> empty >>>>>>>>>> >>>>>>>>>> On Tue, Jul 7, 2015 at 5:47 PM, Kasun Indrasiri <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Yeah, we should always check for an empty stack. >>>>>>>>>>> >>>>>>>>>>> On Tue, Jul 7, 2015 at 5:17 PM, Malaka Silva <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> I think we need to check isEmpty as well. >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Jul 7, 2015 at 3:41 PM, Jagath Sisirakumara Ariyarathne >>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> I am working on [1] and found that the cause of the exception >>>>>>>>>>>>> mentioned below is in the code segment in org >>>>>>>>>>>>> .wso2.carbon.context.internal.CarbonContextDataHolder in >>>>>>>>>>>>> carbon.utils. >>>>>>>>>>>>> >>>>>>>>>>>>> public void endTenantFlow() { >>>>>>>>>>>>> >>>>>>>>>>>>> Stack<CarbonContextDataHolder> carbonContextDataHolders = >>>>>>>>>>>>> parentContextHolderStack.get(); >>>>>>>>>>>>> if (carbonContextDataHolders != null) { >>>>>>>>>>>>> currentContextHolder.set(carbonContextDataHolders.pop()); >>>>>>>>>>>>> } >>>>>>>>>>>>> } >>>>>>>>>>>>> >>>>>>>>>>>>> *Exception :* >>>>>>>>>>>>> >>>>>>>>>>>>> java.util.EmptyStackException >>>>>>>>>>>>> at java.util.Stack.peek(Stack.java:102) >>>>>>>>>>>>> at java.util.Stack.pop(Stack.java:84) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.wso2.carbon.context.internal.CarbonContextDataHolder.endTenantFlow(CarbonContextDataHolder.java:1291) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.wso2.carbon.context.PrivilegedCarbonContext.endTenantFlow(PrivilegedCarbonContext.java:75) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute(TaskQuartzJobAdapter.java:69) >>>>>>>>>>>>> at org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>>> at >>>>>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>>>>>>>>>> at >>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>>>>>>>> at >>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>>>>>> >>>>>>>>>>>>> Issue occurs when it tries to pop elements from >>>>>>>>>>>>> carbonContextDataHolders stack when it is empty. >>>>>>>>>>>>> >>>>>>>>>>>>> Is it a possible scenario that this stack being empty and >>>>>>>>>>>>> shouldn't it be handled at CarbonContextDataHolder (check isEmpty >>>>>>>>>>>>> in stack)? >>>>>>>>>>>>> >>>>>>>>>>>>> [1] - https://wso2.org/jira/browse/ESBJAVA-3832 >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Jagath Ariyarathne >>>>>>>>>>>>> Technical Lead >>>>>>>>>>>>> WSO2 Inc. http://wso2.com/ >>>>>>>>>>>>> Email: [email protected] >>>>>>>>>>>>> Mob : +94 77 386 7048 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> Best Regards, >>>>>>>>>>>> >>>>>>>>>>>> Malaka Silva >>>>>>>>>>>> Senior Tech Lead >>>>>>>>>>>> M: +94 777 219 791 >>>>>>>>>>>> Tel : 94 11 214 5345 >>>>>>>>>>>> Fax :94 11 2145300 >>>>>>>>>>>> Skype : malaka.sampath.silva >>>>>>>>>>>> LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77 >>>>>>>>>>>> Blog : http://mrmalakasilva.blogspot.com/ >>>>>>>>>>>> >>>>>>>>>>>> WSO2, Inc. >>>>>>>>>>>> lean . enterprise . middleware >>>>>>>>>>>> http://www.wso2.com/ >>>>>>>>>>>> http://www.wso2.com/about/team/malaka-silva/ >>>>>>>>>>>> <http://wso2.com/about/team/malaka-silva/> >>>>>>>>>>>> >>>>>>>>>>>> Save a tree -Conserve nature & Save the world for your future. >>>>>>>>>>>> Print this email only if it is absolutely necessary. >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Dev mailing list >>>>>>>>>>>> [email protected] >>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Kasun Indrasiri >>>>>>>>>>> Software Architect >>>>>>>>>>> WSO2, Inc.; http://wso2.com >>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>> >>>>>>>>>>> cell: +94 77 556 5206 >>>>>>>>>>> Blog : http://kasunpanorama.blogspot.com/ >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Dev mailing list >>>>>>>>>>> [email protected] >>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Sameera Jayasoma, >>>>>>>>>> Software Architect, >>>>>>>>>> >>>>>>>>>> WSO2, Inc. (http://wso2.com) >>>>>>>>>> email: [email protected] >>>>>>>>>> blog: http://blog.sameera.org >>>>>>>>>> twitter: https://twitter.com/sameerajayasoma >>>>>>>>>> flickr: http://www.flickr.com/photos/sameera-jayasoma/collections >>>>>>>>>> Mobile: 0094776364456 >>>>>>>>>> >>>>>>>>>> Lean . Enterprise . Middleware >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Sameera Jayasoma, >>>>>>>>> Software Architect, >>>>>>>>> >>>>>>>>> WSO2, Inc. (http://wso2.com) >>>>>>>>> email: [email protected] >>>>>>>>> blog: http://blog.sameera.org >>>>>>>>> twitter: https://twitter.com/sameerajayasoma >>>>>>>>> flickr: http://www.flickr.com/photos/sameera-jayasoma/collections >>>>>>>>> Mobile: 0094776364456 >>>>>>>>> >>>>>>>>> Lean . Enterprise . Middleware >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Dev mailing list >>>>>>>>> [email protected] >>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Jagath Ariyarathne >>>>>>>> Technical Lead >>>>>>>> WSO2 Inc. http://wso2.com/ >>>>>>>> Email: [email protected] >>>>>>>> Mob : +94 77 386 7048 >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Dev mailing list >>>>>>>> [email protected] >>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> -- >>>>>>> Chanaka Fernando >>>>>>> Senior Technical Lead >>>>>>> WSO2, Inc.; http://wso2.com >>>>>>> lean.enterprise.middleware >>>>>>> >>>>>>> mobile: +94 773337238 >>>>>>> Blog : http://soatutorials.blogspot.com >>>>>>> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0 >>>>>>> Twitter:https://twitter.com/chanakaudaya >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> -- >>>>>> Chanaka Fernando >>>>>> Senior Technical Lead >>>>>> WSO2, Inc.; http://wso2.com >>>>>> lean.enterprise.middleware >>>>>> >>>>>> mobile: +94 773337238 >>>>>> Blog : http://soatutorials.blogspot.com >>>>>> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0 >>>>>> Twitter:https://twitter.com/chanakaudaya >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Sameera Jayasoma, >>>>> Software Architect, >>>>> >>>>> WSO2, Inc. (http://wso2.com) >>>>> email: [email protected] >>>>> blog: http://blog.sameera.org >>>>> twitter: https://twitter.com/sameerajayasoma >>>>> flickr: http://www.flickr.com/photos/sameera-jayasoma/collections >>>>> Mobile: 0094776364456 >>>>> >>>>> Lean . Enterprise . Middleware >>>>> >>>>> >>>> >>>> >>>> -- >>>> -- >>>> Chanaka Fernando >>>> Senior Technical Lead >>>> WSO2, Inc.; http://wso2.com >>>> lean.enterprise.middleware >>>> >>>> mobile: +94 773337238 >>>> Blog : http://soatutorials.blogspot.com >>>> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0 >>>> Twitter:https://twitter.com/chanakaudaya >>>> >>>> >>>> >>>> >>>> >>> >>> >>> -- >>> -- >>> Chanaka Fernando >>> Senior Technical Lead >>> WSO2, Inc.; http://wso2.com >>> lean.enterprise.middleware >>> >>> mobile: +94 773337238 >>> Blog : http://soatutorials.blogspot.com >>> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0 >>> Twitter:https://twitter.com/chanakaudaya >>> >>> >>> >>> >>> >>> _______________________________________________ >>> Dev mailing list >>> [email protected] >>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>> >>> >> >> >> -- >> Nuwan Dias >> >> Technical Lead - WSO2, Inc. http://wso2.com >> email : [email protected] >> Phone : +94 777 775 729 >> > > > > -- > Nuwan Dias > > Technical Lead - WSO2, Inc. http://wso2.com > email : [email protected] > Phone : +94 777 775 729 > -- -- Chanaka Fernando Senior Technical Lead WSO2, Inc.; http://wso2.com lean.enterprise.middleware mobile: +94 773337238 Blog : http://soatutorials.blogspot.com LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0 Twitter:https://twitter.com/chanakaudaya
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
