Hi Sameera, Your judgment is correct and it helped us to find the root cause :). I'll fix the code from carbon-mediation side. Thanks for your input.
On Fri, Oct 9, 2015 at 3:19 PM, Sameera Jayasoma <[email protected]> wrote: > Hi Chanaka, > > You don't need to destroy the current context. This part is handle by the > Kernel code. Users of the CarbonContext API do not need to worry about > destroying the context. > > Who has done this change? So my initial judgment is correct. :) Stack > cannot become empty. > > Thanks, > Sameera. > > On Thu, Oct 8, 2015 at 6:14 PM, Chanaka Fernando <[email protected]> > wrote: > >> Hi All, >> >> At last, I was able to find the root cause for this behavior. This is >> actually coming from carbon-mediation code. What actually happens is that >> when ESB starts with a tenant, it will not load the tenant until it >> receives the first request. When It receives the first request, it will >> call the MultitenantMessageReceiver.processRequest() method. Within this >> method, it will call the following method. >> >> PrivilegedCarbonContext.startTenantFlow(); >> >> >> This will get the ThreadLocal variable *parentContextHolderStack *and >> get the stack and push the carbonContextDataHolder object to the stack. >> After this, tenant loading happens and within the initialization of the >> carbon mediation registry we have the following code segment. >> >> *org.wso2.carbon.mediation.registry.WSO2Registry.java* >> >> /** >> * Carbon Kernel mandates to set Threadlocal before calling anything in >> kernel >> */ >> private void setTenantInfo() { >> // Preserve user name >> String username = >> PrivilegedCarbonContext.getThreadLocalCarbonContext().getUsername(); >> *PrivilegedCarbonContext.destroyCurrentContext();* >> PrivilegedCarbonContext cc = >> PrivilegedCarbonContext.getThreadLocalCarbonContext(); >> cc.setTenantDomain(domain); >> cc.setTenantId(tenantId); >> if (username != null) { // Set back the user name >> >> PrivilegedCarbonContext.getThreadLocalCarbonContext().setUsername(username); >> } >> } >> >> Within the above method following line causes the issue. >> >> PrivilegedCarbonContext.destroyCurrentContext(); >> >> When this method is called, it will reset the *parentContextHolderStack *and >> the initial object push into the stack is destroyed. Then after tenant >> loading, within the MultitenantMessageReceiver.processRequest() method, it >> tries to end the tenant flow within finally block and try to pop the object >> which it pushes early. But right now, we have a new context stack and it >> will throw the emptyStackException due to that. >> >> @Sameera/KasunG: Do we really need to use the following code segment >> within the above method? >> >> >> *PrivilegedCarbonContext.destroyCurrentContext();* >> >> I saw this method called within carbon-mediation components in 3 >> different locations. AFAIU, we don't need to destroy the current context >> when we are accessing the ThreadLocalContext. Please share your thoughts >> such that we can fix this issue. Issue is fixed when I comment out the >> above line :) >> >> >> Thanks, >> Chanaka >> >> >> >> On Wed, Oct 7, 2015 at 5:34 PM, Chanaka Fernando <[email protected]> >> wrote: >> >>> Hi Sameera/KasunG, >>> >>> I have debugged the code to find the root cause for this empty stack >>> exception. This is causing several other issues at the ESB layer. What >>> actually happens is that inside the >>> MultitenantMessageReceiver.processRequest() method, we have the following >>> code segment. >>> >>> try { >>> PrivilegedCarbonContext.startTenantFlow(); >>> PrivilegedCarbonContext privilegedCarbonContext = >>> PrivilegedCarbonContext.getThreadLocalCarbonContext(); >>> privilegedCarbonContext.setTenantDomain(tenantDomain, true); >>> // this is to prevent non-blocking transports from sending 202 >>> >>> mainInMsgContext.getOperationContext().setProperty(Constants.RESPONSE_WRITTEN, >>> "SKIP"); >>> >>> ConfigurationContext tenantConfigCtx = >>> >>> TenantAxisUtils.getTenantConfigurationContext(tenantDomain, >>> >>> mainConfigCtx); >>> if (tenantConfigCtx == null) { >>> // Throw AxisFault: Tenant does not exist >>> handleException(mainInMsgContext, new AxisFault("Tenant " + >>> tenantDomain + >>> " not found")); >>> return; >>> } >>> >>> if (mainInMsgContext.isDoingREST()) { // Handle REST requests >>> doREST(mainInMsgContext, to, tenantDomain, tenantConfigCtx, >>> serviceAndOperation); >>> } else { >>> doSOAP(mainInMsgContext, tenantDomain, tenantConfigCtx, >>> serviceAndOperation); >>> } >>> } finally { >>> PrivilegedCarbonContext.endTenantFlow(); >>> } >>> >>> >>> When we are calling the endTenantFlow() method, it will go inside the >>> following method within the CarbonContextDataHolder class. >>> >>> /** >>> * This will end the tenant flow and restore the previous CarbonContext. >>> */ >>> public void endTenantFlow() { >>> Stack<CarbonContextDataHolder> carbonContextDataHolders = >>> parentContextHolderStack.get(); >>> if (carbonContextDataHolders != null) { >>> currentContextHolder.set(carbonContextDataHolders.pop()); >>> } >>> } >>> >>> >>> At the time of calling this method carbonContextDataHolders stack has >>> become empty and causing the EmptyStackException. When I debug the code, I >>> found that there is a scheduled task running in a different thread and >>> accessing the same method frequently. This is coming from >>> AbstractQuartzTaskManager class method. >>> >>> public void triggerComplete(Trigger trigger, JobExecutionContext >>> jobExecutionContext, >>> Trigger.CompletedExecutionInstruction >>> completedExecutionInstruction) { >>> PrivilegedCarbonContext.startTenantFlow(); >>> >>> PrivilegedCarbonContext.getThreadLocalCarbonContext().setTenantId(getTenantId(), >>> true); >>> if(trigger.getNextFireTime() == null) { >>> try { >>> TaskUtils.setTaskFinished(getTaskRepository(), >>> trigger.getJobKey().getName(), true); >>> } catch (TaskException e) { >>> log.error("Error in Finishing Task [" + >>> trigger.getJobKey().getName() + >>> "]: " + e.getMessage(), e); >>> } >>> } >>> PrivilegedCarbonContext.endTenantFlow(); >>> } >>> >>> >>> What I could not figure out is the thread which is emptying the stack. >>> Is it possible that a different thread can access this >>> carbonContextDataHolders stack and popping out the element before the >>> PassThroughMessageProcessor thread access the same? >>> >>> >>> >>> On Wed, Jul 8, 2015 at 11:12 AM, Jagath Sisirakumara Ariyarathne < >>> [email protected]> wrote: >>> >>>> Hi Sameera, >>>> >>>> I will check it and update. >>>> >>>> Thanks. >>>> >>>> On Wed, Jul 8, 2015 at 11:10 AM, Sameera Jayasoma <[email protected]> >>>> wrote: >>>> >>>>> Hi Jagath, >>>>> >>>>> Can you debug and see whey the stack becomes empty? Thats a serious >>>>> problem. Stack should be become empty here. >>>>> >>>>> Checking whether the stack is empty will stop the error log, but it >>>>> doesn't fix the actual problem here. >>>>> >>>>> On Wed, Jul 8, 2015 at 10:50 AM, Sameera Jayasoma <[email protected]> >>>>> wrote: >>>>> >>>>>> I understand, but we need to understand why that stack becomes >>>>>> empty. AFAIK, if we follow the proper APIs, stack should not become >>>>>> empty >>>>>> >>>>>> On Tue, Jul 7, 2015 at 5:47 PM, Kasun Indrasiri <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Yeah, we should always check for an empty stack. >>>>>>> >>>>>>> On Tue, Jul 7, 2015 at 5:17 PM, Malaka Silva <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> I think we need to check isEmpty as well. >>>>>>>> >>>>>>>> On Tue, Jul 7, 2015 at 3:41 PM, Jagath Sisirakumara Ariyarathne < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I am working on [1] and found that the cause of the exception >>>>>>>>> mentioned below is in the code segment in org >>>>>>>>> .wso2.carbon.context.internal.CarbonContextDataHolder in >>>>>>>>> carbon.utils. >>>>>>>>> >>>>>>>>> public void endTenantFlow() { >>>>>>>>> >>>>>>>>> Stack<CarbonContextDataHolder> carbonContextDataHolders = >>>>>>>>> parentContextHolderStack.get(); >>>>>>>>> if (carbonContextDataHolders != null) { >>>>>>>>> currentContextHolder.set(carbonContextDataHolders.pop()); >>>>>>>>> } >>>>>>>>> } >>>>>>>>> >>>>>>>>> *Exception :* >>>>>>>>> >>>>>>>>> java.util.EmptyStackException >>>>>>>>> at java.util.Stack.peek(Stack.java:102) >>>>>>>>> at java.util.Stack.pop(Stack.java:84) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.context.internal.CarbonContextDataHolder.endTenantFlow(CarbonContextDataHolder.java:1291) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.context.PrivilegedCarbonContext.endTenantFlow(PrivilegedCarbonContext.java:75) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute(TaskQuartzJobAdapter.java:69) >>>>>>>>> at org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>> at >>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>>>>>> at >>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>>>> at >>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>> >>>>>>>>> Issue occurs when it tries to pop elements from >>>>>>>>> carbonContextDataHolders stack when it is empty. >>>>>>>>> >>>>>>>>> Is it a possible scenario that this stack being empty and shouldn't >>>>>>>>> it be handled at CarbonContextDataHolder (check isEmpty in stack)? >>>>>>>>> >>>>>>>>> [1] - https://wso2.org/jira/browse/ESBJAVA-3832 >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Jagath Ariyarathne >>>>>>>>> Technical Lead >>>>>>>>> WSO2 Inc. http://wso2.com/ >>>>>>>>> Email: [email protected] >>>>>>>>> Mob : +94 77 386 7048 >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Best Regards, >>>>>>>> >>>>>>>> Malaka Silva >>>>>>>> Senior Tech Lead >>>>>>>> M: +94 777 219 791 >>>>>>>> Tel : 94 11 214 5345 >>>>>>>> Fax :94 11 2145300 >>>>>>>> Skype : malaka.sampath.silva >>>>>>>> LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77 >>>>>>>> Blog : http://mrmalakasilva.blogspot.com/ >>>>>>>> >>>>>>>> WSO2, Inc. >>>>>>>> lean . enterprise . middleware >>>>>>>> http://www.wso2.com/ >>>>>>>> http://www.wso2.com/about/team/malaka-silva/ >>>>>>>> <http://wso2.com/about/team/malaka-silva/> >>>>>>>> >>>>>>>> Save a tree -Conserve nature & Save the world for your future. >>>>>>>> Print this email only if it is absolutely necessary. >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Dev mailing list >>>>>>>> [email protected] >>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Kasun Indrasiri >>>>>>> Software Architect >>>>>>> WSO2, Inc.; http://wso2.com >>>>>>> lean.enterprise.middleware >>>>>>> >>>>>>> cell: +94 77 556 5206 >>>>>>> Blog : http://kasunpanorama.blogspot.com/ >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Dev mailing list >>>>>>> [email protected] >>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Sameera Jayasoma, >>>>>> Software Architect, >>>>>> >>>>>> WSO2, Inc. (http://wso2.com) >>>>>> email: [email protected] >>>>>> blog: http://blog.sameera.org >>>>>> twitter: https://twitter.com/sameerajayasoma >>>>>> flickr: http://www.flickr.com/photos/sameera-jayasoma/collections >>>>>> Mobile: 0094776364456 >>>>>> >>>>>> Lean . Enterprise . Middleware >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Sameera Jayasoma, >>>>> Software Architect, >>>>> >>>>> WSO2, Inc. (http://wso2.com) >>>>> email: [email protected] >>>>> blog: http://blog.sameera.org >>>>> twitter: https://twitter.com/sameerajayasoma >>>>> flickr: http://www.flickr.com/photos/sameera-jayasoma/collections >>>>> Mobile: 0094776364456 >>>>> >>>>> Lean . Enterprise . Middleware >>>>> >>>>> >>>>> _______________________________________________ >>>>> Dev mailing list >>>>> [email protected] >>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>> >>>>> >>>> >>>> >>>> -- >>>> Jagath Ariyarathne >>>> Technical Lead >>>> WSO2 Inc. http://wso2.com/ >>>> Email: [email protected] >>>> Mob : +94 77 386 7048 >>>> >>>> >>>> _______________________________________________ >>>> Dev mailing list >>>> [email protected] >>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>> >>>> >>> >>> >>> -- >>> -- >>> Chanaka Fernando >>> Senior Technical Lead >>> WSO2, Inc.; http://wso2.com >>> lean.enterprise.middleware >>> >>> mobile: +94 773337238 >>> Blog : http://soatutorials.blogspot.com >>> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0 >>> Twitter:https://twitter.com/chanakaudaya >>> >>> >>> >>> >>> >> >> >> -- >> -- >> Chanaka Fernando >> Senior Technical Lead >> WSO2, Inc.; http://wso2.com >> lean.enterprise.middleware >> >> mobile: +94 773337238 >> Blog : http://soatutorials.blogspot.com >> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0 >> Twitter:https://twitter.com/chanakaudaya >> >> >> >> >> > > > -- > Sameera Jayasoma, > Software Architect, > > WSO2, Inc. (http://wso2.com) > email: [email protected] > blog: http://blog.sameera.org > twitter: https://twitter.com/sameerajayasoma > flickr: http://www.flickr.com/photos/sameera-jayasoma/collections > Mobile: 0094776364456 > > Lean . Enterprise . Middleware > > -- -- Chanaka Fernando Senior Technical Lead WSO2, Inc.; http://wso2.com lean.enterprise.middleware mobile: +94 773337238 Blog : http://soatutorials.blogspot.com LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0 Twitter:https://twitter.com/chanakaudaya
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
