Hi ESB folks, Can we get a milestone release of synapse and carbon-mediation including this fix and another PR I sent to synapse? We have to do a milestone release of API Manager this week.
Thanks, NuwanD. On Mon, Oct 12, 2015 at 11:43 AM, Nuwan Dias <[email protected]> wrote: > Thanks a lot Chanaka. > > On Mon, Oct 12, 2015 at 11:39 AM, Chanaka Fernando <[email protected]> > wrote: > >> Hi All, >> >> This issue is fixed with the following PR [1]. Thanks for reporting and >> helping us to figure out the issue. >> >> [1] https://github.com/wso2/carbon-mediation/pull/467 >> >> On Fri, Oct 9, 2015 at 3:41 PM, Chanaka Fernando <[email protected]> >> wrote: >> >>> Hi Sameera, >>> >>> Your judgment is correct and it helped us to find the root cause :). >>> I'll fix the code from carbon-mediation side. Thanks for your input. >>> >>> On Fri, Oct 9, 2015 at 3:19 PM, Sameera Jayasoma <[email protected]> >>> wrote: >>> >>>> Hi Chanaka, >>>> >>>> You don't need to destroy the current context. This part is handle by >>>> the Kernel code. Users of the CarbonContext API do not need to worry about >>>> destroying the context. >>>> >>>> Who has done this change? So my initial judgment is correct. :) Stack >>>> cannot become empty. >>>> >>>> Thanks, >>>> Sameera. >>>> >>>> On Thu, Oct 8, 2015 at 6:14 PM, Chanaka Fernando <[email protected]> >>>> wrote: >>>> >>>>> Hi All, >>>>> >>>>> At last, I was able to find the root cause for this behavior. This is >>>>> actually coming from carbon-mediation code. What actually happens is that >>>>> when ESB starts with a tenant, it will not load the tenant until it >>>>> receives the first request. When It receives the first request, it will >>>>> call the MultitenantMessageReceiver.processRequest() method. Within this >>>>> method, it will call the following method. >>>>> >>>>> PrivilegedCarbonContext.startTenantFlow(); >>>>> >>>>> >>>>> This will get the ThreadLocal variable *parentContextHolderStack *and >>>>> get the stack and push the carbonContextDataHolder object to the >>>>> stack. After this, tenant loading happens and within the initialization of >>>>> the carbon mediation registry we have the following code segment. >>>>> >>>>> *org.wso2.carbon.mediation.registry.WSO2Registry.java* >>>>> >>>>> /** >>>>> * Carbon Kernel mandates to set Threadlocal before calling anything in >>>>> kernel >>>>> */ >>>>> private void setTenantInfo() { >>>>> // Preserve user name >>>>> String username = >>>>> PrivilegedCarbonContext.getThreadLocalCarbonContext().getUsername(); >>>>> *PrivilegedCarbonContext.destroyCurrentContext();* >>>>> PrivilegedCarbonContext cc = >>>>> PrivilegedCarbonContext.getThreadLocalCarbonContext(); >>>>> cc.setTenantDomain(domain); >>>>> cc.setTenantId(tenantId); >>>>> if (username != null) { // Set back the user name >>>>> >>>>> PrivilegedCarbonContext.getThreadLocalCarbonContext().setUsername(username); >>>>> } >>>>> } >>>>> >>>>> Within the above method following line causes the issue. >>>>> >>>>> PrivilegedCarbonContext.destroyCurrentContext(); >>>>> >>>>> When this method is called, it will reset the *parentContextHolderStack >>>>> *and the initial object push into the stack is destroyed. Then after >>>>> tenant loading, within the MultitenantMessageReceiver.processRequest() >>>>> method, it tries to end the tenant flow within finally block and try to >>>>> pop >>>>> the object which it pushes early. But right now, we have a new context >>>>> stack and it will throw the emptyStackException due to that. >>>>> >>>>> @Sameera/KasunG: Do we really need to use the following code segment >>>>> within the above method? >>>>> >>>>> >>>>> *PrivilegedCarbonContext.destroyCurrentContext();* >>>>> >>>>> I saw this method called within carbon-mediation components in 3 >>>>> different locations. AFAIU, we don't need to destroy the current context >>>>> when we are accessing the ThreadLocalContext. Please share your thoughts >>>>> such that we can fix this issue. Issue is fixed when I comment out the >>>>> above line :) >>>>> >>>>> >>>>> Thanks, >>>>> Chanaka >>>>> >>>>> >>>>> >>>>> On Wed, Oct 7, 2015 at 5:34 PM, Chanaka Fernando <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Sameera/KasunG, >>>>>> >>>>>> I have debugged the code to find the root cause for this empty stack >>>>>> exception. This is causing several other issues at the ESB layer. What >>>>>> actually happens is that inside the >>>>>> MultitenantMessageReceiver.processRequest() method, we have the following >>>>>> code segment. >>>>>> >>>>>> try { >>>>>> PrivilegedCarbonContext.startTenantFlow(); >>>>>> PrivilegedCarbonContext privilegedCarbonContext = >>>>>> PrivilegedCarbonContext.getThreadLocalCarbonContext(); >>>>>> privilegedCarbonContext.setTenantDomain(tenantDomain, true); >>>>>> // this is to prevent non-blocking transports from sending 202 >>>>>> >>>>>> mainInMsgContext.getOperationContext().setProperty(Constants.RESPONSE_WRITTEN, >>>>>> "SKIP"); >>>>>> >>>>>> ConfigurationContext tenantConfigCtx = >>>>>> >>>>>> TenantAxisUtils.getTenantConfigurationContext(tenantDomain, >>>>>> >>>>>> mainConfigCtx); >>>>>> if (tenantConfigCtx == null) { >>>>>> // Throw AxisFault: Tenant does not exist >>>>>> handleException(mainInMsgContext, new AxisFault("Tenant " + >>>>>> tenantDomain + >>>>>> " not found")); >>>>>> return; >>>>>> } >>>>>> >>>>>> if (mainInMsgContext.isDoingREST()) { // Handle REST requests >>>>>> doREST(mainInMsgContext, to, tenantDomain, tenantConfigCtx, >>>>>> serviceAndOperation); >>>>>> } else { >>>>>> doSOAP(mainInMsgContext, tenantDomain, tenantConfigCtx, >>>>>> serviceAndOperation); >>>>>> } >>>>>> } finally { >>>>>> PrivilegedCarbonContext.endTenantFlow(); >>>>>> } >>>>>> >>>>>> >>>>>> When we are calling the endTenantFlow() method, it will go inside the >>>>>> following method within the CarbonContextDataHolder class. >>>>>> >>>>>> /** >>>>>> * This will end the tenant flow and restore the previous CarbonContext. >>>>>> */ >>>>>> public void endTenantFlow() { >>>>>> Stack<CarbonContextDataHolder> carbonContextDataHolders = >>>>>> parentContextHolderStack.get(); >>>>>> if (carbonContextDataHolders != null) { >>>>>> currentContextHolder.set(carbonContextDataHolders.pop()); >>>>>> } >>>>>> } >>>>>> >>>>>> >>>>>> At the time of calling this method carbonContextDataHolders stack has >>>>>> become empty and causing the EmptyStackException. When I debug the code, >>>>>> I >>>>>> found that there is a scheduled task running in a different thread and >>>>>> accessing the same method frequently. This is coming from >>>>>> AbstractQuartzTaskManager class method. >>>>>> >>>>>> public void triggerComplete(Trigger trigger, JobExecutionContext >>>>>> jobExecutionContext, >>>>>> Trigger.CompletedExecutionInstruction >>>>>> completedExecutionInstruction) { >>>>>> PrivilegedCarbonContext.startTenantFlow(); >>>>>> >>>>>> PrivilegedCarbonContext.getThreadLocalCarbonContext().setTenantId(getTenantId(), >>>>>> true); >>>>>> if(trigger.getNextFireTime() == null) { >>>>>> try { >>>>>> TaskUtils.setTaskFinished(getTaskRepository(), >>>>>> trigger.getJobKey().getName(), true); >>>>>> } catch (TaskException e) { >>>>>> log.error("Error in Finishing Task [" + >>>>>> trigger.getJobKey().getName() + >>>>>> "]: " + e.getMessage(), e); >>>>>> } >>>>>> } >>>>>> PrivilegedCarbonContext.endTenantFlow(); >>>>>> } >>>>>> >>>>>> >>>>>> What I could not figure out is the thread which is emptying the >>>>>> stack. Is it possible that a different thread can access this >>>>>> carbonContextDataHolders stack and popping out the element before the >>>>>> PassThroughMessageProcessor thread access the same? >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Jul 8, 2015 at 11:12 AM, Jagath Sisirakumara Ariyarathne < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi Sameera, >>>>>>> >>>>>>> I will check it and update. >>>>>>> >>>>>>> Thanks. >>>>>>> >>>>>>> On Wed, Jul 8, 2015 at 11:10 AM, Sameera Jayasoma <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Jagath, >>>>>>>> >>>>>>>> Can you debug and see whey the stack becomes empty? Thats a serious >>>>>>>> problem. Stack should be become empty here. >>>>>>>> >>>>>>>> Checking whether the stack is empty will stop the error log, but it >>>>>>>> doesn't fix the actual problem here. >>>>>>>> >>>>>>>> On Wed, Jul 8, 2015 at 10:50 AM, Sameera Jayasoma <[email protected] >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> I understand, but we need to understand why that stack becomes >>>>>>>>> empty. AFAIK, if we follow the proper APIs, stack should not become >>>>>>>>> empty >>>>>>>>> >>>>>>>>> On Tue, Jul 7, 2015 at 5:47 PM, Kasun Indrasiri <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Yeah, we should always check for an empty stack. >>>>>>>>>> >>>>>>>>>> On Tue, Jul 7, 2015 at 5:17 PM, Malaka Silva <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> I think we need to check isEmpty as well. >>>>>>>>>>> >>>>>>>>>>> On Tue, Jul 7, 2015 at 3:41 PM, Jagath Sisirakumara Ariyarathne >>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> I am working on [1] and found that the cause of the exception >>>>>>>>>>>> mentioned below is in the code segment in org >>>>>>>>>>>> .wso2.carbon.context.internal.CarbonContextDataHolder in >>>>>>>>>>>> carbon.utils. >>>>>>>>>>>> >>>>>>>>>>>> public void endTenantFlow() { >>>>>>>>>>>> >>>>>>>>>>>> Stack<CarbonContextDataHolder> carbonContextDataHolders = >>>>>>>>>>>> parentContextHolderStack.get(); >>>>>>>>>>>> if (carbonContextDataHolders != null) { >>>>>>>>>>>> currentContextHolder.set(carbonContextDataHolders.pop()); >>>>>>>>>>>> } >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> *Exception :* >>>>>>>>>>>> >>>>>>>>>>>> java.util.EmptyStackException >>>>>>>>>>>> at java.util.Stack.peek(Stack.java:102) >>>>>>>>>>>> at java.util.Stack.pop(Stack.java:84) >>>>>>>>>>>> at >>>>>>>>>>>> org.wso2.carbon.context.internal.CarbonContextDataHolder.endTenantFlow(CarbonContextDataHolder.java:1291) >>>>>>>>>>>> at >>>>>>>>>>>> org.wso2.carbon.context.PrivilegedCarbonContext.endTenantFlow(PrivilegedCarbonContext.java:75) >>>>>>>>>>>> at >>>>>>>>>>>> org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute(TaskQuartzJobAdapter.java:69) >>>>>>>>>>>> at org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>> at >>>>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>>>>>>>>> at >>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>>>>>>> at >>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>>>>> >>>>>>>>>>>> Issue occurs when it tries to pop elements from >>>>>>>>>>>> carbonContextDataHolders stack when it is empty. >>>>>>>>>>>> >>>>>>>>>>>> Is it a possible scenario that this stack being empty and >>>>>>>>>>>> shouldn't it be handled at CarbonContextDataHolder (check isEmpty >>>>>>>>>>>> in stack)? >>>>>>>>>>>> >>>>>>>>>>>> [1] - https://wso2.org/jira/browse/ESBJAVA-3832 >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Jagath Ariyarathne >>>>>>>>>>>> Technical Lead >>>>>>>>>>>> WSO2 Inc. http://wso2.com/ >>>>>>>>>>>> Email: [email protected] >>>>>>>>>>>> Mob : +94 77 386 7048 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> Best Regards, >>>>>>>>>>> >>>>>>>>>>> Malaka Silva >>>>>>>>>>> Senior Tech Lead >>>>>>>>>>> M: +94 777 219 791 >>>>>>>>>>> Tel : 94 11 214 5345 >>>>>>>>>>> Fax :94 11 2145300 >>>>>>>>>>> Skype : malaka.sampath.silva >>>>>>>>>>> LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77 >>>>>>>>>>> Blog : http://mrmalakasilva.blogspot.com/ >>>>>>>>>>> >>>>>>>>>>> WSO2, Inc. >>>>>>>>>>> lean . enterprise . middleware >>>>>>>>>>> http://www.wso2.com/ >>>>>>>>>>> http://www.wso2.com/about/team/malaka-silva/ >>>>>>>>>>> <http://wso2.com/about/team/malaka-silva/> >>>>>>>>>>> >>>>>>>>>>> Save a tree -Conserve nature & Save the world for your future. >>>>>>>>>>> Print this email only if it is absolutely necessary. >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Dev mailing list >>>>>>>>>>> [email protected] >>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Kasun Indrasiri >>>>>>>>>> Software Architect >>>>>>>>>> WSO2, Inc.; http://wso2.com >>>>>>>>>> lean.enterprise.middleware >>>>>>>>>> >>>>>>>>>> cell: +94 77 556 5206 >>>>>>>>>> Blog : http://kasunpanorama.blogspot.com/ >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Dev mailing list >>>>>>>>>> [email protected] >>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Sameera Jayasoma, >>>>>>>>> Software Architect, >>>>>>>>> >>>>>>>>> WSO2, Inc. (http://wso2.com) >>>>>>>>> email: [email protected] >>>>>>>>> blog: http://blog.sameera.org >>>>>>>>> twitter: https://twitter.com/sameerajayasoma >>>>>>>>> flickr: http://www.flickr.com/photos/sameera-jayasoma/collections >>>>>>>>> Mobile: 0094776364456 >>>>>>>>> >>>>>>>>> Lean . Enterprise . Middleware >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Sameera Jayasoma, >>>>>>>> Software Architect, >>>>>>>> >>>>>>>> WSO2, Inc. (http://wso2.com) >>>>>>>> email: [email protected] >>>>>>>> blog: http://blog.sameera.org >>>>>>>> twitter: https://twitter.com/sameerajayasoma >>>>>>>> flickr: http://www.flickr.com/photos/sameera-jayasoma/collections >>>>>>>> Mobile: 0094776364456 >>>>>>>> >>>>>>>> Lean . Enterprise . Middleware >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Dev mailing list >>>>>>>> [email protected] >>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Jagath Ariyarathne >>>>>>> Technical Lead >>>>>>> WSO2 Inc. http://wso2.com/ >>>>>>> Email: [email protected] >>>>>>> Mob : +94 77 386 7048 >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Dev mailing list >>>>>>> [email protected] >>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> -- >>>>>> Chanaka Fernando >>>>>> Senior Technical Lead >>>>>> WSO2, Inc.; http://wso2.com >>>>>> lean.enterprise.middleware >>>>>> >>>>>> mobile: +94 773337238 >>>>>> Blog : http://soatutorials.blogspot.com >>>>>> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0 >>>>>> Twitter:https://twitter.com/chanakaudaya >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> -- >>>>> Chanaka Fernando >>>>> Senior Technical Lead >>>>> WSO2, Inc.; http://wso2.com >>>>> lean.enterprise.middleware >>>>> >>>>> mobile: +94 773337238 >>>>> Blog : http://soatutorials.blogspot.com >>>>> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0 >>>>> Twitter:https://twitter.com/chanakaudaya >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Sameera Jayasoma, >>>> Software Architect, >>>> >>>> WSO2, Inc. (http://wso2.com) >>>> email: [email protected] >>>> blog: http://blog.sameera.org >>>> twitter: https://twitter.com/sameerajayasoma >>>> flickr: http://www.flickr.com/photos/sameera-jayasoma/collections >>>> Mobile: 0094776364456 >>>> >>>> Lean . Enterprise . Middleware >>>> >>>> >>> >>> >>> -- >>> -- >>> Chanaka Fernando >>> Senior Technical Lead >>> WSO2, Inc.; http://wso2.com >>> lean.enterprise.middleware >>> >>> mobile: +94 773337238 >>> Blog : http://soatutorials.blogspot.com >>> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0 >>> Twitter:https://twitter.com/chanakaudaya >>> >>> >>> >>> >>> >> >> >> -- >> -- >> Chanaka Fernando >> Senior Technical Lead >> WSO2, Inc.; http://wso2.com >> lean.enterprise.middleware >> >> mobile: +94 773337238 >> Blog : http://soatutorials.blogspot.com >> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0 >> Twitter:https://twitter.com/chanakaudaya >> >> >> >> >> >> _______________________________________________ >> Dev mailing list >> [email protected] >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > Nuwan Dias > > Technical Lead - WSO2, Inc. http://wso2.com > email : [email protected] > Phone : +94 777 775 729 > -- Nuwan Dias Technical Lead - WSO2, Inc. http://wso2.com email : [email protected] Phone : +94 777 775 729
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
