Hi Nuwan, Our plan is to do the first milestone release end of next week. With this we hope to incorporate latest carbon-deployment and kernel releases (if it released during this week).
Thanks. On Wed, Oct 14, 2015 at 11:07 AM, Chanaka Fernando <[email protected]> wrote: > [Adding Jagath] > > On Wed, Oct 14, 2015 at 10:17 AM, Nuwan Dias <[email protected]> wrote: > >> Hi ESB folks, >> >> Can we get a milestone release of synapse and carbon-mediation including >> this fix and another PR I sent to synapse? We have to do a milestone >> release of API Manager this week. >> >> Thanks, >> NuwanD. >> >> On Mon, Oct 12, 2015 at 11:43 AM, Nuwan Dias <[email protected]> wrote: >> >>> Thanks a lot Chanaka. >>> >>> On Mon, Oct 12, 2015 at 11:39 AM, Chanaka Fernando <[email protected]> >>> wrote: >>> >>>> Hi All, >>>> >>>> This issue is fixed with the following PR [1]. Thanks for reporting and >>>> helping us to figure out the issue. >>>> >>>> [1] https://github.com/wso2/carbon-mediation/pull/467 >>>> >>>> On Fri, Oct 9, 2015 at 3:41 PM, Chanaka Fernando <[email protected]> >>>> wrote: >>>> >>>>> Hi Sameera, >>>>> >>>>> Your judgment is correct and it helped us to find the root cause :). >>>>> I'll fix the code from carbon-mediation side. Thanks for your input. >>>>> >>>>> On Fri, Oct 9, 2015 at 3:19 PM, Sameera Jayasoma <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Chanaka, >>>>>> >>>>>> You don't need to destroy the current context. This part is handle by >>>>>> the Kernel code. Users of the CarbonContext API do not need to worry >>>>>> about >>>>>> destroying the context. >>>>>> >>>>>> Who has done this change? So my initial judgment is correct. :) Stack >>>>>> cannot become empty. >>>>>> >>>>>> Thanks, >>>>>> Sameera. >>>>>> >>>>>> On Thu, Oct 8, 2015 at 6:14 PM, Chanaka Fernando <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> At last, I was able to find the root cause for this behavior. This >>>>>>> is actually coming from carbon-mediation code. What actually happens is >>>>>>> that when ESB starts with a tenant, it will not load the tenant until it >>>>>>> receives the first request. When It receives the first request, it will >>>>>>> call the MultitenantMessageReceiver.processRequest() method. Within this >>>>>>> method, it will call the following method. >>>>>>> >>>>>>> PrivilegedCarbonContext.startTenantFlow(); >>>>>>> >>>>>>> >>>>>>> This will get the ThreadLocal variable *parentContextHolderStack *and >>>>>>> get the stack and push the carbonContextDataHolder object to the >>>>>>> stack. After this, tenant loading happens and within the initialization >>>>>>> of >>>>>>> the carbon mediation registry we have the following code segment. >>>>>>> >>>>>>> *org.wso2.carbon.mediation.registry.WSO2Registry.java* >>>>>>> >>>>>>> /** >>>>>>> * Carbon Kernel mandates to set Threadlocal before calling anything >>>>>>> in kernel >>>>>>> */ >>>>>>> private void setTenantInfo() { >>>>>>> // Preserve user name >>>>>>> String username = >>>>>>> PrivilegedCarbonContext.getThreadLocalCarbonContext().getUsername(); >>>>>>> *PrivilegedCarbonContext.destroyCurrentContext();* >>>>>>> PrivilegedCarbonContext cc = >>>>>>> PrivilegedCarbonContext.getThreadLocalCarbonContext(); >>>>>>> cc.setTenantDomain(domain); >>>>>>> cc.setTenantId(tenantId); >>>>>>> if (username != null) { // Set back the user name >>>>>>> >>>>>>> PrivilegedCarbonContext.getThreadLocalCarbonContext().setUsername(username); >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> Within the above method following line causes the issue. >>>>>>> >>>>>>> PrivilegedCarbonContext.destroyCurrentContext(); >>>>>>> >>>>>>> When this method is called, it will reset the *parentContextHolderStack >>>>>>> *and the initial object push into the stack is destroyed. Then >>>>>>> after tenant loading, within the >>>>>>> MultitenantMessageReceiver.processRequest() method, it tries to end the >>>>>>> tenant flow within finally block and try to pop the object which it >>>>>>> pushes >>>>>>> early. But right now, we have a new context stack and it will throw the >>>>>>> emptyStackException due to that. >>>>>>> >>>>>>> @Sameera/KasunG: Do we really need to use the following code segment >>>>>>> within the above method? >>>>>>> >>>>>>> >>>>>>> *PrivilegedCarbonContext.destroyCurrentContext();* >>>>>>> >>>>>>> I saw this method called within carbon-mediation components in 3 >>>>>>> different locations. AFAIU, we don't need to destroy the current context >>>>>>> when we are accessing the ThreadLocalContext. Please share your thoughts >>>>>>> such that we can fix this issue. Issue is fixed when I comment out the >>>>>>> above line :) >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> Chanaka >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Oct 7, 2015 at 5:34 PM, Chanaka Fernando <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Sameera/KasunG, >>>>>>>> >>>>>>>> I have debugged the code to find the root cause for this empty >>>>>>>> stack exception. This is causing several other issues at the ESB layer. >>>>>>>> What actually happens is that inside the >>>>>>>> MultitenantMessageReceiver.processRequest() method, we have the >>>>>>>> following >>>>>>>> code segment. >>>>>>>> >>>>>>>> try { >>>>>>>> PrivilegedCarbonContext.startTenantFlow(); >>>>>>>> PrivilegedCarbonContext privilegedCarbonContext = >>>>>>>> PrivilegedCarbonContext.getThreadLocalCarbonContext(); >>>>>>>> privilegedCarbonContext.setTenantDomain(tenantDomain, true); >>>>>>>> // this is to prevent non-blocking transports from sending 202 >>>>>>>> >>>>>>>> mainInMsgContext.getOperationContext().setProperty(Constants.RESPONSE_WRITTEN, >>>>>>>> "SKIP"); >>>>>>>> >>>>>>>> ConfigurationContext tenantConfigCtx = >>>>>>>> >>>>>>>> TenantAxisUtils.getTenantConfigurationContext(tenantDomain, >>>>>>>> >>>>>>>> mainConfigCtx); >>>>>>>> if (tenantConfigCtx == null) { >>>>>>>> // Throw AxisFault: Tenant does not exist >>>>>>>> handleException(mainInMsgContext, new AxisFault("Tenant " + >>>>>>>> tenantDomain + >>>>>>>> " not >>>>>>>> found")); >>>>>>>> return; >>>>>>>> } >>>>>>>> >>>>>>>> if (mainInMsgContext.isDoingREST()) { // Handle REST requests >>>>>>>> doREST(mainInMsgContext, to, tenantDomain, tenantConfigCtx, >>>>>>>> serviceAndOperation); >>>>>>>> } else { >>>>>>>> doSOAP(mainInMsgContext, tenantDomain, tenantConfigCtx, >>>>>>>> serviceAndOperation); >>>>>>>> } >>>>>>>> } finally { >>>>>>>> PrivilegedCarbonContext.endTenantFlow(); >>>>>>>> } >>>>>>>> >>>>>>>> >>>>>>>> When we are calling the endTenantFlow() method, it will go inside >>>>>>>> the following method within the CarbonContextDataHolder class. >>>>>>>> >>>>>>>> /** >>>>>>>> * This will end the tenant flow and restore the previous >>>>>>>> CarbonContext. >>>>>>>> */ >>>>>>>> public void endTenantFlow() { >>>>>>>> Stack<CarbonContextDataHolder> carbonContextDataHolders = >>>>>>>> parentContextHolderStack.get(); >>>>>>>> if (carbonContextDataHolders != null) { >>>>>>>> currentContextHolder.set(carbonContextDataHolders.pop()); >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>>>> >>>>>>>> At the time of calling this method carbonContextDataHolders stack >>>>>>>> has become empty and causing the EmptyStackException. When I debug the >>>>>>>> code, I found that there is a scheduled task running in a different >>>>>>>> thread >>>>>>>> and accessing the same method frequently. This is coming from >>>>>>>> AbstractQuartzTaskManager class method. >>>>>>>> >>>>>>>> public void triggerComplete(Trigger trigger, JobExecutionContext >>>>>>>> jobExecutionContext, >>>>>>>> Trigger.CompletedExecutionInstruction >>>>>>>> completedExecutionInstruction) { >>>>>>>> PrivilegedCarbonContext.startTenantFlow(); >>>>>>>> >>>>>>>> PrivilegedCarbonContext.getThreadLocalCarbonContext().setTenantId(getTenantId(), >>>>>>>> true); >>>>>>>> if(trigger.getNextFireTime() == null) { >>>>>>>> try { >>>>>>>> TaskUtils.setTaskFinished(getTaskRepository(), >>>>>>>> trigger.getJobKey().getName(), true); >>>>>>>> } catch (TaskException e) { >>>>>>>> log.error("Error in Finishing Task [" + >>>>>>>> trigger.getJobKey().getName() + >>>>>>>> "]: " + e.getMessage(), e); >>>>>>>> } >>>>>>>> } >>>>>>>> PrivilegedCarbonContext.endTenantFlow(); >>>>>>>> } >>>>>>>> >>>>>>>> >>>>>>>> What I could not figure out is the thread which is emptying the >>>>>>>> stack. Is it possible that a different thread can access this >>>>>>>> carbonContextDataHolders stack and popping out the element before the >>>>>>>> PassThroughMessageProcessor thread access the same? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Jul 8, 2015 at 11:12 AM, Jagath Sisirakumara Ariyarathne < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi Sameera, >>>>>>>>> >>>>>>>>> I will check it and update. >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>>> >>>>>>>>> On Wed, Jul 8, 2015 at 11:10 AM, Sameera Jayasoma < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi Jagath, >>>>>>>>>> >>>>>>>>>> Can you debug and see whey the stack becomes empty? Thats a >>>>>>>>>> serious problem. Stack should be become empty here. >>>>>>>>>> >>>>>>>>>> Checking whether the stack is empty will stop the error log, but >>>>>>>>>> it doesn't fix the actual problem here. >>>>>>>>>> >>>>>>>>>> On Wed, Jul 8, 2015 at 10:50 AM, Sameera Jayasoma < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> I understand, but we need to understand why that stack becomes >>>>>>>>>>> empty. AFAIK, if we follow the proper APIs, stack should not >>>>>>>>>>> become empty >>>>>>>>>>> >>>>>>>>>>> On Tue, Jul 7, 2015 at 5:47 PM, Kasun Indrasiri <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Yeah, we should always check for an empty stack. >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Jul 7, 2015 at 5:17 PM, Malaka Silva <[email protected]> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I think we need to check isEmpty as well. >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Jul 7, 2015 at 3:41 PM, Jagath Sisirakumara >>>>>>>>>>>>> Ariyarathne <[email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I am working on [1] and found that the cause of the exception >>>>>>>>>>>>>> mentioned below is in the code segment in org >>>>>>>>>>>>>> .wso2.carbon.context.internal.CarbonContextDataHolder in >>>>>>>>>>>>>> carbon.utils. >>>>>>>>>>>>>> >>>>>>>>>>>>>> public void endTenantFlow() { >>>>>>>>>>>>>> >>>>>>>>>>>>>> Stack<CarbonContextDataHolder> carbonContextDataHolders = >>>>>>>>>>>>>> parentContextHolderStack.get(); >>>>>>>>>>>>>> if (carbonContextDataHolders != null) { >>>>>>>>>>>>>> currentContextHolder.set(carbonContextDataHolders.pop()); >>>>>>>>>>>>>> } >>>>>>>>>>>>>> } >>>>>>>>>>>>>> >>>>>>>>>>>>>> *Exception :* >>>>>>>>>>>>>> >>>>>>>>>>>>>> java.util.EmptyStackException >>>>>>>>>>>>>> at java.util.Stack.peek(Stack.java:102) >>>>>>>>>>>>>> at java.util.Stack.pop(Stack.java:84) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.wso2.carbon.context.internal.CarbonContextDataHolder.endTenantFlow(CarbonContextDataHolder.java:1291) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.wso2.carbon.context.PrivilegedCarbonContext.endTenantFlow(PrivilegedCarbonContext.java:75) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute(TaskQuartzJobAdapter.java:69) >>>>>>>>>>>>>> at org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>>>>>>> >>>>>>>>>>>>>> Issue occurs when it tries to pop elements from >>>>>>>>>>>>>> carbonContextDataHolders stack when it is empty. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Is it a possible scenario that this stack being empty and >>>>>>>>>>>>>> shouldn't it be handled at CarbonContextDataHolder (check >>>>>>>>>>>>>> isEmpty in stack)? >>>>>>>>>>>>>> >>>>>>>>>>>>>> [1] - https://wso2.org/jira/browse/ESBJAVA-3832 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Jagath Ariyarathne >>>>>>>>>>>>>> Technical Lead >>>>>>>>>>>>>> WSO2 Inc. http://wso2.com/ >>>>>>>>>>>>>> Email: [email protected] >>>>>>>>>>>>>> Mob : +94 77 386 7048 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> >>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>> >>>>>>>>>>>>> Malaka Silva >>>>>>>>>>>>> Senior Tech Lead >>>>>>>>>>>>> M: +94 777 219 791 >>>>>>>>>>>>> Tel : 94 11 214 5345 >>>>>>>>>>>>> Fax :94 11 2145300 >>>>>>>>>>>>> Skype : malaka.sampath.silva >>>>>>>>>>>>> LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77 >>>>>>>>>>>>> Blog : http://mrmalakasilva.blogspot.com/ >>>>>>>>>>>>> >>>>>>>>>>>>> WSO2, Inc. >>>>>>>>>>>>> lean . enterprise . middleware >>>>>>>>>>>>> http://www.wso2.com/ >>>>>>>>>>>>> http://www.wso2.com/about/team/malaka-silva/ >>>>>>>>>>>>> <http://wso2.com/about/team/malaka-silva/> >>>>>>>>>>>>> >>>>>>>>>>>>> Save a tree -Conserve nature & Save the world for your future. >>>>>>>>>>>>> Print this email only if it is absolutely necessary. >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Dev mailing list >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Kasun Indrasiri >>>>>>>>>>>> Software Architect >>>>>>>>>>>> WSO2, Inc.; http://wso2.com >>>>>>>>>>>> lean.enterprise.middleware >>>>>>>>>>>> >>>>>>>>>>>> cell: +94 77 556 5206 >>>>>>>>>>>> Blog : http://kasunpanorama.blogspot.com/ >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Dev mailing list >>>>>>>>>>>> [email protected] >>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Sameera Jayasoma, >>>>>>>>>>> Software Architect, >>>>>>>>>>> >>>>>>>>>>> WSO2, Inc. (http://wso2.com) >>>>>>>>>>> email: [email protected] >>>>>>>>>>> blog: http://blog.sameera.org >>>>>>>>>>> twitter: https://twitter.com/sameerajayasoma >>>>>>>>>>> flickr: >>>>>>>>>>> http://www.flickr.com/photos/sameera-jayasoma/collections >>>>>>>>>>> Mobile: 0094776364456 >>>>>>>>>>> >>>>>>>>>>> Lean . Enterprise . Middleware >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Sameera Jayasoma, >>>>>>>>>> Software Architect, >>>>>>>>>> >>>>>>>>>> WSO2, Inc. (http://wso2.com) >>>>>>>>>> email: [email protected] >>>>>>>>>> blog: http://blog.sameera.org >>>>>>>>>> twitter: https://twitter.com/sameerajayasoma >>>>>>>>>> flickr: http://www.flickr.com/photos/sameera-jayasoma/collections >>>>>>>>>> Mobile: 0094776364456 >>>>>>>>>> >>>>>>>>>> Lean . Enterprise . Middleware >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Dev mailing list >>>>>>>>>> [email protected] >>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Jagath Ariyarathne >>>>>>>>> Technical Lead >>>>>>>>> WSO2 Inc. http://wso2.com/ >>>>>>>>> Email: [email protected] >>>>>>>>> Mob : +94 77 386 7048 >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Dev mailing list >>>>>>>>> [email protected] >>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> -- >>>>>>>> Chanaka Fernando >>>>>>>> Senior Technical Lead >>>>>>>> WSO2, Inc.; http://wso2.com >>>>>>>> lean.enterprise.middleware >>>>>>>> >>>>>>>> mobile: +94 773337238 >>>>>>>> Blog : http://soatutorials.blogspot.com >>>>>>>> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0 >>>>>>>> Twitter:https://twitter.com/chanakaudaya >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> -- >>>>>>> Chanaka Fernando >>>>>>> Senior Technical Lead >>>>>>> WSO2, Inc.; http://wso2.com >>>>>>> lean.enterprise.middleware >>>>>>> >>>>>>> mobile: +94 773337238 >>>>>>> Blog : http://soatutorials.blogspot.com >>>>>>> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0 >>>>>>> Twitter:https://twitter.com/chanakaudaya >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Sameera Jayasoma, >>>>>> Software Architect, >>>>>> >>>>>> WSO2, Inc. (http://wso2.com) >>>>>> email: [email protected] >>>>>> blog: http://blog.sameera.org >>>>>> twitter: https://twitter.com/sameerajayasoma >>>>>> flickr: http://www.flickr.com/photos/sameera-jayasoma/collections >>>>>> Mobile: 0094776364456 >>>>>> >>>>>> Lean . Enterprise . Middleware >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> -- >>>>> Chanaka Fernando >>>>> Senior Technical Lead >>>>> WSO2, Inc.; http://wso2.com >>>>> lean.enterprise.middleware >>>>> >>>>> mobile: +94 773337238 >>>>> Blog : http://soatutorials.blogspot.com >>>>> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0 >>>>> Twitter:https://twitter.com/chanakaudaya >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> -- >>>> Chanaka Fernando >>>> Senior Technical Lead >>>> WSO2, Inc.; http://wso2.com >>>> lean.enterprise.middleware >>>> >>>> mobile: +94 773337238 >>>> Blog : http://soatutorials.blogspot.com >>>> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0 >>>> Twitter:https://twitter.com/chanakaudaya >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Dev mailing list >>>> [email protected] >>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>> >>>> >>> >>> >>> -- >>> Nuwan Dias >>> >>> Technical Lead - WSO2, Inc. http://wso2.com >>> email : [email protected] >>> Phone : +94 777 775 729 >>> >> >> >> >> -- >> Nuwan Dias >> >> Technical Lead - WSO2, Inc. http://wso2.com >> email : [email protected] >> Phone : +94 777 775 729 >> > > > > -- > -- > Chanaka Fernando > Senior Technical Lead > WSO2, Inc.; http://wso2.com > lean.enterprise.middleware > > mobile: +94 773337238 > Blog : http://soatutorials.blogspot.com > LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0 > Twitter:https://twitter.com/chanakaudaya > > > > > -- Jagath Ariyarathne Technical Lead WSO2 Inc. http://wso2.com/ Email: [email protected] Mob : +94 77 386 7048
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
