[Adding Kishanthan]
On Thu, Oct 8, 2015 at 6:14 PM, Chanaka Fernando <[email protected]> wrote:
> Hi All,
>
> At last, I was able to find the root cause for this behavior. This is
> actually coming from carbon-mediation code. What actually happens is that
> when ESB starts with a tenant, it will not load the tenant until it
> receives the first request. When It receives the first request, it will
> call the MultitenantMessageReceiver.processRequest() method. Within this
> method, it will call the following method.
>
> PrivilegedCarbonContext.startTenantFlow();
>
>
> This will get the ThreadLocal variable *parentContextHolderStack *and get
> the stack and push the carbonContextDataHolder object to the stack.
> After this, tenant loading happens and within the initialization of the
> carbon mediation registry we have the following code segment.
>
> *org.wso2.carbon.mediation.registry.WSO2Registry.java*
>
> /**
> * Carbon Kernel mandates to set Threadlocal before calling anything in
> kernel
> */
> private void setTenantInfo() {
> // Preserve user name
> String username =
> PrivilegedCarbonContext.getThreadLocalCarbonContext().getUsername();
> *PrivilegedCarbonContext.destroyCurrentContext();*
> PrivilegedCarbonContext cc =
> PrivilegedCarbonContext.getThreadLocalCarbonContext();
> cc.setTenantDomain(domain);
> cc.setTenantId(tenantId);
> if (username != null) { // Set back the user name
>
> PrivilegedCarbonContext.getThreadLocalCarbonContext().setUsername(username);
> }
> }
>
> Within the above method following line causes the issue.
>
> PrivilegedCarbonContext.destroyCurrentContext();
>
> When this method is called, it will reset the *parentContextHolderStack *and
> the initial object push into the stack is destroyed. Then after tenant
> loading, within the MultitenantMessageReceiver.processRequest() method, it
> tries to end the tenant flow within finally block and try to pop the object
> which it pushes early. But right now, we have a new context stack and it
> will throw the emptyStackException due to that.
>
> @Sameera/KasunG: Do we really need to use the following code segment
> within the above method?
>
>
> *PrivilegedCarbonContext.destroyCurrentContext();*
>
> I saw this method called within carbon-mediation components in 3 different
> locations. AFAIU, we don't need to destroy the current context when we are
> accessing the ThreadLocalContext. Please share your thoughts such that we
> can fix this issue. Issue is fixed when I comment out the above line :)
>
>
> Thanks,
> Chanaka
>
>
>
> On Wed, Oct 7, 2015 at 5:34 PM, Chanaka Fernando <[email protected]>
> wrote:
>
>> Hi Sameera/KasunG,
>>
>> I have debugged the code to find the root cause for this empty stack
>> exception. This is causing several other issues at the ESB layer. What
>> actually happens is that inside the
>> MultitenantMessageReceiver.processRequest() method, we have the following
>> code segment.
>>
>> try {
>> PrivilegedCarbonContext.startTenantFlow();
>> PrivilegedCarbonContext privilegedCarbonContext =
>> PrivilegedCarbonContext.getThreadLocalCarbonContext();
>> privilegedCarbonContext.setTenantDomain(tenantDomain, true);
>> // this is to prevent non-blocking transports from sending 202
>>
>> mainInMsgContext.getOperationContext().setProperty(Constants.RESPONSE_WRITTEN,
>> "SKIP");
>>
>> ConfigurationContext tenantConfigCtx =
>>
>> TenantAxisUtils.getTenantConfigurationContext(tenantDomain,
>>
>> mainConfigCtx);
>> if (tenantConfigCtx == null) {
>> // Throw AxisFault: Tenant does not exist
>> handleException(mainInMsgContext, new AxisFault("Tenant " +
>> tenantDomain +
>> " not found"));
>> return;
>> }
>>
>> if (mainInMsgContext.isDoingREST()) { // Handle REST requests
>> doREST(mainInMsgContext, to, tenantDomain, tenantConfigCtx,
>> serviceAndOperation);
>> } else {
>> doSOAP(mainInMsgContext, tenantDomain, tenantConfigCtx,
>> serviceAndOperation);
>> }
>> } finally {
>> PrivilegedCarbonContext.endTenantFlow();
>> }
>>
>>
>> When we are calling the endTenantFlow() method, it will go inside the
>> following method within the CarbonContextDataHolder class.
>>
>> /**
>> * This will end the tenant flow and restore the previous CarbonContext.
>> */
>> public void endTenantFlow() {
>> Stack<CarbonContextDataHolder> carbonContextDataHolders =
>> parentContextHolderStack.get();
>> if (carbonContextDataHolders != null) {
>> currentContextHolder.set(carbonContextDataHolders.pop());
>> }
>> }
>>
>>
>> At the time of calling this method carbonContextDataHolders stack has
>> become empty and causing the EmptyStackException. When I debug the code, I
>> found that there is a scheduled task running in a different thread and
>> accessing the same method frequently. This is coming from
>> AbstractQuartzTaskManager class method.
>>
>> public void triggerComplete(Trigger trigger, JobExecutionContext
>> jobExecutionContext,
>> Trigger.CompletedExecutionInstruction completedExecutionInstruction)
>> {
>> PrivilegedCarbonContext.startTenantFlow();
>>
>> PrivilegedCarbonContext.getThreadLocalCarbonContext().setTenantId(getTenantId(),
>> true);
>> if(trigger.getNextFireTime() == null) {
>> try {
>> TaskUtils.setTaskFinished(getTaskRepository(),
>> trigger.getJobKey().getName(), true);
>> } catch (TaskException e) {
>> log.error("Error in Finishing Task [" +
>> trigger.getJobKey().getName() +
>> "]: " + e.getMessage(), e);
>> }
>> }
>> PrivilegedCarbonContext.endTenantFlow();
>> }
>>
>>
>> What I could not figure out is the thread which is emptying the stack. Is
>> it possible that a different thread can access this
>> carbonContextDataHolders stack and popping out the element before the
>> PassThroughMessageProcessor thread access the same?
>>
>>
>>
>> On Wed, Jul 8, 2015 at 11:12 AM, Jagath Sisirakumara Ariyarathne <
>> [email protected]> wrote:
>>
>>> Hi Sameera,
>>>
>>> I will check it and update.
>>>
>>> Thanks.
>>>
>>> On Wed, Jul 8, 2015 at 11:10 AM, Sameera Jayasoma <[email protected]>
>>> wrote:
>>>
>>>> Hi Jagath,
>>>>
>>>> Can you debug and see whey the stack becomes empty? Thats a serious
>>>> problem. Stack should be become empty here.
>>>>
>>>> Checking whether the stack is empty will stop the error log, but it
>>>> doesn't fix the actual problem here.
>>>>
>>>> On Wed, Jul 8, 2015 at 10:50 AM, Sameera Jayasoma <[email protected]>
>>>> wrote:
>>>>
>>>>> I understand, but we need to understand why that stack becomes empty.
>>>>> AFAIK, if we follow the proper APIs, stack should not become empty
>>>>>
>>>>> On Tue, Jul 7, 2015 at 5:47 PM, Kasun Indrasiri <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> Yeah, we should always check for an empty stack.
>>>>>>
>>>>>> On Tue, Jul 7, 2015 at 5:17 PM, Malaka Silva <[email protected]> wrote:
>>>>>>
>>>>>>> I think we need to check isEmpty as well.
>>>>>>>
>>>>>>> On Tue, Jul 7, 2015 at 3:41 PM, Jagath Sisirakumara Ariyarathne <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I am working on [1] and found that the cause of the exception
>>>>>>>> mentioned below is in the code segment in org
>>>>>>>> .wso2.carbon.context.internal.CarbonContextDataHolder in
>>>>>>>> carbon.utils.
>>>>>>>>
>>>>>>>> public void endTenantFlow() {
>>>>>>>>
>>>>>>>> Stack<CarbonContextDataHolder> carbonContextDataHolders =
>>>>>>>> parentContextHolderStack.get();
>>>>>>>> if (carbonContextDataHolders != null) {
>>>>>>>> currentContextHolder.set(carbonContextDataHolders.pop());
>>>>>>>> }
>>>>>>>> }
>>>>>>>>
>>>>>>>> *Exception :*
>>>>>>>>
>>>>>>>> java.util.EmptyStackException
>>>>>>>> at java.util.Stack.peek(Stack.java:102)
>>>>>>>> at java.util.Stack.pop(Stack.java:84)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.context.internal.CarbonContextDataHolder.endTenantFlow(CarbonContextDataHolder.java:1291)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.context.PrivilegedCarbonContext.endTenantFlow(PrivilegedCarbonContext.java:75)
>>>>>>>> at
>>>>>>>> org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute(TaskQuartzJobAdapter.java:69)
>>>>>>>> at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>>>>>>>> at
>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>
>>>>>>>> Issue occurs when it tries to pop elements from
>>>>>>>> carbonContextDataHolders stack when it is empty.
>>>>>>>>
>>>>>>>> Is it a possible scenario that this stack being empty and shouldn't it
>>>>>>>> be handled at CarbonContextDataHolder (check isEmpty in stack)?
>>>>>>>>
>>>>>>>> [1] - https://wso2.org/jira/browse/ESBJAVA-3832
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> --
>>>>>>>> Jagath Ariyarathne
>>>>>>>> Technical Lead
>>>>>>>> WSO2 Inc. http://wso2.com/
>>>>>>>> Email: [email protected]
>>>>>>>> Mob : +94 77 386 7048
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Best Regards,
>>>>>>>
>>>>>>> Malaka Silva
>>>>>>> Senior Tech Lead
>>>>>>> M: +94 777 219 791
>>>>>>> Tel : 94 11 214 5345
>>>>>>> Fax :94 11 2145300
>>>>>>> Skype : malaka.sampath.silva
>>>>>>> LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77
>>>>>>> Blog : http://mrmalakasilva.blogspot.com/
>>>>>>>
>>>>>>> WSO2, Inc.
>>>>>>> lean . enterprise . middleware
>>>>>>> http://www.wso2.com/
>>>>>>> http://www.wso2.com/about/team/malaka-silva/
>>>>>>> <http://wso2.com/about/team/malaka-silva/>
>>>>>>>
>>>>>>> Save a tree -Conserve nature & Save the world for your future. Print
>>>>>>> this email only if it is absolutely necessary.
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Dev mailing list
>>>>>>> [email protected]
>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Kasun Indrasiri
>>>>>> Software Architect
>>>>>> WSO2, Inc.; http://wso2.com
>>>>>> lean.enterprise.middleware
>>>>>>
>>>>>> cell: +94 77 556 5206
>>>>>> Blog : http://kasunpanorama.blogspot.com/
>>>>>>
>>>>>> _______________________________________________
>>>>>> Dev mailing list
>>>>>> [email protected]
>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Sameera Jayasoma,
>>>>> Software Architect,
>>>>>
>>>>> WSO2, Inc. (http://wso2.com)
>>>>> email: [email protected]
>>>>> blog: http://blog.sameera.org
>>>>> twitter: https://twitter.com/sameerajayasoma
>>>>> flickr: http://www.flickr.com/photos/sameera-jayasoma/collections
>>>>> Mobile: 0094776364456
>>>>>
>>>>> Lean . Enterprise . Middleware
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Sameera Jayasoma,
>>>> Software Architect,
>>>>
>>>> WSO2, Inc. (http://wso2.com)
>>>> email: [email protected]
>>>> blog: http://blog.sameera.org
>>>> twitter: https://twitter.com/sameerajayasoma
>>>> flickr: http://www.flickr.com/photos/sameera-jayasoma/collections
>>>> Mobile: 0094776364456
>>>>
>>>> Lean . Enterprise . Middleware
>>>>
>>>>
>>>> _______________________________________________
>>>> Dev mailing list
>>>> [email protected]
>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>
>>>>
>>>
>>>
>>> --
>>> Jagath Ariyarathne
>>> Technical Lead
>>> WSO2 Inc. http://wso2.com/
>>> Email: [email protected]
>>> Mob : +94 77 386 7048
>>>
>>>
>>> _______________________________________________
>>> Dev mailing list
>>> [email protected]
>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>
>>>
>>
>>
>> --
>> --
>> Chanaka Fernando
>> Senior Technical Lead
>> WSO2, Inc.; http://wso2.com
>> lean.enterprise.middleware
>>
>> mobile: +94 773337238
>> Blog : http://soatutorials.blogspot.com
>> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0
>> Twitter:https://twitter.com/chanakaudaya
>>
>>
>>
>>
>>
>
>
> --
> --
> Chanaka Fernando
> Senior Technical Lead
> WSO2, Inc.; http://wso2.com
> lean.enterprise.middleware
>
> mobile: +94 773337238
> Blog : http://soatutorials.blogspot.com
> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0
> Twitter:https://twitter.com/chanakaudaya
>
>
>
>
>
--
--
Chanaka Fernando
Senior Technical Lead
WSO2, Inc.; http://wso2.com
lean.enterprise.middleware
mobile: +94 773337238
Blog : http://soatutorials.blogspot.com
LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0
Twitter:https://twitter.com/chanakaudaya
_______________________________________________
Dev mailing list
[email protected]
http://wso2.org/cgi-bin/mailman/listinfo/dev
