Hi All,
At last, I was able to find the root cause for this behavior. This is
actually coming from carbon-mediation code. What actually happens is that
when ESB starts with a tenant, it will not load the tenant until it
receives the first request. When It receives the first request, it will
call the MultitenantMessageReceiver.processRequest() method. Within this
method, it will call the following method.
PrivilegedCarbonContext.startTenantFlow();
This will get the ThreadLocal variable *parentContextHolderStack *and get
the stack and push the carbonContextDataHolder object to the stack. After
this, tenant loading happens and within the initialization of the carbon
mediation registry we have the following code segment.
*org.wso2.carbon.mediation.registry.WSO2Registry.java*
/**
* Carbon Kernel mandates to set Threadlocal before calling anything in kernel
*/
private void setTenantInfo() {
// Preserve user name
String username =
PrivilegedCarbonContext.getThreadLocalCarbonContext().getUsername();
*PrivilegedCarbonContext.destroyCurrentContext();*
PrivilegedCarbonContext cc =
PrivilegedCarbonContext.getThreadLocalCarbonContext();
cc.setTenantDomain(domain);
cc.setTenantId(tenantId);
if (username != null) { // Set back the user name
PrivilegedCarbonContext.getThreadLocalCarbonContext().setUsername(username);
}
}
Within the above method following line causes the issue.
PrivilegedCarbonContext.destroyCurrentContext();
When this method is called, it will reset the *parentContextHolderStack *and
the initial object push into the stack is destroyed. Then after tenant
loading, within the MultitenantMessageReceiver.processRequest() method, it
tries to end the tenant flow within finally block and try to pop the object
which it pushes early. But right now, we have a new context stack and it
will throw the emptyStackException due to that.
@Sameera/KasunG: Do we really need to use the following code segment within
the above method?
*PrivilegedCarbonContext.destroyCurrentContext();*
I saw this method called within carbon-mediation components in 3 different
locations. AFAIU, we don't need to destroy the current context when we are
accessing the ThreadLocalContext. Please share your thoughts such that we
can fix this issue. Issue is fixed when I comment out the above line :)
Thanks,
Chanaka
On Wed, Oct 7, 2015 at 5:34 PM, Chanaka Fernando <[email protected]> wrote:
> Hi Sameera/KasunG,
>
> I have debugged the code to find the root cause for this empty stack
> exception. This is causing several other issues at the ESB layer. What
> actually happens is that inside the
> MultitenantMessageReceiver.processRequest() method, we have the following
> code segment.
>
> try {
> PrivilegedCarbonContext.startTenantFlow();
> PrivilegedCarbonContext privilegedCarbonContext =
> PrivilegedCarbonContext.getThreadLocalCarbonContext();
> privilegedCarbonContext.setTenantDomain(tenantDomain, true);
> // this is to prevent non-blocking transports from sending 202
>
> mainInMsgContext.getOperationContext().setProperty(Constants.RESPONSE_WRITTEN,
> "SKIP");
>
> ConfigurationContext tenantConfigCtx =
>
> TenantAxisUtils.getTenantConfigurationContext(tenantDomain,
>
> mainConfigCtx);
> if (tenantConfigCtx == null) {
> // Throw AxisFault: Tenant does not exist
> handleException(mainInMsgContext, new AxisFault("Tenant " +
> tenantDomain +
> " not found"));
> return;
> }
>
> if (mainInMsgContext.isDoingREST()) { // Handle REST requests
> doREST(mainInMsgContext, to, tenantDomain, tenantConfigCtx,
> serviceAndOperation);
> } else {
> doSOAP(mainInMsgContext, tenantDomain, tenantConfigCtx,
> serviceAndOperation);
> }
> } finally {
> PrivilegedCarbonContext.endTenantFlow();
> }
>
>
> When we are calling the endTenantFlow() method, it will go inside the
> following method within the CarbonContextDataHolder class.
>
> /**
> * This will end the tenant flow and restore the previous CarbonContext.
> */
> public void endTenantFlow() {
> Stack<CarbonContextDataHolder> carbonContextDataHolders =
> parentContextHolderStack.get();
> if (carbonContextDataHolders != null) {
> currentContextHolder.set(carbonContextDataHolders.pop());
> }
> }
>
>
> At the time of calling this method carbonContextDataHolders stack has
> become empty and causing the EmptyStackException. When I debug the code, I
> found that there is a scheduled task running in a different thread and
> accessing the same method frequently. This is coming from
> AbstractQuartzTaskManager class method.
>
> public void triggerComplete(Trigger trigger, JobExecutionContext
> jobExecutionContext,
> Trigger.CompletedExecutionInstruction completedExecutionInstruction) {
> PrivilegedCarbonContext.startTenantFlow();
>
> PrivilegedCarbonContext.getThreadLocalCarbonContext().setTenantId(getTenantId(),
> true);
> if(trigger.getNextFireTime() == null) {
> try {
> TaskUtils.setTaskFinished(getTaskRepository(),
> trigger.getJobKey().getName(), true);
> } catch (TaskException e) {
> log.error("Error in Finishing Task [" +
> trigger.getJobKey().getName() +
> "]: " + e.getMessage(), e);
> }
> }
> PrivilegedCarbonContext.endTenantFlow();
> }
>
>
> What I could not figure out is the thread which is emptying the stack. Is
> it possible that a different thread can access this
> carbonContextDataHolders stack and popping out the element before the
> PassThroughMessageProcessor thread access the same?
>
>
>
> On Wed, Jul 8, 2015 at 11:12 AM, Jagath Sisirakumara Ariyarathne <
> [email protected]> wrote:
>
>> Hi Sameera,
>>
>> I will check it and update.
>>
>> Thanks.
>>
>> On Wed, Jul 8, 2015 at 11:10 AM, Sameera Jayasoma <[email protected]>
>> wrote:
>>
>>> Hi Jagath,
>>>
>>> Can you debug and see whey the stack becomes empty? Thats a serious
>>> problem. Stack should be become empty here.
>>>
>>> Checking whether the stack is empty will stop the error log, but it
>>> doesn't fix the actual problem here.
>>>
>>> On Wed, Jul 8, 2015 at 10:50 AM, Sameera Jayasoma <[email protected]>
>>> wrote:
>>>
>>>> I understand, but we need to understand why that stack becomes empty.
>>>> AFAIK, if we follow the proper APIs, stack should not become empty
>>>>
>>>> On Tue, Jul 7, 2015 at 5:47 PM, Kasun Indrasiri <[email protected]> wrote:
>>>>
>>>>> Yeah, we should always check for an empty stack.
>>>>>
>>>>> On Tue, Jul 7, 2015 at 5:17 PM, Malaka Silva <[email protected]> wrote:
>>>>>
>>>>>> I think we need to check isEmpty as well.
>>>>>>
>>>>>> On Tue, Jul 7, 2015 at 3:41 PM, Jagath Sisirakumara Ariyarathne <
>>>>>> [email protected]> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am working on [1] and found that the cause of the exception
>>>>>>> mentioned below is in the code segment in org
>>>>>>> .wso2.carbon.context.internal.CarbonContextDataHolder in
>>>>>>> carbon.utils.
>>>>>>>
>>>>>>> public void endTenantFlow() {
>>>>>>>
>>>>>>> Stack<CarbonContextDataHolder> carbonContextDataHolders =
>>>>>>> parentContextHolderStack.get();
>>>>>>> if (carbonContextDataHolders != null) {
>>>>>>> currentContextHolder.set(carbonContextDataHolders.pop());
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>> *Exception :*
>>>>>>>
>>>>>>> java.util.EmptyStackException
>>>>>>> at java.util.Stack.peek(Stack.java:102)
>>>>>>> at java.util.Stack.pop(Stack.java:84)
>>>>>>> at
>>>>>>> org.wso2.carbon.context.internal.CarbonContextDataHolder.endTenantFlow(CarbonContextDataHolder.java:1291)
>>>>>>> at
>>>>>>> org.wso2.carbon.context.PrivilegedCarbonContext.endTenantFlow(PrivilegedCarbonContext.java:75)
>>>>>>> at
>>>>>>> org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute(TaskQuartzJobAdapter.java:69)
>>>>>>> at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>>>>>>> at
>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>
>>>>>>> Issue occurs when it tries to pop elements from
>>>>>>> carbonContextDataHolders stack when it is empty.
>>>>>>>
>>>>>>> Is it a possible scenario that this stack being empty and shouldn't it
>>>>>>> be handled at CarbonContextDataHolder (check isEmpty in stack)?
>>>>>>>
>>>>>>> [1] - https://wso2.org/jira/browse/ESBJAVA-3832
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> --
>>>>>>> Jagath Ariyarathne
>>>>>>> Technical Lead
>>>>>>> WSO2 Inc. http://wso2.com/
>>>>>>> Email: [email protected]
>>>>>>> Mob : +94 77 386 7048
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Best Regards,
>>>>>>
>>>>>> Malaka Silva
>>>>>> Senior Tech Lead
>>>>>> M: +94 777 219 791
>>>>>> Tel : 94 11 214 5345
>>>>>> Fax :94 11 2145300
>>>>>> Skype : malaka.sampath.silva
>>>>>> LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77
>>>>>> Blog : http://mrmalakasilva.blogspot.com/
>>>>>>
>>>>>> WSO2, Inc.
>>>>>> lean . enterprise . middleware
>>>>>> http://www.wso2.com/
>>>>>> http://www.wso2.com/about/team/malaka-silva/
>>>>>> <http://wso2.com/about/team/malaka-silva/>
>>>>>>
>>>>>> Save a tree -Conserve nature & Save the world for your future. Print
>>>>>> this email only if it is absolutely necessary.
>>>>>>
>>>>>> _______________________________________________
>>>>>> Dev mailing list
>>>>>> [email protected]
>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Kasun Indrasiri
>>>>> Software Architect
>>>>> WSO2, Inc.; http://wso2.com
>>>>> lean.enterprise.middleware
>>>>>
>>>>> cell: +94 77 556 5206
>>>>> Blog : http://kasunpanorama.blogspot.com/
>>>>>
>>>>> _______________________________________________
>>>>> Dev mailing list
>>>>> [email protected]
>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Sameera Jayasoma,
>>>> Software Architect,
>>>>
>>>> WSO2, Inc. (http://wso2.com)
>>>> email: [email protected]
>>>> blog: http://blog.sameera.org
>>>> twitter: https://twitter.com/sameerajayasoma
>>>> flickr: http://www.flickr.com/photos/sameera-jayasoma/collections
>>>> Mobile: 0094776364456
>>>>
>>>> Lean . Enterprise . Middleware
>>>>
>>>>
>>>
>>>
>>> --
>>> Sameera Jayasoma,
>>> Software Architect,
>>>
>>> WSO2, Inc. (http://wso2.com)
>>> email: [email protected]
>>> blog: http://blog.sameera.org
>>> twitter: https://twitter.com/sameerajayasoma
>>> flickr: http://www.flickr.com/photos/sameera-jayasoma/collections
>>> Mobile: 0094776364456
>>>
>>> Lean . Enterprise . Middleware
>>>
>>>
>>> _______________________________________________
>>> Dev mailing list
>>> [email protected]
>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>
>>>
>>
>>
>> --
>> Jagath Ariyarathne
>> Technical Lead
>> WSO2 Inc. http://wso2.com/
>> Email: [email protected]
>> Mob : +94 77 386 7048
>>
>>
>> _______________________________________________
>> Dev mailing list
>> [email protected]
>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>
>>
>
>
> --
> --
> Chanaka Fernando
> Senior Technical Lead
> WSO2, Inc.; http://wso2.com
> lean.enterprise.middleware
>
> mobile: +94 773337238
> Blog : http://soatutorials.blogspot.com
> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0
> Twitter:https://twitter.com/chanakaudaya
>
>
>
>
>
--
--
Chanaka Fernando
Senior Technical Lead
WSO2, Inc.; http://wso2.com
lean.enterprise.middleware
mobile: +94 773337238
Blog : http://soatutorials.blogspot.com
LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0
Twitter:https://twitter.com/chanakaudaya
_______________________________________________
Dev mailing list
[email protected]
http://wso2.org/cgi-bin/mailman/listinfo/dev
