Any update on this please?
On Thu, Oct 8, 2015 at 10:50 PM, Chanaka Fernando <chana...@wso2.com> wrote:
> Hi NuwanD,
>
> The fix is not in kernel but in carbon-mediation. What I need is kernel
> team's opinion on usage of the aforementioned method. If we can get their
> input, fix is straightforward.
>
> On Thu, Oct 8, 2015 at 9:58 PM, Nuwan Dias <nuw...@wso2.com> wrote:
>
>> Hi all,
>>
>> Will this be fixed in kernel 4.4.2? Its causing integration test failures
>> in API Manager and we would like this to be fixed soon.
>>
>> Thanks,
>> NuwanD.
>>
>> On Thu, Oct 8, 2015 at 6:22 PM, Chanaka Fernando <chana...@wso2.com>
>> wrote:
>>
>>> [Adding Kishanthan]
>>>
>>> On Thu, Oct 8, 2015 at 6:14 PM, Chanaka Fernando <chana...@wso2.com>
>>> wrote:
>>>
>>>> Hi All,
>>>>
>>>> At last, I was able to find the root cause for this behavior. This is
>>>> actually coming from carbon-mediation code. What actually happens is that
>>>> when ESB starts with a tenant, it will not load the tenant until it
>>>> receives the first request. When It receives the first request, it will
>>>> call the MultitenantMessageReceiver.processRequest() method. Within this
>>>> method, it will call the following method.
>>>>
>>>> PrivilegedCarbonContext.startTenantFlow();
>>>>
>>>>
>>>> This will get the ThreadLocal variable *parentContextHolderStack *and
>>>> get the stack and push the carbonContextDataHolder object to the
>>>> stack. After this, tenant loading happens and within the initialization of
>>>> the carbon mediation registry we have the following code segment.
>>>>
>>>> *org.wso2.carbon.mediation.registry.WSO2Registry.java*
>>>>
>>>> /**
>>>> * Carbon Kernel mandates to set Threadlocal before calling anything in
>>>> kernel
>>>> */
>>>> private void setTenantInfo() {
>>>> // Preserve user name
>>>> String username =
>>>> PrivilegedCarbonContext.getThreadLocalCarbonContext().getUsername();
>>>> *PrivilegedCarbonContext.destroyCurrentContext();*
>>>> PrivilegedCarbonContext cc =
>>>> PrivilegedCarbonContext.getThreadLocalCarbonContext();
>>>> cc.setTenantDomain(domain);
>>>> cc.setTenantId(tenantId);
>>>> if (username != null) { // Set back the user name
>>>>
>>>> PrivilegedCarbonContext.getThreadLocalCarbonContext().setUsername(username);
>>>> }
>>>> }
>>>>
>>>> Within the above method following line causes the issue.
>>>>
>>>> PrivilegedCarbonContext.destroyCurrentContext();
>>>>
>>>> When this method is called, it will reset the *parentContextHolderStack
>>>> *and the initial object push into the stack is destroyed. Then after
>>>> tenant loading, within the MultitenantMessageReceiver.processRequest()
>>>> method, it tries to end the tenant flow within finally block and try to pop
>>>> the object which it pushes early. But right now, we have a new context
>>>> stack and it will throw the emptyStackException due to that.
>>>>
>>>> @Sameera/KasunG: Do we really need to use the following code segment
>>>> within the above method?
>>>>
>>>>
>>>> *PrivilegedCarbonContext.destroyCurrentContext();*
>>>>
>>>> I saw this method called within carbon-mediation components in 3
>>>> different locations. AFAIU, we don't need to destroy the current context
>>>> when we are accessing the ThreadLocalContext. Please share your thoughts
>>>> such that we can fix this issue. Issue is fixed when I comment out the
>>>> above line :)
>>>>
>>>>
>>>> Thanks,
>>>> Chanaka
>>>>
>>>>
>>>>
>>>> On Wed, Oct 7, 2015 at 5:34 PM, Chanaka Fernando <chana...@wso2.com>
>>>> wrote:
>>>>
>>>>> Hi Sameera/KasunG,
>>>>>
>>>>> I have debugged the code to find the root cause for this empty stack
>>>>> exception. This is causing several other issues at the ESB layer. What
>>>>> actually happens is that inside the
>>>>> MultitenantMessageReceiver.processRequest() method, we have the following
>>>>> code segment.
>>>>>
>>>>> try {
>>>>> PrivilegedCarbonContext.startTenantFlow();
>>>>> PrivilegedCarbonContext privilegedCarbonContext =
>>>>> PrivilegedCarbonContext.getThreadLocalCarbonContext();
>>>>> privilegedCarbonContext.setTenantDomain(tenantDomain, true);
>>>>> // this is to prevent non-blocking transports from sending 202
>>>>>
>>>>> mainInMsgContext.getOperationContext().setProperty(Constants.RESPONSE_WRITTEN,
>>>>> "SKIP");
>>>>>
>>>>> ConfigurationContext tenantConfigCtx =
>>>>>
>>>>> TenantAxisUtils.getTenantConfigurationContext(tenantDomain,
>>>>>
>>>>> mainConfigCtx);
>>>>> if (tenantConfigCtx == null) {
>>>>> // Throw AxisFault: Tenant does not exist
>>>>> handleException(mainInMsgContext, new AxisFault("Tenant " +
>>>>> tenantDomain +
>>>>> " not found"));
>>>>> return;
>>>>> }
>>>>>
>>>>> if (mainInMsgContext.isDoingREST()) { // Handle REST requests
>>>>> doREST(mainInMsgContext, to, tenantDomain, tenantConfigCtx,
>>>>> serviceAndOperation);
>>>>> } else {
>>>>> doSOAP(mainInMsgContext, tenantDomain, tenantConfigCtx,
>>>>> serviceAndOperation);
>>>>> }
>>>>> } finally {
>>>>> PrivilegedCarbonContext.endTenantFlow();
>>>>> }
>>>>>
>>>>>
>>>>> When we are calling the endTenantFlow() method, it will go inside the
>>>>> following method within the CarbonContextDataHolder class.
>>>>>
>>>>> /**
>>>>> * This will end the tenant flow and restore the previous CarbonContext.
>>>>> */
>>>>> public void endTenantFlow() {
>>>>> Stack<CarbonContextDataHolder> carbonContextDataHolders =
>>>>> parentContextHolderStack.get();
>>>>> if (carbonContextDataHolders != null) {
>>>>> currentContextHolder.set(carbonContextDataHolders.pop());
>>>>> }
>>>>> }
>>>>>
>>>>>
>>>>> At the time of calling this method carbonContextDataHolders stack has
>>>>> become empty and causing the EmptyStackException. When I debug the code, I
>>>>> found that there is a scheduled task running in a different thread and
>>>>> accessing the same method frequently. This is coming from
>>>>> AbstractQuartzTaskManager class method.
>>>>>
>>>>> public void triggerComplete(Trigger trigger, JobExecutionContext
>>>>> jobExecutionContext,
>>>>> Trigger.CompletedExecutionInstruction
>>>>> completedExecutionInstruction) {
>>>>> PrivilegedCarbonContext.startTenantFlow();
>>>>>
>>>>> PrivilegedCarbonContext.getThreadLocalCarbonContext().setTenantId(getTenantId(),
>>>>> true);
>>>>> if(trigger.getNextFireTime() == null) {
>>>>> try {
>>>>> TaskUtils.setTaskFinished(getTaskRepository(),
>>>>> trigger.getJobKey().getName(), true);
>>>>> } catch (TaskException e) {
>>>>> log.error("Error in Finishing Task [" +
>>>>> trigger.getJobKey().getName() +
>>>>> "]: " + e.getMessage(), e);
>>>>> }
>>>>> }
>>>>> PrivilegedCarbonContext.endTenantFlow();
>>>>> }
>>>>>
>>>>>
>>>>> What I could not figure out is the thread which is emptying the stack.
>>>>> Is it possible that a different thread can access this
>>>>> carbonContextDataHolders stack and popping out the element before the
>>>>> PassThroughMessageProcessor thread access the same?
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Jul 8, 2015 at 11:12 AM, Jagath Sisirakumara Ariyarathne <
>>>>> jaga...@wso2.com> wrote:
>>>>>
>>>>>> Hi Sameera,
>>>>>>
>>>>>> I will check it and update.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> On Wed, Jul 8, 2015 at 11:10 AM, Sameera Jayasoma <same...@wso2.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Jagath,
>>>>>>>
>>>>>>> Can you debug and see whey the stack becomes empty? Thats a serious
>>>>>>> problem. Stack should be become empty here.
>>>>>>>
>>>>>>> Checking whether the stack is empty will stop the error log, but it
>>>>>>> doesn't fix the actual problem here.
>>>>>>>
>>>>>>> On Wed, Jul 8, 2015 at 10:50 AM, Sameera Jayasoma <same...@wso2.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I understand, but we need to understand why that stack becomes
>>>>>>>> empty. AFAIK, if we follow the proper APIs, stack should not become
>>>>>>>> empty
>>>>>>>>
>>>>>>>> On Tue, Jul 7, 2015 at 5:47 PM, Kasun Indrasiri <ka...@wso2.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Yeah, we should always check for an empty stack.
>>>>>>>>>
>>>>>>>>> On Tue, Jul 7, 2015 at 5:17 PM, Malaka Silva <mal...@wso2.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I think we need to check isEmpty as well.
>>>>>>>>>>
>>>>>>>>>> On Tue, Jul 7, 2015 at 3:41 PM, Jagath Sisirakumara Ariyarathne <
>>>>>>>>>> jaga...@wso2.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I am working on [1] and found that the cause of the exception
>>>>>>>>>>> mentioned below is in the code segment in org
>>>>>>>>>>> .wso2.carbon.context.internal.CarbonContextDataHolder in
>>>>>>>>>>> carbon.utils.
>>>>>>>>>>>
>>>>>>>>>>> public void endTenantFlow() {
>>>>>>>>>>>
>>>>>>>>>>> Stack<CarbonContextDataHolder> carbonContextDataHolders =
>>>>>>>>>>> parentContextHolderStack.get();
>>>>>>>>>>> if (carbonContextDataHolders != null) {
>>>>>>>>>>> currentContextHolder.set(carbonContextDataHolders.pop());
>>>>>>>>>>> }
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> *Exception :*
>>>>>>>>>>>
>>>>>>>>>>> java.util.EmptyStackException
>>>>>>>>>>> at java.util.Stack.peek(Stack.java:102)
>>>>>>>>>>> at java.util.Stack.pop(Stack.java:84)
>>>>>>>>>>> at
>>>>>>>>>>> org.wso2.carbon.context.internal.CarbonContextDataHolder.endTenantFlow(CarbonContextDataHolder.java:1291)
>>>>>>>>>>> at
>>>>>>>>>>> org.wso2.carbon.context.PrivilegedCarbonContext.endTenantFlow(PrivilegedCarbonContext.java:75)
>>>>>>>>>>> at
>>>>>>>>>>> org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute(TaskQuartzJobAdapter.java:69)
>>>>>>>>>>> at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>>>>
>>>>>>>>>>> Issue occurs when it tries to pop elements from
>>>>>>>>>>> carbonContextDataHolders stack when it is empty.
>>>>>>>>>>>
>>>>>>>>>>> Is it a possible scenario that this stack being empty and shouldn't
>>>>>>>>>>> it be handled at CarbonContextDataHolder (check isEmpty in stack)?
>>>>>>>>>>>
>>>>>>>>>>> [1] - https://wso2.org/jira/browse/ESBJAVA-3832
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Jagath Ariyarathne
>>>>>>>>>>> Technical Lead
>>>>>>>>>>> WSO2 Inc. http://wso2.com/
>>>>>>>>>>> Email: jaga...@wso2.com
>>>>>>>>>>> Mob : +94 77 386 7048
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>> Best Regards,
>>>>>>>>>>
>>>>>>>>>> Malaka Silva
>>>>>>>>>> Senior Tech Lead
>>>>>>>>>> M: +94 777 219 791
>>>>>>>>>> Tel : 94 11 214 5345
>>>>>>>>>> Fax :94 11 2145300
>>>>>>>>>> Skype : malaka.sampath.silva
>>>>>>>>>> LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77
>>>>>>>>>> Blog : http://mrmalakasilva.blogspot.com/
>>>>>>>>>>
>>>>>>>>>> WSO2, Inc.
>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>> http://www.wso2.com/
>>>>>>>>>> http://www.wso2.com/about/team/malaka-silva/
>>>>>>>>>> <http://wso2.com/about/team/malaka-silva/>
>>>>>>>>>>
>>>>>>>>>> Save a tree -Conserve nature & Save the world for your future.
>>>>>>>>>> Print this email only if it is absolutely necessary.
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Dev mailing list
>>>>>>>>>> Dev@wso2.org
>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Kasun Indrasiri
>>>>>>>>> Software Architect
>>>>>>>>> WSO2, Inc.; http://wso2.com
>>>>>>>>> lean.enterprise.middleware
>>>>>>>>>
>>>>>>>>> cell: +94 77 556 5206
>>>>>>>>> Blog : http://kasunpanorama.blogspot.com/
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Dev mailing list
>>>>>>>>> Dev@wso2.org
>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Sameera Jayasoma,
>>>>>>>> Software Architect,
>>>>>>>>
>>>>>>>> WSO2, Inc. (http://wso2.com)
>>>>>>>> email: same...@wso2.com
>>>>>>>> blog: http://blog.sameera.org
>>>>>>>> twitter: https://twitter.com/sameerajayasoma
>>>>>>>> flickr: http://www.flickr.com/photos/sameera-jayasoma/collections
>>>>>>>> Mobile: 0094776364456
>>>>>>>>
>>>>>>>> Lean . Enterprise . Middleware
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Sameera Jayasoma,
>>>>>>> Software Architect,
>>>>>>>
>>>>>>> WSO2, Inc. (http://wso2.com)
>>>>>>> email: same...@wso2.com
>>>>>>> blog: http://blog.sameera.org
>>>>>>> twitter: https://twitter.com/sameerajayasoma
>>>>>>> flickr: http://www.flickr.com/photos/sameera-jayasoma/collections
>>>>>>> Mobile: 0094776364456
>>>>>>>
>>>>>>> Lean . Enterprise . Middleware
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Dev mailing list
>>>>>>> Dev@wso2.org
>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Jagath Ariyarathne
>>>>>> Technical Lead
>>>>>> WSO2 Inc. http://wso2.com/
>>>>>> Email: jaga...@wso2.com
>>>>>> Mob : +94 77 386 7048
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Dev mailing list
>>>>>> Dev@wso2.org
>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> --
>>>>> Chanaka Fernando
>>>>> Senior Technical Lead
>>>>> WSO2, Inc.; http://wso2.com
>>>>> lean.enterprise.middleware
>>>>>
>>>>> mobile: +94 773337238
>>>>> Blog : http://soatutorials.blogspot.com
>>>>> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0
>>>>> Twitter:https://twitter.com/chanakaudaya
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> --
>>>> Chanaka Fernando
>>>> Senior Technical Lead
>>>> WSO2, Inc.; http://wso2.com
>>>> lean.enterprise.middleware
>>>>
>>>> mobile: +94 773337238
>>>> Blog : http://soatutorials.blogspot.com
>>>> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0
>>>> Twitter:https://twitter.com/chanakaudaya
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> --
>>> Chanaka Fernando
>>> Senior Technical Lead
>>> WSO2, Inc.; http://wso2.com
>>> lean.enterprise.middleware
>>>
>>> mobile: +94 773337238
>>> Blog : http://soatutorials.blogspot.com
>>> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0
>>> Twitter:https://twitter.com/chanakaudaya
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Dev mailing list
>>> Dev@wso2.org
>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>
>>>
>>
>>
>> --
>> Nuwan Dias
>>
>> Technical Lead - WSO2, Inc. http://wso2.com
>> email : nuw...@wso2.com
>> Phone : +94 777 775 729
>>
>
>
>
> --
> --
> Chanaka Fernando
> Senior Technical Lead
> WSO2, Inc.; http://wso2.com
> lean.enterprise.middleware
>
> mobile: +94 773337238
> Blog : http://soatutorials.blogspot.com
> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0
> Twitter:https://twitter.com/chanakaudaya
>
>
>
>
>
--
--
Chanaka Fernando
Senior Technical Lead
WSO2, Inc.; http://wso2.com
lean.enterprise.middleware
mobile: +94 773337238
Blog : http://soatutorials.blogspot.com
LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0
Twitter:https://twitter.com/chanakaudaya
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
