Hi NuwanD,
The fix is not in kernel but in carbon-mediation. What I need is kernel
team's opinion on usage of the aforementioned method. If we can get their
input, fix is straightforward.
On Thu, Oct 8, 2015 at 9:58 PM, Nuwan Dias <nuw...@wso2.com> wrote:
> Hi all,
>
> Will this be fixed in kernel 4.4.2? Its causing integration test failures
> in API Manager and we would like this to be fixed soon.
>
> Thanks,
> NuwanD.
>
> On Thu, Oct 8, 2015 at 6:22 PM, Chanaka Fernando <chana...@wso2.com>
> wrote:
>
>> [Adding Kishanthan]
>>
>> On Thu, Oct 8, 2015 at 6:14 PM, Chanaka Fernando <chana...@wso2.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> At last, I was able to find the root cause for this behavior. This is
>>> actually coming from carbon-mediation code. What actually happens is that
>>> when ESB starts with a tenant, it will not load the tenant until it
>>> receives the first request. When It receives the first request, it will
>>> call the MultitenantMessageReceiver.processRequest() method. Within this
>>> method, it will call the following method.
>>>
>>> PrivilegedCarbonContext.startTenantFlow();
>>>
>>>
>>> This will get the ThreadLocal variable *parentContextHolderStack *and
>>> get the stack and push the carbonContextDataHolder object to the
>>> stack. After this, tenant loading happens and within the initialization of
>>> the carbon mediation registry we have the following code segment.
>>>
>>> *org.wso2.carbon.mediation.registry.WSO2Registry.java*
>>>
>>> /**
>>> * Carbon Kernel mandates to set Threadlocal before calling anything in
>>> kernel
>>> */
>>> private void setTenantInfo() {
>>> // Preserve user name
>>> String username =
>>> PrivilegedCarbonContext.getThreadLocalCarbonContext().getUsername();
>>> *PrivilegedCarbonContext.destroyCurrentContext();*
>>> PrivilegedCarbonContext cc =
>>> PrivilegedCarbonContext.getThreadLocalCarbonContext();
>>> cc.setTenantDomain(domain);
>>> cc.setTenantId(tenantId);
>>> if (username != null) { // Set back the user name
>>>
>>> PrivilegedCarbonContext.getThreadLocalCarbonContext().setUsername(username);
>>> }
>>> }
>>>
>>> Within the above method following line causes the issue.
>>>
>>> PrivilegedCarbonContext.destroyCurrentContext();
>>>
>>> When this method is called, it will reset the *parentContextHolderStack
>>> *and the initial object push into the stack is destroyed. Then after
>>> tenant loading, within the MultitenantMessageReceiver.processRequest()
>>> method, it tries to end the tenant flow within finally block and try to pop
>>> the object which it pushes early. But right now, we have a new context
>>> stack and it will throw the emptyStackException due to that.
>>>
>>> @Sameera/KasunG: Do we really need to use the following code segment
>>> within the above method?
>>>
>>>
>>> *PrivilegedCarbonContext.destroyCurrentContext();*
>>>
>>> I saw this method called within carbon-mediation components in 3
>>> different locations. AFAIU, we don't need to destroy the current context
>>> when we are accessing the ThreadLocalContext. Please share your thoughts
>>> such that we can fix this issue. Issue is fixed when I comment out the
>>> above line :)
>>>
>>>
>>> Thanks,
>>> Chanaka
>>>
>>>
>>>
>>> On Wed, Oct 7, 2015 at 5:34 PM, Chanaka Fernando <chana...@wso2.com>
>>> wrote:
>>>
>>>> Hi Sameera/KasunG,
>>>>
>>>> I have debugged the code to find the root cause for this empty stack
>>>> exception. This is causing several other issues at the ESB layer. What
>>>> actually happens is that inside the
>>>> MultitenantMessageReceiver.processRequest() method, we have the following
>>>> code segment.
>>>>
>>>> try {
>>>> PrivilegedCarbonContext.startTenantFlow();
>>>> PrivilegedCarbonContext privilegedCarbonContext =
>>>> PrivilegedCarbonContext.getThreadLocalCarbonContext();
>>>> privilegedCarbonContext.setTenantDomain(tenantDomain, true);
>>>> // this is to prevent non-blocking transports from sending 202
>>>>
>>>> mainInMsgContext.getOperationContext().setProperty(Constants.RESPONSE_WRITTEN,
>>>> "SKIP");
>>>>
>>>> ConfigurationContext tenantConfigCtx =
>>>>
>>>> TenantAxisUtils.getTenantConfigurationContext(tenantDomain,
>>>>
>>>> mainConfigCtx);
>>>> if (tenantConfigCtx == null) {
>>>> // Throw AxisFault: Tenant does not exist
>>>> handleException(mainInMsgContext, new AxisFault("Tenant " +
>>>> tenantDomain +
>>>> " not found"));
>>>> return;
>>>> }
>>>>
>>>> if (mainInMsgContext.isDoingREST()) { // Handle REST requests
>>>> doREST(mainInMsgContext, to, tenantDomain, tenantConfigCtx,
>>>> serviceAndOperation);
>>>> } else {
>>>> doSOAP(mainInMsgContext, tenantDomain, tenantConfigCtx,
>>>> serviceAndOperation);
>>>> }
>>>> } finally {
>>>> PrivilegedCarbonContext.endTenantFlow();
>>>> }
>>>>
>>>>
>>>> When we are calling the endTenantFlow() method, it will go inside the
>>>> following method within the CarbonContextDataHolder class.
>>>>
>>>> /**
>>>> * This will end the tenant flow and restore the previous CarbonContext.
>>>> */
>>>> public void endTenantFlow() {
>>>> Stack<CarbonContextDataHolder> carbonContextDataHolders =
>>>> parentContextHolderStack.get();
>>>> if (carbonContextDataHolders != null) {
>>>> currentContextHolder.set(carbonContextDataHolders.pop());
>>>> }
>>>> }
>>>>
>>>>
>>>> At the time of calling this method carbonContextDataHolders stack has
>>>> become empty and causing the EmptyStackException. When I debug the code, I
>>>> found that there is a scheduled task running in a different thread and
>>>> accessing the same method frequently. This is coming from
>>>> AbstractQuartzTaskManager class method.
>>>>
>>>> public void triggerComplete(Trigger trigger, JobExecutionContext
>>>> jobExecutionContext,
>>>> Trigger.CompletedExecutionInstruction
>>>> completedExecutionInstruction) {
>>>> PrivilegedCarbonContext.startTenantFlow();
>>>>
>>>> PrivilegedCarbonContext.getThreadLocalCarbonContext().setTenantId(getTenantId(),
>>>> true);
>>>> if(trigger.getNextFireTime() == null) {
>>>> try {
>>>> TaskUtils.setTaskFinished(getTaskRepository(),
>>>> trigger.getJobKey().getName(), true);
>>>> } catch (TaskException e) {
>>>> log.error("Error in Finishing Task [" +
>>>> trigger.getJobKey().getName() +
>>>> "]: " + e.getMessage(), e);
>>>> }
>>>> }
>>>> PrivilegedCarbonContext.endTenantFlow();
>>>> }
>>>>
>>>>
>>>> What I could not figure out is the thread which is emptying the stack.
>>>> Is it possible that a different thread can access this
>>>> carbonContextDataHolders stack and popping out the element before the
>>>> PassThroughMessageProcessor thread access the same?
>>>>
>>>>
>>>>
>>>> On Wed, Jul 8, 2015 at 11:12 AM, Jagath Sisirakumara Ariyarathne <
>>>> jaga...@wso2.com> wrote:
>>>>
>>>>> Hi Sameera,
>>>>>
>>>>> I will check it and update.
>>>>>
>>>>> Thanks.
>>>>>
>>>>> On Wed, Jul 8, 2015 at 11:10 AM, Sameera Jayasoma <same...@wso2.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Jagath,
>>>>>>
>>>>>> Can you debug and see whey the stack becomes empty? Thats a serious
>>>>>> problem. Stack should be become empty here.
>>>>>>
>>>>>> Checking whether the stack is empty will stop the error log, but it
>>>>>> doesn't fix the actual problem here.
>>>>>>
>>>>>> On Wed, Jul 8, 2015 at 10:50 AM, Sameera Jayasoma <same...@wso2.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I understand, but we need to understand why that stack becomes
>>>>>>> empty. AFAIK, if we follow the proper APIs, stack should not become
>>>>>>> empty
>>>>>>>
>>>>>>> On Tue, Jul 7, 2015 at 5:47 PM, Kasun Indrasiri <ka...@wso2.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Yeah, we should always check for an empty stack.
>>>>>>>>
>>>>>>>> On Tue, Jul 7, 2015 at 5:17 PM, Malaka Silva <mal...@wso2.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I think we need to check isEmpty as well.
>>>>>>>>>
>>>>>>>>> On Tue, Jul 7, 2015 at 3:41 PM, Jagath Sisirakumara Ariyarathne <
>>>>>>>>> jaga...@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I am working on [1] and found that the cause of the exception
>>>>>>>>>> mentioned below is in the code segment in org
>>>>>>>>>> .wso2.carbon.context.internal.CarbonContextDataHolder in
>>>>>>>>>> carbon.utils.
>>>>>>>>>>
>>>>>>>>>> public void endTenantFlow() {
>>>>>>>>>>
>>>>>>>>>> Stack<CarbonContextDataHolder> carbonContextDataHolders =
>>>>>>>>>> parentContextHolderStack.get();
>>>>>>>>>> if (carbonContextDataHolders != null) {
>>>>>>>>>> currentContextHolder.set(carbonContextDataHolders.pop());
>>>>>>>>>> }
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> *Exception :*
>>>>>>>>>>
>>>>>>>>>> java.util.EmptyStackException
>>>>>>>>>> at java.util.Stack.peek(Stack.java:102)
>>>>>>>>>> at java.util.Stack.pop(Stack.java:84)
>>>>>>>>>> at
>>>>>>>>>> org.wso2.carbon.context.internal.CarbonContextDataHolder.endTenantFlow(CarbonContextDataHolder.java:1291)
>>>>>>>>>> at
>>>>>>>>>> org.wso2.carbon.context.PrivilegedCarbonContext.endTenantFlow(PrivilegedCarbonContext.java:75)
>>>>>>>>>> at
>>>>>>>>>> org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute(TaskQuartzJobAdapter.java:69)
>>>>>>>>>> at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>>>
>>>>>>>>>> Issue occurs when it tries to pop elements from
>>>>>>>>>> carbonContextDataHolders stack when it is empty.
>>>>>>>>>>
>>>>>>>>>> Is it a possible scenario that this stack being empty and shouldn't
>>>>>>>>>> it be handled at CarbonContextDataHolder (check isEmpty in stack)?
>>>>>>>>>>
>>>>>>>>>> [1] - https://wso2.org/jira/browse/ESBJAVA-3832
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Jagath Ariyarathne
>>>>>>>>>> Technical Lead
>>>>>>>>>> WSO2 Inc. http://wso2.com/
>>>>>>>>>> Email: jaga...@wso2.com
>>>>>>>>>> Mob : +94 77 386 7048
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> Best Regards,
>>>>>>>>>
>>>>>>>>> Malaka Silva
>>>>>>>>> Senior Tech Lead
>>>>>>>>> M: +94 777 219 791
>>>>>>>>> Tel : 94 11 214 5345
>>>>>>>>> Fax :94 11 2145300
>>>>>>>>> Skype : malaka.sampath.silva
>>>>>>>>> LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77
>>>>>>>>> Blog : http://mrmalakasilva.blogspot.com/
>>>>>>>>>
>>>>>>>>> WSO2, Inc.
>>>>>>>>> lean . enterprise . middleware
>>>>>>>>> http://www.wso2.com/
>>>>>>>>> http://www.wso2.com/about/team/malaka-silva/
>>>>>>>>> <http://wso2.com/about/team/malaka-silva/>
>>>>>>>>>
>>>>>>>>> Save a tree -Conserve nature & Save the world for your future.
>>>>>>>>> Print this email only if it is absolutely necessary.
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Dev mailing list
>>>>>>>>> Dev@wso2.org
>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Kasun Indrasiri
>>>>>>>> Software Architect
>>>>>>>> WSO2, Inc.; http://wso2.com
>>>>>>>> lean.enterprise.middleware
>>>>>>>>
>>>>>>>> cell: +94 77 556 5206
>>>>>>>> Blog : http://kasunpanorama.blogspot.com/
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Dev mailing list
>>>>>>>> Dev@wso2.org
>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Sameera Jayasoma,
>>>>>>> Software Architect,
>>>>>>>
>>>>>>> WSO2, Inc. (http://wso2.com)
>>>>>>> email: same...@wso2.com
>>>>>>> blog: http://blog.sameera.org
>>>>>>> twitter: https://twitter.com/sameerajayasoma
>>>>>>> flickr: http://www.flickr.com/photos/sameera-jayasoma/collections
>>>>>>> Mobile: 0094776364456
>>>>>>>
>>>>>>> Lean . Enterprise . Middleware
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Sameera Jayasoma,
>>>>>> Software Architect,
>>>>>>
>>>>>> WSO2, Inc. (http://wso2.com)
>>>>>> email: same...@wso2.com
>>>>>> blog: http://blog.sameera.org
>>>>>> twitter: https://twitter.com/sameerajayasoma
>>>>>> flickr: http://www.flickr.com/photos/sameera-jayasoma/collections
>>>>>> Mobile: 0094776364456
>>>>>>
>>>>>> Lean . Enterprise . Middleware
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Dev mailing list
>>>>>> Dev@wso2.org
>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Jagath Ariyarathne
>>>>> Technical Lead
>>>>> WSO2 Inc. http://wso2.com/
>>>>> Email: jaga...@wso2.com
>>>>> Mob : +94 77 386 7048
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Dev mailing list
>>>>> Dev@wso2.org
>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> --
>>>> Chanaka Fernando
>>>> Senior Technical Lead
>>>> WSO2, Inc.; http://wso2.com
>>>> lean.enterprise.middleware
>>>>
>>>> mobile: +94 773337238
>>>> Blog : http://soatutorials.blogspot.com
>>>> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0
>>>> Twitter:https://twitter.com/chanakaudaya
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> --
>>> Chanaka Fernando
>>> Senior Technical Lead
>>> WSO2, Inc.; http://wso2.com
>>> lean.enterprise.middleware
>>>
>>> mobile: +94 773337238
>>> Blog : http://soatutorials.blogspot.com
>>> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0
>>> Twitter:https://twitter.com/chanakaudaya
>>>
>>>
>>>
>>>
>>>
>>
>>
>> --
>> --
>> Chanaka Fernando
>> Senior Technical Lead
>> WSO2, Inc.; http://wso2.com
>> lean.enterprise.middleware
>>
>> mobile: +94 773337238
>> Blog : http://soatutorials.blogspot.com
>> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0
>> Twitter:https://twitter.com/chanakaudaya
>>
>>
>>
>>
>>
>> _______________________________________________
>> Dev mailing list
>> Dev@wso2.org
>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>
>>
>
>
> --
> Nuwan Dias
>
> Technical Lead - WSO2, Inc. http://wso2.com
> email : nuw...@wso2.com
> Phone : +94 777 775 729
>
--
--
Chanaka Fernando
Senior Technical Lead
WSO2, Inc.; http://wso2.com
lean.enterprise.middleware
mobile: +94 773337238
Blog : http://soatutorials.blogspot.com
LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0
Twitter:https://twitter.com/chanakaudaya
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
