Re: [Dev] [ESB] EmptyStackException when resuming a paused message processor

[Sanjeewa Malalgoda]

Please find the complete error log below.

2016-01-26 15:30:27,128] ERROR - PassThroughHttpSender Failed to submit the
response
java.lang.NullPointerException
at
org.apache.synapse.transport.passthru.util.SourceResponseFactory.create(SourceResponseFactory.java:64)
at
org.apache.synapse.transport.passthru.PassThroughHttpSender.submitResponse(PassThroughHttpSender.java:462)
at
org.apache.synapse.transport.passthru.PassThroughHttpSender.invoke(PassThroughHttpSender.java:267)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
at org.apache.synapse.core.axis2.Axis2Sender.sendBack(Axis2Sender.java:212)
at
org.apache.synapse.core.axis2.Axis2SynapseEnvironment.send(Axis2SynapseEnvironment.java:493)
at
org.apache.synapse.mediators.builtin.SendMediator.mediate(SendMediator.java:108)
at
org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:81)
at
org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:48)
at
org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:155)
at org.apache.synapse.rest.Resource.process(Resource.java:297)
at org.apache.synapse.rest.API.process(API.java:335)
at
org.apache.synapse.rest.RESTRequestHandler.dispatchToAPI(RESTRequestHandler.java:86)
at
org.apache.synapse.rest.RESTRequestHandler.process(RESTRequestHandler.java:52)
at
org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:295)
at
org.apache.synapse.core.axis2.SynapseCallbackReceiver.handleMessage(SynapseCallbackReceiver.java:529)
at
org.apache.synapse.core.axis2.SynapseCallbackReceiver.receive(SynapseCallbackReceiver.java:172)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at
org.apache.synapse.transport.passthru.ClientWorker.run(ClientWorker.java:251)
at
org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2016-01-26 15:30:27,138] ERROR - Axis2Sender
Accept-Ranges:bytes,Access-Control-Allow-Headers:authorization,Access-Control-Allow-Origin,Content-Type,Access-Control-Allow-Methods:POST,PATCH,GET,DELETE,PUT,Access-Control-Allow-Origin:*,ETag:"b1-4fdc9b19d2b93",Last-Modified:Wed,
09 Jul 2014 21:50:16 GMT,Vary:Accept-Encoding,http://schemas.xmlsoap.org/soap/envelope/;>
Unexpected error sending message back
org.apache.axis2.AxisFault: Failed to submit the response
at
org.apache.synapse.transport.passthru.PassThroughHttpSender.handleException(PassThroughHttpSender.java:610)
at
org.apache.synapse.transport.passthru.PassThroughHttpSender.invoke(PassThroughHttpSender.java:269)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
at org.apache.synapse.core.axis2.Axis2Sender.sendBack(Axis2Sender.java:212)
at
org.apache.synapse.core.axis2.Axis2SynapseEnvironment.send(Axis2SynapseEnvironment.java:493)
at
org.apache.synapse.mediators.builtin.SendMediator.mediate(SendMediator.java:108)
at
org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:81)
at
org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:48)
at
org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:155)
at org.apache.synapse.rest.Resource.process(Resource.java:297)
at org.apache.synapse.rest.API.process(API.java:335)
at
org.apache.synapse.rest.RESTRequestHandler.dispatchToAPI(RESTRequestHandler.java:86)
at
org.apache.synapse.rest.RESTRequestHandler.process(RESTRequestHandler.java:52)
at
org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:295)
at
org.apache.synapse.core.axis2.SynapseCallbackReceiver.handleMessage(SynapseCallbackReceiver.java:529)
at
org.apache.synapse.core.axis2.SynapseCallbackReceiver.receive(SynapseCallbackReceiver.java:172)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at
org.apache.synapse.transport.passthru.ClientWorker.run(ClientWorker.java:251)
at
org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at
org.apache.synapse.transport.passthru.util.SourceResponseFactory.create(SourceResponseFactory.java:64)
at
org.apache.synapse.transport.passthru.PassThroughHttpSender.submitResponse(PassThroughHttpSender.java:462)
at
org.apache.synapse.transport.passthru.PassThroughHttpSender.invoke(PassThroughHttpSender.java:267)
... 20 more
[2016-01-26 15:30:27,156]  INFO - LogMediator STATUS = Executing default
'fault' sequence, ERROR_CODE = 0, ERROR_MESSAGE =

Re: [Dev] [ESB] EmptyStackException when resuming a paused message processor

[Sanjeewa Malalgoda]

Hi All,
Did this fix included to last release?
I'm getting exact same issue in API Manager 1.10 with high concurrency.
Getting exact same NPE mentioned in below JIRA[1]

[1]https://wso2.org/jira/browse/ESBJAVA-4142

Thanks,
sanjeewa.


On Wed, Oct 14, 2015 at 11:09 AM, Jagath Sisirakumara Ariyarathne <
jaga...@wso2.com> wrote:

> Hi Nuwan,
>
> Our plan is to do the first milestone release end of next week. With this
> we hope to incorporate latest carbon-deployment and kernel releases (if it
> released during this week).
>
> Thanks.
>
> On Wed, Oct 14, 2015 at 11:07 AM, Chanaka Fernando 
> wrote:
>
>> [Adding Jagath]
>>
>> On Wed, Oct 14, 2015 at 10:17 AM, Nuwan Dias  wrote:
>>
>>> Hi ESB folks,
>>>
>>> Can we get a milestone release of synapse and carbon-mediation including
>>> this fix and another PR I sent to synapse? We have to do a milestone
>>> release of API Manager this week.
>>>
>>> Thanks,
>>> NuwanD.
>>>
>>> On Mon, Oct 12, 2015 at 11:43 AM, Nuwan Dias  wrote:
>>>
 Thanks a lot Chanaka.

 On Mon, Oct 12, 2015 at 11:39 AM, Chanaka Fernando 
 wrote:

> Hi All,
>
> This issue is fixed with the following PR [1]. Thanks for reporting
> and helping us to figure out the issue.
>
> [1] https://github.com/wso2/carbon-mediation/pull/467
>
> On Fri, Oct 9, 2015 at 3:41 PM, Chanaka Fernando 
> wrote:
>
>> Hi Sameera,
>>
>> Your judgment is correct and it helped us to find the root cause :).
>> I'll fix the code from carbon-mediation side. Thanks for your input.
>>
>> On Fri, Oct 9, 2015 at 3:19 PM, Sameera Jayasoma 
>> wrote:
>>
>>> Hi Chanaka,
>>>
>>> You don't need to destroy the current context. This part is handle
>>> by the Kernel code. Users of the CarbonContext API do not need to worry
>>> about destroying the context.
>>>
>>> Who has done this change? So my initial judgment is correct. :)
>>> Stack cannot become empty.
>>>
>>> Thanks,
>>> Sameera.
>>>
>>> On Thu, Oct 8, 2015 at 6:14 PM, Chanaka Fernando 
>>> wrote:
>>>
 Hi All,

 At last, I was able to find the root cause for this behavior. This
 is actually coming from carbon-mediation code. What actually happens is
 that when ESB starts with a tenant, it will not load the tenant until 
 it
 receives the first request. When It receives the first request, it will
 call the MultitenantMessageReceiver.processRequest() method. Within 
 this
 method, it will call the following method.

 PrivilegedCarbonContext.startTenantFlow();


 This will get the ThreadLocal variable *parentContextHolderStack *and
 get the stack  and push the carbonContextDataHolder object to the
 stack. After this, tenant loading happens and within the 
 initialization of
 the carbon mediation registry we have the following code segment.

 *org.wso2.carbon.mediation.registry.WSO2Registry.java*

 /**
  *  Carbon Kernel mandates to set Threadlocal before calling anything 
 in kernel
  */
 private void setTenantInfo() {
 // Preserve user name
 String username = 
 PrivilegedCarbonContext.getThreadLocalCarbonContext().getUsername();
 *PrivilegedCarbonContext.destroyCurrentContext();*
 PrivilegedCarbonContext cc = 
 PrivilegedCarbonContext.getThreadLocalCarbonContext();
 cc.setTenantDomain(domain);
 cc.setTenantId(tenantId);
 if (username != null) { // Set back the user name
 
 PrivilegedCarbonContext.getThreadLocalCarbonContext().setUsername(username);
 }
 }

 Within the above method following line causes the issue.

 PrivilegedCarbonContext.destroyCurrentContext();

 When this method is called, it will reset the *parentContextHolderStack
 *and the initial object push into the stack is destroyed. Then
 after tenant loading, within the
 MultitenantMessageReceiver.processRequest() method, it tries to end the
 tenant flow within finally block and try to pop the object which it 
 pushes
 early. But right now, we have a new context stack and it will throw the
 emptyStackException due to that.

 @Sameera/KasunG: Do we really need to use the following code
 segment within the above method?


 *PrivilegedCarbonContext.destroyCurrentContext();*

 I saw this method called within carbon-mediation components in 3
 different locations. AFAIU, we don't need to destroy the current 
 

Re: [Dev] [ESB] EmptyStackException when resuming a paused message processor

[Nuwan Dias]

Hi ESB folks,

Can we get a milestone release of synapse and carbon-mediation including
this fix and another PR I sent to synapse? We have to do a milestone
release of API Manager this week.

Thanks,
NuwanD.

On Mon, Oct 12, 2015 at 11:43 AM, Nuwan Dias  wrote:

> Thanks a lot Chanaka.
>
> On Mon, Oct 12, 2015 at 11:39 AM, Chanaka Fernando 
> wrote:
>
>> Hi All,
>>
>> This issue is fixed with the following PR [1]. Thanks for reporting and
>> helping us to figure out the issue.
>>
>> [1] https://github.com/wso2/carbon-mediation/pull/467
>>
>> On Fri, Oct 9, 2015 at 3:41 PM, Chanaka Fernando 
>> wrote:
>>
>>> Hi Sameera,
>>>
>>> Your judgment is correct and it helped us to find the root cause :).
>>> I'll fix the code from carbon-mediation side. Thanks for your input.
>>>
>>> On Fri, Oct 9, 2015 at 3:19 PM, Sameera Jayasoma 
>>> wrote:
>>>
 Hi Chanaka,

 You don't need to destroy the current context. This part is handle by
 the Kernel code. Users of the CarbonContext API do not need to worry about
 destroying the context.

 Who has done this change? So my initial judgment is correct. :) Stack
 cannot become empty.

 Thanks,
 Sameera.

 On Thu, Oct 8, 2015 at 6:14 PM, Chanaka Fernando 
 wrote:

> Hi All,
>
> At last, I was able to find the root cause for this behavior. This is
> actually coming from carbon-mediation code. What actually happens is that
> when ESB starts with a tenant, it will not load the tenant until it
> receives the first request. When It receives the first request, it will
> call the MultitenantMessageReceiver.processRequest() method. Within this
> method, it will call the following method.
>
> PrivilegedCarbonContext.startTenantFlow();
>
>
> This will get the ThreadLocal variable *parentContextHolderStack *and
> get the stack  and push the carbonContextDataHolder object to the
> stack. After this, tenant loading happens and within the initialization of
> the carbon mediation registry we have the following code segment.
>
> *org.wso2.carbon.mediation.registry.WSO2Registry.java*
>
> /**
>  *  Carbon Kernel mandates to set Threadlocal before calling anything in 
> kernel
>  */
> private void setTenantInfo() {
> // Preserve user name
> String username = 
> PrivilegedCarbonContext.getThreadLocalCarbonContext().getUsername();
> *PrivilegedCarbonContext.destroyCurrentContext();*
> PrivilegedCarbonContext cc = 
> PrivilegedCarbonContext.getThreadLocalCarbonContext();
> cc.setTenantDomain(domain);
> cc.setTenantId(tenantId);
> if (username != null) { // Set back the user name
> 
> PrivilegedCarbonContext.getThreadLocalCarbonContext().setUsername(username);
> }
> }
>
> Within the above method following line causes the issue.
>
> PrivilegedCarbonContext.destroyCurrentContext();
>
> When this method is called, it will reset the *parentContextHolderStack
> *and the initial object push into the stack is destroyed. Then after
> tenant loading, within the MultitenantMessageReceiver.processRequest()
> method, it tries to end the tenant flow within finally block and try to 
> pop
> the object which it pushes early. But right now, we have a new context
> stack and it will throw the emptyStackException due to that.
>
> @Sameera/KasunG: Do we really need to use the following code segment
> within the above method?
>
>
> *PrivilegedCarbonContext.destroyCurrentContext();*
>
> I saw this method called within carbon-mediation components in 3
> different locations. AFAIU, we don't need to destroy the current context
> when we are accessing the ThreadLocalContext. Please share your thoughts
> such that we can fix this issue. Issue is fixed when I comment out the
> above line :)
>
>
> Thanks,
> Chanaka
>
>
>
> On Wed, Oct 7, 2015 at 5:34 PM, Chanaka Fernando 
> wrote:
>
>> Hi Sameera/KasunG,
>>
>> I have debugged the code to find the root cause for this empty stack
>> exception. This is causing several other issues at the ESB layer. What
>> actually happens is that inside the
>> MultitenantMessageReceiver.processRequest() method, we have the following
>> code segment.
>>
>> try {
>> PrivilegedCarbonContext.startTenantFlow();
>> PrivilegedCarbonContext privilegedCarbonContext = 
>> PrivilegedCarbonContext.getThreadLocalCarbonContext();
>> privilegedCarbonContext.setTenantDomain(tenantDomain, true);
>> // this is to prevent non-blocking transports from sending 202
>> 
>> 

Re: [Dev] [ESB] EmptyStackException when resuming a paused message processor

[Jagath Sisirakumara Ariyarathne]

Hi Nuwan,

Our plan is to do the first milestone release end of next week. With this
we hope to incorporate latest carbon-deployment and kernel releases (if it
released during this week).

Thanks.

On Wed, Oct 14, 2015 at 11:07 AM, Chanaka Fernando 
wrote:

> [Adding Jagath]
>
> On Wed, Oct 14, 2015 at 10:17 AM, Nuwan Dias  wrote:
>
>> Hi ESB folks,
>>
>> Can we get a milestone release of synapse and carbon-mediation including
>> this fix and another PR I sent to synapse? We have to do a milestone
>> release of API Manager this week.
>>
>> Thanks,
>> NuwanD.
>>
>> On Mon, Oct 12, 2015 at 11:43 AM, Nuwan Dias  wrote:
>>
>>> Thanks a lot Chanaka.
>>>
>>> On Mon, Oct 12, 2015 at 11:39 AM, Chanaka Fernando 
>>> wrote:
>>>
 Hi All,

 This issue is fixed with the following PR [1]. Thanks for reporting and
 helping us to figure out the issue.

 [1] https://github.com/wso2/carbon-mediation/pull/467

 On Fri, Oct 9, 2015 at 3:41 PM, Chanaka Fernando 
 wrote:

> Hi Sameera,
>
> Your judgment is correct and it helped us to find the root cause :).
> I'll fix the code from carbon-mediation side. Thanks for your input.
>
> On Fri, Oct 9, 2015 at 3:19 PM, Sameera Jayasoma 
> wrote:
>
>> Hi Chanaka,
>>
>> You don't need to destroy the current context. This part is handle by
>> the Kernel code. Users of the CarbonContext API do not need to worry 
>> about
>> destroying the context.
>>
>> Who has done this change? So my initial judgment is correct. :) Stack
>> cannot become empty.
>>
>> Thanks,
>> Sameera.
>>
>> On Thu, Oct 8, 2015 at 6:14 PM, Chanaka Fernando 
>> wrote:
>>
>>> Hi All,
>>>
>>> At last, I was able to find the root cause for this behavior. This
>>> is actually coming from carbon-mediation code. What actually happens is
>>> that when ESB starts with a tenant, it will not load the tenant until it
>>> receives the first request. When It receives the first request, it will
>>> call the MultitenantMessageReceiver.processRequest() method. Within this
>>> method, it will call the following method.
>>>
>>> PrivilegedCarbonContext.startTenantFlow();
>>>
>>>
>>> This will get the ThreadLocal variable *parentContextHolderStack *and
>>> get the stack  and push the carbonContextDataHolder object to the
>>> stack. After this, tenant loading happens and within the initialization 
>>> of
>>> the carbon mediation registry we have the following code segment.
>>>
>>> *org.wso2.carbon.mediation.registry.WSO2Registry.java*
>>>
>>> /**
>>>  *  Carbon Kernel mandates to set Threadlocal before calling anything 
>>> in kernel
>>>  */
>>> private void setTenantInfo() {
>>> // Preserve user name
>>> String username = 
>>> PrivilegedCarbonContext.getThreadLocalCarbonContext().getUsername();
>>> *PrivilegedCarbonContext.destroyCurrentContext();*
>>> PrivilegedCarbonContext cc = 
>>> PrivilegedCarbonContext.getThreadLocalCarbonContext();
>>> cc.setTenantDomain(domain);
>>> cc.setTenantId(tenantId);
>>> if (username != null) { // Set back the user name
>>> 
>>> PrivilegedCarbonContext.getThreadLocalCarbonContext().setUsername(username);
>>> }
>>> }
>>>
>>> Within the above method following line causes the issue.
>>>
>>> PrivilegedCarbonContext.destroyCurrentContext();
>>>
>>> When this method is called, it will reset the *parentContextHolderStack
>>> *and the initial object push into the stack is destroyed. Then
>>> after tenant loading, within the
>>> MultitenantMessageReceiver.processRequest() method, it tries to end the
>>> tenant flow within finally block and try to pop the object which it 
>>> pushes
>>> early. But right now, we have a new context stack and it will throw the
>>> emptyStackException due to that.
>>>
>>> @Sameera/KasunG: Do we really need to use the following code segment
>>> within the above method?
>>>
>>>
>>> *PrivilegedCarbonContext.destroyCurrentContext();*
>>>
>>> I saw this method called within carbon-mediation components in 3
>>> different locations. AFAIU, we don't need to destroy the current context
>>> when we are accessing the ThreadLocalContext. Please share your thoughts
>>> such that we can fix this issue. Issue is fixed when I comment out the
>>> above line :)
>>>
>>>
>>> Thanks,
>>> Chanaka
>>>
>>>
>>>
>>> On Wed, Oct 7, 2015 at 5:34 PM, Chanaka Fernando 
>>> wrote:
>>>
 Hi Sameera/KasunG,

 I have debugged the code to find the root cause for this empty
 stack 

Re: [Dev] [ESB] EmptyStackException when resuming a paused message processor

[Chanaka Fernando]

[Adding Jagath]

On Wed, Oct 14, 2015 at 10:17 AM, Nuwan Dias  wrote:

> Hi ESB folks,
>
> Can we get a milestone release of synapse and carbon-mediation including
> this fix and another PR I sent to synapse? We have to do a milestone
> release of API Manager this week.
>
> Thanks,
> NuwanD.
>
> On Mon, Oct 12, 2015 at 11:43 AM, Nuwan Dias  wrote:
>
>> Thanks a lot Chanaka.
>>
>> On Mon, Oct 12, 2015 at 11:39 AM, Chanaka Fernando 
>> wrote:
>>
>>> Hi All,
>>>
>>> This issue is fixed with the following PR [1]. Thanks for reporting and
>>> helping us to figure out the issue.
>>>
>>> [1] https://github.com/wso2/carbon-mediation/pull/467
>>>
>>> On Fri, Oct 9, 2015 at 3:41 PM, Chanaka Fernando 
>>> wrote:
>>>
 Hi Sameera,

 Your judgment is correct and it helped us to find the root cause :).
 I'll fix the code from carbon-mediation side. Thanks for your input.

 On Fri, Oct 9, 2015 at 3:19 PM, Sameera Jayasoma 
 wrote:

> Hi Chanaka,
>
> You don't need to destroy the current context. This part is handle by
> the Kernel code. Users of the CarbonContext API do not need to worry about
> destroying the context.
>
> Who has done this change? So my initial judgment is correct. :) Stack
> cannot become empty.
>
> Thanks,
> Sameera.
>
> On Thu, Oct 8, 2015 at 6:14 PM, Chanaka Fernando 
> wrote:
>
>> Hi All,
>>
>> At last, I was able to find the root cause for this behavior. This is
>> actually coming from carbon-mediation code. What actually happens is that
>> when ESB starts with a tenant, it will not load the tenant until it
>> receives the first request. When It receives the first request, it will
>> call the MultitenantMessageReceiver.processRequest() method. Within this
>> method, it will call the following method.
>>
>> PrivilegedCarbonContext.startTenantFlow();
>>
>>
>> This will get the ThreadLocal variable *parentContextHolderStack *and
>> get the stack  and push the carbonContextDataHolder object to the
>> stack. After this, tenant loading happens and within the initialization 
>> of
>> the carbon mediation registry we have the following code segment.
>>
>> *org.wso2.carbon.mediation.registry.WSO2Registry.java*
>>
>> /**
>>  *  Carbon Kernel mandates to set Threadlocal before calling anything in 
>> kernel
>>  */
>> private void setTenantInfo() {
>> // Preserve user name
>> String username = 
>> PrivilegedCarbonContext.getThreadLocalCarbonContext().getUsername();
>> *PrivilegedCarbonContext.destroyCurrentContext();*
>> PrivilegedCarbonContext cc = 
>> PrivilegedCarbonContext.getThreadLocalCarbonContext();
>> cc.setTenantDomain(domain);
>> cc.setTenantId(tenantId);
>> if (username != null) { // Set back the user name
>> 
>> PrivilegedCarbonContext.getThreadLocalCarbonContext().setUsername(username);
>> }
>> }
>>
>> Within the above method following line causes the issue.
>>
>> PrivilegedCarbonContext.destroyCurrentContext();
>>
>> When this method is called, it will reset the *parentContextHolderStack
>> *and the initial object push into the stack is destroyed. Then after
>> tenant loading, within the MultitenantMessageReceiver.processRequest()
>> method, it tries to end the tenant flow within finally block and try to 
>> pop
>> the object which it pushes early. But right now, we have a new context
>> stack and it will throw the emptyStackException due to that.
>>
>> @Sameera/KasunG: Do we really need to use the following code segment
>> within the above method?
>>
>>
>> *PrivilegedCarbonContext.destroyCurrentContext();*
>>
>> I saw this method called within carbon-mediation components in 3
>> different locations. AFAIU, we don't need to destroy the current context
>> when we are accessing the ThreadLocalContext. Please share your thoughts
>> such that we can fix this issue. Issue is fixed when I comment out the
>> above line :)
>>
>>
>> Thanks,
>> Chanaka
>>
>>
>>
>> On Wed, Oct 7, 2015 at 5:34 PM, Chanaka Fernando 
>> wrote:
>>
>>> Hi Sameera/KasunG,
>>>
>>> I have debugged the code to find the root cause for this empty stack
>>> exception. This is causing several other issues at the ESB layer. What
>>> actually happens is that inside the
>>> MultitenantMessageReceiver.processRequest() method, we have the 
>>> following
>>> code segment.
>>>
>>> try {
>>> PrivilegedCarbonContext.startTenantFlow();
>>> PrivilegedCarbonContext privilegedCarbonContext = 
>>> 

Re: [Dev] [ESB] EmptyStackException when resuming a paused message processor

[Chanaka Fernando]

Hi All,

This issue is fixed with the following PR [1]. Thanks for reporting and
helping us to figure out the issue.

[1] https://github.com/wso2/carbon-mediation/pull/467

On Fri, Oct 9, 2015 at 3:41 PM, Chanaka Fernando  wrote:

> Hi Sameera,
>
> Your judgment is correct and it helped us to find the root cause :). I'll
> fix the code from carbon-mediation side. Thanks for your input.
>
> On Fri, Oct 9, 2015 at 3:19 PM, Sameera Jayasoma  wrote:
>
>> Hi Chanaka,
>>
>> You don't need to destroy the current context. This part is handle by the
>> Kernel code. Users of the CarbonContext API do not need to worry about
>> destroying the context.
>>
>> Who has done this change? So my initial judgment is correct. :) Stack
>> cannot become empty.
>>
>> Thanks,
>> Sameera.
>>
>> On Thu, Oct 8, 2015 at 6:14 PM, Chanaka Fernando 
>> wrote:
>>
>>> Hi All,
>>>
>>> At last, I was able to find the root cause for this behavior. This is
>>> actually coming from carbon-mediation code. What actually happens is that
>>> when ESB starts with a tenant, it will not load the tenant until it
>>> receives the first request. When It receives the first request, it will
>>> call the MultitenantMessageReceiver.processRequest() method. Within this
>>> method, it will call the following method.
>>>
>>> PrivilegedCarbonContext.startTenantFlow();
>>>
>>>
>>> This will get the ThreadLocal variable *parentContextHolderStack *and
>>> get the stack  and push the carbonContextDataHolder object to the
>>> stack. After this, tenant loading happens and within the initialization of
>>> the carbon mediation registry we have the following code segment.
>>>
>>> *org.wso2.carbon.mediation.registry.WSO2Registry.java*
>>>
>>> /**
>>>  *  Carbon Kernel mandates to set Threadlocal before calling anything in 
>>> kernel
>>>  */
>>> private void setTenantInfo() {
>>> // Preserve user name
>>> String username = 
>>> PrivilegedCarbonContext.getThreadLocalCarbonContext().getUsername();
>>> *PrivilegedCarbonContext.destroyCurrentContext();*
>>> PrivilegedCarbonContext cc = 
>>> PrivilegedCarbonContext.getThreadLocalCarbonContext();
>>> cc.setTenantDomain(domain);
>>> cc.setTenantId(tenantId);
>>> if (username != null) { // Set back the user name
>>> 
>>> PrivilegedCarbonContext.getThreadLocalCarbonContext().setUsername(username);
>>> }
>>> }
>>>
>>> Within the above method following line causes the issue.
>>>
>>> PrivilegedCarbonContext.destroyCurrentContext();
>>>
>>> When this method is called, it will reset the *parentContextHolderStack
>>> *and the initial object push into the stack is destroyed. Then after
>>> tenant loading, within the MultitenantMessageReceiver.processRequest()
>>> method, it tries to end the tenant flow within finally block and try to pop
>>> the object which it pushes early. But right now, we have a new context
>>> stack and it will throw the emptyStackException due to that.
>>>
>>> @Sameera/KasunG: Do we really need to use the following code segment
>>> within the above method?
>>>
>>>
>>> *PrivilegedCarbonContext.destroyCurrentContext();*
>>>
>>> I saw this method called within carbon-mediation components in 3
>>> different locations. AFAIU, we don't need to destroy the current context
>>> when we are accessing the ThreadLocalContext. Please share your thoughts
>>> such that we can fix this issue. Issue is fixed when I comment out the
>>> above line :)
>>>
>>>
>>> Thanks,
>>> Chanaka
>>>
>>>
>>>
>>> On Wed, Oct 7, 2015 at 5:34 PM, Chanaka Fernando 
>>> wrote:
>>>
 Hi Sameera/KasunG,

 I have debugged the code to find the root cause for this empty stack
 exception. This is causing several other issues at the ESB layer. What
 actually happens is that inside the
 MultitenantMessageReceiver.processRequest() method, we have the following
 code segment.

 try {
 PrivilegedCarbonContext.startTenantFlow();
 PrivilegedCarbonContext privilegedCarbonContext = 
 PrivilegedCarbonContext.getThreadLocalCarbonContext();
 privilegedCarbonContext.setTenantDomain(tenantDomain, true);
 // this is to prevent non-blocking transports from sending 202
 
 mainInMsgContext.getOperationContext().setProperty(Constants.RESPONSE_WRITTEN,
  "SKIP");

 ConfigurationContext tenantConfigCtx =

 TenantAxisUtils.getTenantConfigurationContext(tenantDomain,

   mainConfigCtx);
 if (tenantConfigCtx == null) {
 // Throw AxisFault: Tenant does not exist
 handleException(mainInMsgContext, new AxisFault("Tenant " + 
 tenantDomain +
 "  not found"));
 return;
 }

 if 

Re: [Dev] [ESB] EmptyStackException when resuming a paused message processor

[Nuwan Dias]

Thanks a lot Chanaka.

On Mon, Oct 12, 2015 at 11:39 AM, Chanaka Fernando 
wrote:

> Hi All,
>
> This issue is fixed with the following PR [1]. Thanks for reporting and
> helping us to figure out the issue.
>
> [1] https://github.com/wso2/carbon-mediation/pull/467
>
> On Fri, Oct 9, 2015 at 3:41 PM, Chanaka Fernando 
> wrote:
>
>> Hi Sameera,
>>
>> Your judgment is correct and it helped us to find the root cause :). I'll
>> fix the code from carbon-mediation side. Thanks for your input.
>>
>> On Fri, Oct 9, 2015 at 3:19 PM, Sameera Jayasoma 
>> wrote:
>>
>>> Hi Chanaka,
>>>
>>> You don't need to destroy the current context. This part is handle by
>>> the Kernel code. Users of the CarbonContext API do not need to worry about
>>> destroying the context.
>>>
>>> Who has done this change? So my initial judgment is correct. :) Stack
>>> cannot become empty.
>>>
>>> Thanks,
>>> Sameera.
>>>
>>> On Thu, Oct 8, 2015 at 6:14 PM, Chanaka Fernando 
>>> wrote:
>>>
 Hi All,

 At last, I was able to find the root cause for this behavior. This is
 actually coming from carbon-mediation code. What actually happens is that
 when ESB starts with a tenant, it will not load the tenant until it
 receives the first request. When It receives the first request, it will
 call the MultitenantMessageReceiver.processRequest() method. Within this
 method, it will call the following method.

 PrivilegedCarbonContext.startTenantFlow();


 This will get the ThreadLocal variable *parentContextHolderStack *and
 get the stack  and push the carbonContextDataHolder object to the
 stack. After this, tenant loading happens and within the initialization of
 the carbon mediation registry we have the following code segment.

 *org.wso2.carbon.mediation.registry.WSO2Registry.java*

 /**
  *  Carbon Kernel mandates to set Threadlocal before calling anything in 
 kernel
  */
 private void setTenantInfo() {
 // Preserve user name
 String username = 
 PrivilegedCarbonContext.getThreadLocalCarbonContext().getUsername();
 *PrivilegedCarbonContext.destroyCurrentContext();*
 PrivilegedCarbonContext cc = 
 PrivilegedCarbonContext.getThreadLocalCarbonContext();
 cc.setTenantDomain(domain);
 cc.setTenantId(tenantId);
 if (username != null) { // Set back the user name
 
 PrivilegedCarbonContext.getThreadLocalCarbonContext().setUsername(username);
 }
 }

 Within the above method following line causes the issue.

 PrivilegedCarbonContext.destroyCurrentContext();

 When this method is called, it will reset the *parentContextHolderStack
 *and the initial object push into the stack is destroyed. Then after
 tenant loading, within the MultitenantMessageReceiver.processRequest()
 method, it tries to end the tenant flow within finally block and try to pop
 the object which it pushes early. But right now, we have a new context
 stack and it will throw the emptyStackException due to that.

 @Sameera/KasunG: Do we really need to use the following code segment
 within the above method?


 *PrivilegedCarbonContext.destroyCurrentContext();*

 I saw this method called within carbon-mediation components in 3
 different locations. AFAIU, we don't need to destroy the current context
 when we are accessing the ThreadLocalContext. Please share your thoughts
 such that we can fix this issue. Issue is fixed when I comment out the
 above line :)


 Thanks,
 Chanaka



 On Wed, Oct 7, 2015 at 5:34 PM, Chanaka Fernando 
 wrote:

> Hi Sameera/KasunG,
>
> I have debugged the code to find the root cause for this empty stack
> exception. This is causing several other issues at the ESB layer. What
> actually happens is that inside the
> MultitenantMessageReceiver.processRequest() method, we have the following
> code segment.
>
> try {
> PrivilegedCarbonContext.startTenantFlow();
> PrivilegedCarbonContext privilegedCarbonContext = 
> PrivilegedCarbonContext.getThreadLocalCarbonContext();
> privilegedCarbonContext.setTenantDomain(tenantDomain, true);
> // this is to prevent non-blocking transports from sending 202
> 
> mainInMsgContext.getOperationContext().setProperty(Constants.RESPONSE_WRITTEN,
>  "SKIP");
>
> ConfigurationContext tenantConfigCtx =
>
> TenantAxisUtils.getTenantConfigurationContext(tenantDomain,
>   
>mainConfigCtx);
> if (tenantConfigCtx == null) {
> // Throw AxisFault: Tenant does not exist

Re: [Dev] [ESB] EmptyStackException when resuming a paused message processor

[Chanaka Fernando]

Any update on this please?

On Thu, Oct 8, 2015 at 10:50 PM, Chanaka Fernando  wrote:

> Hi NuwanD,
>
> The fix is not in kernel but in carbon-mediation. What I need is kernel
> team's opinion on usage of the aforementioned method. If we can get their
> input, fix is straightforward.
>
> On Thu, Oct 8, 2015 at 9:58 PM, Nuwan Dias  wrote:
>
>> Hi all,
>>
>> Will this be fixed in kernel 4.4.2? Its causing integration test failures
>> in API Manager and we would like this to be fixed soon.
>>
>> Thanks,
>> NuwanD.
>>
>> On Thu, Oct 8, 2015 at 6:22 PM, Chanaka Fernando 
>> wrote:
>>
>>> [Adding Kishanthan]
>>>
>>> On Thu, Oct 8, 2015 at 6:14 PM, Chanaka Fernando 
>>> wrote:
>>>
 Hi All,

 At last, I was able to find the root cause for this behavior. This is
 actually coming from carbon-mediation code. What actually happens is that
 when ESB starts with a tenant, it will not load the tenant until it
 receives the first request. When It receives the first request, it will
 call the MultitenantMessageReceiver.processRequest() method. Within this
 method, it will call the following method.

 PrivilegedCarbonContext.startTenantFlow();


 This will get the ThreadLocal variable *parentContextHolderStack *and
 get the stack  and push the carbonContextDataHolder object to the
 stack. After this, tenant loading happens and within the initialization of
 the carbon mediation registry we have the following code segment.

 *org.wso2.carbon.mediation.registry.WSO2Registry.java*

 /**
  *  Carbon Kernel mandates to set Threadlocal before calling anything in 
 kernel
  */
 private void setTenantInfo() {
 // Preserve user name
 String username = 
 PrivilegedCarbonContext.getThreadLocalCarbonContext().getUsername();
 *PrivilegedCarbonContext.destroyCurrentContext();*
 PrivilegedCarbonContext cc = 
 PrivilegedCarbonContext.getThreadLocalCarbonContext();
 cc.setTenantDomain(domain);
 cc.setTenantId(tenantId);
 if (username != null) { // Set back the user name
 
 PrivilegedCarbonContext.getThreadLocalCarbonContext().setUsername(username);
 }
 }

 Within the above method following line causes the issue.

 PrivilegedCarbonContext.destroyCurrentContext();

 When this method is called, it will reset the *parentContextHolderStack
 *and the initial object push into the stack is destroyed. Then after
 tenant loading, within the MultitenantMessageReceiver.processRequest()
 method, it tries to end the tenant flow within finally block and try to pop
 the object which it pushes early. But right now, we have a new context
 stack and it will throw the emptyStackException due to that.

 @Sameera/KasunG: Do we really need to use the following code segment
 within the above method?


 *PrivilegedCarbonContext.destroyCurrentContext();*

 I saw this method called within carbon-mediation components in 3
 different locations. AFAIU, we don't need to destroy the current context
 when we are accessing the ThreadLocalContext. Please share your thoughts
 such that we can fix this issue. Issue is fixed when I comment out the
 above line :)


 Thanks,
 Chanaka



 On Wed, Oct 7, 2015 at 5:34 PM, Chanaka Fernando 
 wrote:

> Hi Sameera/KasunG,
>
> I have debugged the code to find the root cause for this empty stack
> exception. This is causing several other issues at the ESB layer. What
> actually happens is that inside the
> MultitenantMessageReceiver.processRequest() method, we have the following
> code segment.
>
> try {
> PrivilegedCarbonContext.startTenantFlow();
> PrivilegedCarbonContext privilegedCarbonContext = 
> PrivilegedCarbonContext.getThreadLocalCarbonContext();
> privilegedCarbonContext.setTenantDomain(tenantDomain, true);
> // this is to prevent non-blocking transports from sending 202
> 
> mainInMsgContext.getOperationContext().setProperty(Constants.RESPONSE_WRITTEN,
>  "SKIP");
>
> ConfigurationContext tenantConfigCtx =
>
> TenantAxisUtils.getTenantConfigurationContext(tenantDomain,
>   
>mainConfigCtx);
> if (tenantConfigCtx == null) {
> // Throw AxisFault: Tenant does not exist
> handleException(mainInMsgContext, new AxisFault("Tenant " + 
> tenantDomain +
> "  not found"));
> return;
> }
>
> if (mainInMsgContext.isDoingREST()) { // Handle REST requests

Re: [Dev] [ESB] EmptyStackException when resuming a paused message processor

[Chanaka Fernando]

Hi Sameera,

Your judgment is correct and it helped us to find the root cause :). I'll
fix the code from carbon-mediation side. Thanks for your input.

On Fri, Oct 9, 2015 at 3:19 PM, Sameera Jayasoma  wrote:

> Hi Chanaka,
>
> You don't need to destroy the current context. This part is handle by the
> Kernel code. Users of the CarbonContext API do not need to worry about
> destroying the context.
>
> Who has done this change? So my initial judgment is correct. :) Stack
> cannot become empty.
>
> Thanks,
> Sameera.
>
> On Thu, Oct 8, 2015 at 6:14 PM, Chanaka Fernando 
> wrote:
>
>> Hi All,
>>
>> At last, I was able to find the root cause for this behavior. This is
>> actually coming from carbon-mediation code. What actually happens is that
>> when ESB starts with a tenant, it will not load the tenant until it
>> receives the first request. When It receives the first request, it will
>> call the MultitenantMessageReceiver.processRequest() method. Within this
>> method, it will call the following method.
>>
>> PrivilegedCarbonContext.startTenantFlow();
>>
>>
>> This will get the ThreadLocal variable *parentContextHolderStack *and
>> get the stack  and push the carbonContextDataHolder object to the stack.
>> After this, tenant loading happens and within the initialization of the
>> carbon mediation registry we have the following code segment.
>>
>> *org.wso2.carbon.mediation.registry.WSO2Registry.java*
>>
>> /**
>>  *  Carbon Kernel mandates to set Threadlocal before calling anything in 
>> kernel
>>  */
>> private void setTenantInfo() {
>> // Preserve user name
>> String username = 
>> PrivilegedCarbonContext.getThreadLocalCarbonContext().getUsername();
>> *PrivilegedCarbonContext.destroyCurrentContext();*
>> PrivilegedCarbonContext cc = 
>> PrivilegedCarbonContext.getThreadLocalCarbonContext();
>> cc.setTenantDomain(domain);
>> cc.setTenantId(tenantId);
>> if (username != null) { // Set back the user name
>> 
>> PrivilegedCarbonContext.getThreadLocalCarbonContext().setUsername(username);
>> }
>> }
>>
>> Within the above method following line causes the issue.
>>
>> PrivilegedCarbonContext.destroyCurrentContext();
>>
>> When this method is called, it will reset the *parentContextHolderStack *and
>> the initial object push into the stack is destroyed. Then after tenant
>> loading, within the MultitenantMessageReceiver.processRequest() method, it
>> tries to end the tenant flow within finally block and try to pop the object
>> which it pushes early. But right now, we have a new context stack and it
>> will throw the emptyStackException due to that.
>>
>> @Sameera/KasunG: Do we really need to use the following code segment
>> within the above method?
>>
>>
>> *PrivilegedCarbonContext.destroyCurrentContext();*
>>
>> I saw this method called within carbon-mediation components in 3
>> different locations. AFAIU, we don't need to destroy the current context
>> when we are accessing the ThreadLocalContext. Please share your thoughts
>> such that we can fix this issue. Issue is fixed when I comment out the
>> above line :)
>>
>>
>> Thanks,
>> Chanaka
>>
>>
>>
>> On Wed, Oct 7, 2015 at 5:34 PM, Chanaka Fernando 
>> wrote:
>>
>>> Hi Sameera/KasunG,
>>>
>>> I have debugged the code to find the root cause for this empty stack
>>> exception. This is causing several other issues at the ESB layer. What
>>> actually happens is that inside the
>>> MultitenantMessageReceiver.processRequest() method, we have the following
>>> code segment.
>>>
>>> try {
>>> PrivilegedCarbonContext.startTenantFlow();
>>> PrivilegedCarbonContext privilegedCarbonContext = 
>>> PrivilegedCarbonContext.getThreadLocalCarbonContext();
>>> privilegedCarbonContext.setTenantDomain(tenantDomain, true);
>>> // this is to prevent non-blocking transports from sending 202
>>> 
>>> mainInMsgContext.getOperationContext().setProperty(Constants.RESPONSE_WRITTEN,
>>>  "SKIP");
>>>
>>> ConfigurationContext tenantConfigCtx =
>>>
>>> TenantAxisUtils.getTenantConfigurationContext(tenantDomain,
>>> 
>>>  mainConfigCtx);
>>> if (tenantConfigCtx == null) {
>>> // Throw AxisFault: Tenant does not exist
>>> handleException(mainInMsgContext, new AxisFault("Tenant " + 
>>> tenantDomain +
>>> "  not found"));
>>> return;
>>> }
>>>
>>> if (mainInMsgContext.isDoingREST()) { // Handle REST requests
>>> doREST(mainInMsgContext, to, tenantDomain, tenantConfigCtx, 
>>> serviceAndOperation);
>>> } else {
>>> doSOAP(mainInMsgContext, tenantDomain, tenantConfigCtx, 
>>> serviceAndOperation);
>>> }
>>> } finally {
>>> PrivilegedCarbonContext.endTenantFlow();
>>> }
>>>
>>>
>>> When we are calling the 

Re: [Dev] [ESB] EmptyStackException when resuming a paused message processor

[Sameera Jayasoma]

Hi Chanaka,

You don't need to destroy the current context. This part is handle by the
Kernel code. Users of the CarbonContext API do not need to worry about
destroying the context.

Who has done this change? So my initial judgment is correct. :) Stack
cannot become empty.

Thanks,
Sameera.

On Thu, Oct 8, 2015 at 6:14 PM, Chanaka Fernando  wrote:

> Hi All,
>
> At last, I was able to find the root cause for this behavior. This is
> actually coming from carbon-mediation code. What actually happens is that
> when ESB starts with a tenant, it will not load the tenant until it
> receives the first request. When It receives the first request, it will
> call the MultitenantMessageReceiver.processRequest() method. Within this
> method, it will call the following method.
>
> PrivilegedCarbonContext.startTenantFlow();
>
>
> This will get the ThreadLocal variable *parentContextHolderStack *and get
> the stack  and push the carbonContextDataHolder object to the stack.
> After this, tenant loading happens and within the initialization of the
> carbon mediation registry we have the following code segment.
>
> *org.wso2.carbon.mediation.registry.WSO2Registry.java*
>
> /**
>  *  Carbon Kernel mandates to set Threadlocal before calling anything in 
> kernel
>  */
> private void setTenantInfo() {
> // Preserve user name
> String username = 
> PrivilegedCarbonContext.getThreadLocalCarbonContext().getUsername();
> *PrivilegedCarbonContext.destroyCurrentContext();*
> PrivilegedCarbonContext cc = 
> PrivilegedCarbonContext.getThreadLocalCarbonContext();
> cc.setTenantDomain(domain);
> cc.setTenantId(tenantId);
> if (username != null) { // Set back the user name
> 
> PrivilegedCarbonContext.getThreadLocalCarbonContext().setUsername(username);
> }
> }
>
> Within the above method following line causes the issue.
>
> PrivilegedCarbonContext.destroyCurrentContext();
>
> When this method is called, it will reset the *parentContextHolderStack *and
> the initial object push into the stack is destroyed. Then after tenant
> loading, within the MultitenantMessageReceiver.processRequest() method, it
> tries to end the tenant flow within finally block and try to pop the object
> which it pushes early. But right now, we have a new context stack and it
> will throw the emptyStackException due to that.
>
> @Sameera/KasunG: Do we really need to use the following code segment
> within the above method?
>
>
> *PrivilegedCarbonContext.destroyCurrentContext();*
>
> I saw this method called within carbon-mediation components in 3 different
> locations. AFAIU, we don't need to destroy the current context when we are
> accessing the ThreadLocalContext. Please share your thoughts such that we
> can fix this issue. Issue is fixed when I comment out the above line :)
>
>
> Thanks,
> Chanaka
>
>
>
> On Wed, Oct 7, 2015 at 5:34 PM, Chanaka Fernando 
> wrote:
>
>> Hi Sameera/KasunG,
>>
>> I have debugged the code to find the root cause for this empty stack
>> exception. This is causing several other issues at the ESB layer. What
>> actually happens is that inside the
>> MultitenantMessageReceiver.processRequest() method, we have the following
>> code segment.
>>
>> try {
>> PrivilegedCarbonContext.startTenantFlow();
>> PrivilegedCarbonContext privilegedCarbonContext = 
>> PrivilegedCarbonContext.getThreadLocalCarbonContext();
>> privilegedCarbonContext.setTenantDomain(tenantDomain, true);
>> // this is to prevent non-blocking transports from sending 202
>> 
>> mainInMsgContext.getOperationContext().setProperty(Constants.RESPONSE_WRITTEN,
>>  "SKIP");
>>
>> ConfigurationContext tenantConfigCtx =
>>
>> TenantAxisUtils.getTenantConfigurationContext(tenantDomain,
>>  
>> mainConfigCtx);
>> if (tenantConfigCtx == null) {
>> // Throw AxisFault: Tenant does not exist
>> handleException(mainInMsgContext, new AxisFault("Tenant " + 
>> tenantDomain +
>> "  not found"));
>> return;
>> }
>>
>> if (mainInMsgContext.isDoingREST()) { // Handle REST requests
>> doREST(mainInMsgContext, to, tenantDomain, tenantConfigCtx, 
>> serviceAndOperation);
>> } else {
>> doSOAP(mainInMsgContext, tenantDomain, tenantConfigCtx, 
>> serviceAndOperation);
>> }
>> } finally {
>> PrivilegedCarbonContext.endTenantFlow();
>> }
>>
>>
>> When we are calling the endTenantFlow() method, it will go inside the
>> following method within the CarbonContextDataHolder class.
>>
>> /**
>>  * This will end the tenant flow and restore the previous CarbonContext.
>>  */
>> public void endTenantFlow() {
>> Stack carbonContextDataHolders = 
>> parentContextHolderStack.get();
>> if (carbonContextDataHolders != null) {
>> 

Re: [Dev] [ESB] EmptyStackException when resuming a paused message processor

[Chanaka Fernando]

[Adding Kishanthan]

On Thu, Oct 8, 2015 at 6:14 PM, Chanaka Fernando  wrote:

> Hi All,
>
> At last, I was able to find the root cause for this behavior. This is
> actually coming from carbon-mediation code. What actually happens is that
> when ESB starts with a tenant, it will not load the tenant until it
> receives the first request. When It receives the first request, it will
> call the MultitenantMessageReceiver.processRequest() method. Within this
> method, it will call the following method.
>
> PrivilegedCarbonContext.startTenantFlow();
>
>
> This will get the ThreadLocal variable *parentContextHolderStack *and get
> the stack  and push the carbonContextDataHolder object to the stack.
> After this, tenant loading happens and within the initialization of the
> carbon mediation registry we have the following code segment.
>
> *org.wso2.carbon.mediation.registry.WSO2Registry.java*
>
> /**
>  *  Carbon Kernel mandates to set Threadlocal before calling anything in 
> kernel
>  */
> private void setTenantInfo() {
> // Preserve user name
> String username = 
> PrivilegedCarbonContext.getThreadLocalCarbonContext().getUsername();
> *PrivilegedCarbonContext.destroyCurrentContext();*
> PrivilegedCarbonContext cc = 
> PrivilegedCarbonContext.getThreadLocalCarbonContext();
> cc.setTenantDomain(domain);
> cc.setTenantId(tenantId);
> if (username != null) { // Set back the user name
> 
> PrivilegedCarbonContext.getThreadLocalCarbonContext().setUsername(username);
> }
> }
>
> Within the above method following line causes the issue.
>
> PrivilegedCarbonContext.destroyCurrentContext();
>
> When this method is called, it will reset the *parentContextHolderStack *and
> the initial object push into the stack is destroyed. Then after tenant
> loading, within the MultitenantMessageReceiver.processRequest() method, it
> tries to end the tenant flow within finally block and try to pop the object
> which it pushes early. But right now, we have a new context stack and it
> will throw the emptyStackException due to that.
>
> @Sameera/KasunG: Do we really need to use the following code segment
> within the above method?
>
>
> *PrivilegedCarbonContext.destroyCurrentContext();*
>
> I saw this method called within carbon-mediation components in 3 different
> locations. AFAIU, we don't need to destroy the current context when we are
> accessing the ThreadLocalContext. Please share your thoughts such that we
> can fix this issue. Issue is fixed when I comment out the above line :)
>
>
> Thanks,
> Chanaka
>
>
>
> On Wed, Oct 7, 2015 at 5:34 PM, Chanaka Fernando 
> wrote:
>
>> Hi Sameera/KasunG,
>>
>> I have debugged the code to find the root cause for this empty stack
>> exception. This is causing several other issues at the ESB layer. What
>> actually happens is that inside the
>> MultitenantMessageReceiver.processRequest() method, we have the following
>> code segment.
>>
>> try {
>> PrivilegedCarbonContext.startTenantFlow();
>> PrivilegedCarbonContext privilegedCarbonContext = 
>> PrivilegedCarbonContext.getThreadLocalCarbonContext();
>> privilegedCarbonContext.setTenantDomain(tenantDomain, true);
>> // this is to prevent non-blocking transports from sending 202
>> 
>> mainInMsgContext.getOperationContext().setProperty(Constants.RESPONSE_WRITTEN,
>>  "SKIP");
>>
>> ConfigurationContext tenantConfigCtx =
>>
>> TenantAxisUtils.getTenantConfigurationContext(tenantDomain,
>>  
>> mainConfigCtx);
>> if (tenantConfigCtx == null) {
>> // Throw AxisFault: Tenant does not exist
>> handleException(mainInMsgContext, new AxisFault("Tenant " + 
>> tenantDomain +
>> "  not found"));
>> return;
>> }
>>
>> if (mainInMsgContext.isDoingREST()) { // Handle REST requests
>> doREST(mainInMsgContext, to, tenantDomain, tenantConfigCtx, 
>> serviceAndOperation);
>> } else {
>> doSOAP(mainInMsgContext, tenantDomain, tenantConfigCtx, 
>> serviceAndOperation);
>> }
>> } finally {
>> PrivilegedCarbonContext.endTenantFlow();
>> }
>>
>>
>> When we are calling the endTenantFlow() method, it will go inside the
>> following method within the CarbonContextDataHolder class.
>>
>> /**
>>  * This will end the tenant flow and restore the previous CarbonContext.
>>  */
>> public void endTenantFlow() {
>> Stack carbonContextDataHolders = 
>> parentContextHolderStack.get();
>> if (carbonContextDataHolders != null) {
>> currentContextHolder.set(carbonContextDataHolders.pop());
>> }
>> }
>>
>>
>> At the time of calling this method carbonContextDataHolders stack has
>> become empty and causing the EmptyStackException. When I debug the code, I
>> found that there is a scheduled task running in a 

Re: [Dev] [ESB] EmptyStackException when resuming a paused message processor

[Nuwan Dias]

Hi all,

Will this be fixed in kernel 4.4.2? Its causing integration test failures
in API Manager and we would like this to be fixed soon.

Thanks,
NuwanD.

On Thu, Oct 8, 2015 at 6:22 PM, Chanaka Fernando  wrote:

> [Adding Kishanthan]
>
> On Thu, Oct 8, 2015 at 6:14 PM, Chanaka Fernando 
> wrote:
>
>> Hi All,
>>
>> At last, I was able to find the root cause for this behavior. This is
>> actually coming from carbon-mediation code. What actually happens is that
>> when ESB starts with a tenant, it will not load the tenant until it
>> receives the first request. When It receives the first request, it will
>> call the MultitenantMessageReceiver.processRequest() method. Within this
>> method, it will call the following method.
>>
>> PrivilegedCarbonContext.startTenantFlow();
>>
>>
>> This will get the ThreadLocal variable *parentContextHolderStack *and
>> get the stack  and push the carbonContextDataHolder object to the stack.
>> After this, tenant loading happens and within the initialization of the
>> carbon mediation registry we have the following code segment.
>>
>> *org.wso2.carbon.mediation.registry.WSO2Registry.java*
>>
>> /**
>>  *  Carbon Kernel mandates to set Threadlocal before calling anything in 
>> kernel
>>  */
>> private void setTenantInfo() {
>> // Preserve user name
>> String username = 
>> PrivilegedCarbonContext.getThreadLocalCarbonContext().getUsername();
>> *PrivilegedCarbonContext.destroyCurrentContext();*
>> PrivilegedCarbonContext cc = 
>> PrivilegedCarbonContext.getThreadLocalCarbonContext();
>> cc.setTenantDomain(domain);
>> cc.setTenantId(tenantId);
>> if (username != null) { // Set back the user name
>> 
>> PrivilegedCarbonContext.getThreadLocalCarbonContext().setUsername(username);
>> }
>> }
>>
>> Within the above method following line causes the issue.
>>
>> PrivilegedCarbonContext.destroyCurrentContext();
>>
>> When this method is called, it will reset the *parentContextHolderStack *and
>> the initial object push into the stack is destroyed. Then after tenant
>> loading, within the MultitenantMessageReceiver.processRequest() method, it
>> tries to end the tenant flow within finally block and try to pop the object
>> which it pushes early. But right now, we have a new context stack and it
>> will throw the emptyStackException due to that.
>>
>> @Sameera/KasunG: Do we really need to use the following code segment
>> within the above method?
>>
>>
>> *PrivilegedCarbonContext.destroyCurrentContext();*
>>
>> I saw this method called within carbon-mediation components in 3
>> different locations. AFAIU, we don't need to destroy the current context
>> when we are accessing the ThreadLocalContext. Please share your thoughts
>> such that we can fix this issue. Issue is fixed when I comment out the
>> above line :)
>>
>>
>> Thanks,
>> Chanaka
>>
>>
>>
>> On Wed, Oct 7, 2015 at 5:34 PM, Chanaka Fernando 
>> wrote:
>>
>>> Hi Sameera/KasunG,
>>>
>>> I have debugged the code to find the root cause for this empty stack
>>> exception. This is causing several other issues at the ESB layer. What
>>> actually happens is that inside the
>>> MultitenantMessageReceiver.processRequest() method, we have the following
>>> code segment.
>>>
>>> try {
>>> PrivilegedCarbonContext.startTenantFlow();
>>> PrivilegedCarbonContext privilegedCarbonContext = 
>>> PrivilegedCarbonContext.getThreadLocalCarbonContext();
>>> privilegedCarbonContext.setTenantDomain(tenantDomain, true);
>>> // this is to prevent non-blocking transports from sending 202
>>> 
>>> mainInMsgContext.getOperationContext().setProperty(Constants.RESPONSE_WRITTEN,
>>>  "SKIP");
>>>
>>> ConfigurationContext tenantConfigCtx =
>>>
>>> TenantAxisUtils.getTenantConfigurationContext(tenantDomain,
>>> 
>>>  mainConfigCtx);
>>> if (tenantConfigCtx == null) {
>>> // Throw AxisFault: Tenant does not exist
>>> handleException(mainInMsgContext, new AxisFault("Tenant " + 
>>> tenantDomain +
>>> "  not found"));
>>> return;
>>> }
>>>
>>> if (mainInMsgContext.isDoingREST()) { // Handle REST requests
>>> doREST(mainInMsgContext, to, tenantDomain, tenantConfigCtx, 
>>> serviceAndOperation);
>>> } else {
>>> doSOAP(mainInMsgContext, tenantDomain, tenantConfigCtx, 
>>> serviceAndOperation);
>>> }
>>> } finally {
>>> PrivilegedCarbonContext.endTenantFlow();
>>> }
>>>
>>>
>>> When we are calling the endTenantFlow() method, it will go inside the
>>> following method within the CarbonContextDataHolder class.
>>>
>>> /**
>>>  * This will end the tenant flow and restore the previous CarbonContext.
>>>  */
>>> public void endTenantFlow() {
>>> Stack carbonContextDataHolders = 
>>> 

Re: [Dev] [ESB] EmptyStackException when resuming a paused message processor

[Chanaka Fernando]

Hi NuwanD,

The fix is not in kernel but in carbon-mediation. What I need is kernel
team's opinion on usage of the aforementioned method. If we can get their
input, fix is straightforward.

On Thu, Oct 8, 2015 at 9:58 PM, Nuwan Dias  wrote:

> Hi all,
>
> Will this be fixed in kernel 4.4.2? Its causing integration test failures
> in API Manager and we would like this to be fixed soon.
>
> Thanks,
> NuwanD.
>
> On Thu, Oct 8, 2015 at 6:22 PM, Chanaka Fernando 
> wrote:
>
>> [Adding Kishanthan]
>>
>> On Thu, Oct 8, 2015 at 6:14 PM, Chanaka Fernando 
>> wrote:
>>
>>> Hi All,
>>>
>>> At last, I was able to find the root cause for this behavior. This is
>>> actually coming from carbon-mediation code. What actually happens is that
>>> when ESB starts with a tenant, it will not load the tenant until it
>>> receives the first request. When It receives the first request, it will
>>> call the MultitenantMessageReceiver.processRequest() method. Within this
>>> method, it will call the following method.
>>>
>>> PrivilegedCarbonContext.startTenantFlow();
>>>
>>>
>>> This will get the ThreadLocal variable *parentContextHolderStack *and
>>> get the stack  and push the carbonContextDataHolder object to the
>>> stack. After this, tenant loading happens and within the initialization of
>>> the carbon mediation registry we have the following code segment.
>>>
>>> *org.wso2.carbon.mediation.registry.WSO2Registry.java*
>>>
>>> /**
>>>  *  Carbon Kernel mandates to set Threadlocal before calling anything in 
>>> kernel
>>>  */
>>> private void setTenantInfo() {
>>> // Preserve user name
>>> String username = 
>>> PrivilegedCarbonContext.getThreadLocalCarbonContext().getUsername();
>>> *PrivilegedCarbonContext.destroyCurrentContext();*
>>> PrivilegedCarbonContext cc = 
>>> PrivilegedCarbonContext.getThreadLocalCarbonContext();
>>> cc.setTenantDomain(domain);
>>> cc.setTenantId(tenantId);
>>> if (username != null) { // Set back the user name
>>> 
>>> PrivilegedCarbonContext.getThreadLocalCarbonContext().setUsername(username);
>>> }
>>> }
>>>
>>> Within the above method following line causes the issue.
>>>
>>> PrivilegedCarbonContext.destroyCurrentContext();
>>>
>>> When this method is called, it will reset the *parentContextHolderStack
>>> *and the initial object push into the stack is destroyed. Then after
>>> tenant loading, within the MultitenantMessageReceiver.processRequest()
>>> method, it tries to end the tenant flow within finally block and try to pop
>>> the object which it pushes early. But right now, we have a new context
>>> stack and it will throw the emptyStackException due to that.
>>>
>>> @Sameera/KasunG: Do we really need to use the following code segment
>>> within the above method?
>>>
>>>
>>> *PrivilegedCarbonContext.destroyCurrentContext();*
>>>
>>> I saw this method called within carbon-mediation components in 3
>>> different locations. AFAIU, we don't need to destroy the current context
>>> when we are accessing the ThreadLocalContext. Please share your thoughts
>>> such that we can fix this issue. Issue is fixed when I comment out the
>>> above line :)
>>>
>>>
>>> Thanks,
>>> Chanaka
>>>
>>>
>>>
>>> On Wed, Oct 7, 2015 at 5:34 PM, Chanaka Fernando 
>>> wrote:
>>>
 Hi Sameera/KasunG,

 I have debugged the code to find the root cause for this empty stack
 exception. This is causing several other issues at the ESB layer. What
 actually happens is that inside the
 MultitenantMessageReceiver.processRequest() method, we have the following
 code segment.

 try {
 PrivilegedCarbonContext.startTenantFlow();
 PrivilegedCarbonContext privilegedCarbonContext = 
 PrivilegedCarbonContext.getThreadLocalCarbonContext();
 privilegedCarbonContext.setTenantDomain(tenantDomain, true);
 // this is to prevent non-blocking transports from sending 202
 
 mainInMsgContext.getOperationContext().setProperty(Constants.RESPONSE_WRITTEN,
  "SKIP");

 ConfigurationContext tenantConfigCtx =

 TenantAxisUtils.getTenantConfigurationContext(tenantDomain,

   mainConfigCtx);
 if (tenantConfigCtx == null) {
 // Throw AxisFault: Tenant does not exist
 handleException(mainInMsgContext, new AxisFault("Tenant " + 
 tenantDomain +
 "  not found"));
 return;
 }

 if (mainInMsgContext.isDoingREST()) { // Handle REST requests
 doREST(mainInMsgContext, to, tenantDomain, tenantConfigCtx, 
 serviceAndOperation);
 } else {
 doSOAP(mainInMsgContext, tenantDomain, tenantConfigCtx, 
 serviceAndOperation);
 }
 } 

Re: [Dev] [ESB] EmptyStackException when resuming a paused message processor

[Chanaka Fernando]

Hi Sameera/KasunG,

I have debugged the code to find the root cause for this empty stack
exception. This is causing several other issues at the ESB layer. What
actually happens is that inside the
MultitenantMessageReceiver.processRequest() method, we have the following
code segment.

try {
PrivilegedCarbonContext.startTenantFlow();
PrivilegedCarbonContext privilegedCarbonContext =
PrivilegedCarbonContext.getThreadLocalCarbonContext();
privilegedCarbonContext.setTenantDomain(tenantDomain, true);
// this is to prevent non-blocking transports from sending 202

mainInMsgContext.getOperationContext().setProperty(Constants.RESPONSE_WRITTEN,
"SKIP");

ConfigurationContext tenantConfigCtx =

TenantAxisUtils.getTenantConfigurationContext(tenantDomain,

  mainConfigCtx);
if (tenantConfigCtx == null) {
// Throw AxisFault: Tenant does not exist
handleException(mainInMsgContext, new AxisFault("Tenant " +
tenantDomain +
"  not found"));
return;
}

if (mainInMsgContext.isDoingREST()) { // Handle REST requests
doREST(mainInMsgContext, to, tenantDomain, tenantConfigCtx,
serviceAndOperation);
} else {
doSOAP(mainInMsgContext, tenantDomain, tenantConfigCtx,
serviceAndOperation);
}
} finally {
PrivilegedCarbonContext.endTenantFlow();
}


When we are calling the endTenantFlow() method, it will go inside the
following method within the CarbonContextDataHolder class.

/**
 * This will end the tenant flow and restore the previous CarbonContext.
 */
public void endTenantFlow() {
Stack carbonContextDataHolders =
parentContextHolderStack.get();
if (carbonContextDataHolders != null) {
currentContextHolder.set(carbonContextDataHolders.pop());
}
}


At the time of calling this method carbonContextDataHolders stack has
become empty and causing the EmptyStackException. When I debug the code, I
found that there is a scheduled task running in a different thread and
accessing the same method frequently. This is coming from
AbstractQuartzTaskManager class method.

public void triggerComplete(Trigger trigger, JobExecutionContext
jobExecutionContext,
Trigger.CompletedExecutionInstruction completedExecutionInstruction) {
PrivilegedCarbonContext.startTenantFlow();

PrivilegedCarbonContext.getThreadLocalCarbonContext().setTenantId(getTenantId(),
true);
if(trigger.getNextFireTime() == null) {
try {
TaskUtils.setTaskFinished(getTaskRepository(),
trigger.getJobKey().getName(), true);
} catch (TaskException e) {
log.error("Error in Finishing Task [" +
trigger.getJobKey().getName() +
"]: " + e.getMessage(), e);
}
}
PrivilegedCarbonContext.endTenantFlow();
}


What I could not figure out is the thread which is emptying the stack. Is
it possible that a different thread can access this
carbonContextDataHolders stack and popping out the element before the
PassThroughMessageProcessor thread access the same?



On Wed, Jul 8, 2015 at 11:12 AM, Jagath Sisirakumara Ariyarathne <
jaga...@wso2.com> wrote:

> Hi Sameera,
>
> I will check it and update.
>
> Thanks.
>
> On Wed, Jul 8, 2015 at 11:10 AM, Sameera Jayasoma 
> wrote:
>
>> Hi Jagath,
>>
>> Can you debug and see whey the stack becomes empty? Thats a serious
>> problem. Stack should be become empty here.
>>
>> Checking whether the stack is empty will stop the error log, but it
>> doesn't fix the actual problem here.
>>
>> On Wed, Jul 8, 2015 at 10:50 AM, Sameera Jayasoma 
>> wrote:
>>
>>> I understand, but we need to understand why that stack becomes empty.
>>> AFAIK, if we follow the proper APIs, stack should not become empty
>>>
>>> On Tue, Jul 7, 2015 at 5:47 PM, Kasun Indrasiri  wrote:
>>>
 Yeah, we should always check for an empty stack.

 On Tue, Jul 7, 2015 at 5:17 PM, Malaka Silva  wrote:

> I think we need to check isEmpty as well.
>
> On Tue, Jul 7, 2015 at 3:41 PM, Jagath Sisirakumara Ariyarathne <
> jaga...@wso2.com> wrote:
>
>> Hi,
>>
>> I am working on [1] and found that the cause of the exception
>> mentioned below is in the code segment in org
>> .wso2.carbon.context.internal.CarbonContextDataHolder in
>> carbon.utils.
>>
>> public void endTenantFlow() {
>>
>> Stack carbonContextDataHolders = 
>> parentContextHolderStack.get();
>> if (carbonContextDataHolders != null) {
>> currentContextHolder.set(carbonContextDataHolders.pop());
>> }
>> }
>>
>> *Exception :*
>>
>> java.util.EmptyStackException
>>  at java.util.Stack.peek(Stack.java:102)
>>  at java.util.Stack.pop(Stack.java:84)
>>  at 
>> org.wso2.carbon.context.internal.CarbonContextDataHolder.endTenantFlow(CarbonContextDataHolder.java:1291)
>>  at 

Re: [Dev] [ESB] EmptyStackException when resuming a paused message processor

[Malaka Silva]

I think we need to check isEmpty as well.

On Tue, Jul 7, 2015 at 3:41 PM, Jagath Sisirakumara Ariyarathne 
jaga...@wso2.com wrote:

 Hi,

 I am working on [1] and found that the cause of the exception mentioned
 below is in the code segment in 
 org.wso2.carbon.context.internal.CarbonContextDataHolder
 in carbon.utils.

 public void endTenantFlow() {

 StackCarbonContextDataHolder carbonContextDataHolders = 
 parentContextHolderStack.get();
 if (carbonContextDataHolders != null) {
 currentContextHolder.set(carbonContextDataHolders.pop());
 }
 }

 *Exception :*

 java.util.EmptyStackException
   at java.util.Stack.peek(Stack.java:102)
   at java.util.Stack.pop(Stack.java:84)
   at 
 org.wso2.carbon.context.internal.CarbonContextDataHolder.endTenantFlow(CarbonContextDataHolder.java:1291)
   at 
 org.wso2.carbon.context.PrivilegedCarbonContext.endTenantFlow(PrivilegedCarbonContext.java:75)
   at 
 org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute(TaskQuartzJobAdapter.java:69)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
   at 
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:745)

 Issue occurs when it tries to pop elements from carbonContextDataHolders 
 stack when it is empty.

 Is it a possible scenario that this stack being empty and shouldn't it be 
 handled at CarbonContextDataHolder (check isEmpty in stack)?

 [1] - https://wso2.org/jira/browse/ESBJAVA-3832

 Thanks

 --
 Jagath Ariyarathne
 Technical Lead
 WSO2 Inc.  http://wso2.com/
 Email: jaga...@wso2.com
 Mob  : +94 77 386 7048




-- 

Best Regards,

Malaka Silva
Senior Tech Lead
M: +94 777 219 791
Tel : 94 11 214 5345
Fax :94 11 2145300
Skype : malaka.sampath.silva
LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77
Blog : http://mrmalakasilva.blogspot.com/

WSO2, Inc.
lean . enterprise . middleware
http://www.wso2.com/
http://www.wso2.com/about/team/malaka-silva/
http://wso2.com/about/team/malaka-silva/

Save a tree -Conserve nature  Save the world for your future. Print this
email only if it is absolutely necessary.
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [ESB] EmptyStackException when resuming a paused message processor

[Kasun Indrasiri]

Yeah, we should always check for an empty stack.

On Tue, Jul 7, 2015 at 5:17 PM, Malaka Silva mal...@wso2.com wrote:

 I think we need to check isEmpty as well.

 On Tue, Jul 7, 2015 at 3:41 PM, Jagath Sisirakumara Ariyarathne 
 jaga...@wso2.com wrote:

 Hi,

 I am working on [1] and found that the cause of the exception mentioned
 below is in the code segment in 
 org.wso2.carbon.context.internal.CarbonContextDataHolder
 in carbon.utils.

 public void endTenantFlow() {

 StackCarbonContextDataHolder carbonContextDataHolders = 
 parentContextHolderStack.get();
 if (carbonContextDataHolders != null) {
 currentContextHolder.set(carbonContextDataHolders.pop());
 }
 }

 *Exception :*

 java.util.EmptyStackException
  at java.util.Stack.peek(Stack.java:102)
  at java.util.Stack.pop(Stack.java:84)
  at 
 org.wso2.carbon.context.internal.CarbonContextDataHolder.endTenantFlow(CarbonContextDataHolder.java:1291)
  at 
 org.wso2.carbon.context.PrivilegedCarbonContext.endTenantFlow(PrivilegedCarbonContext.java:75)
  at 
 org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute(TaskQuartzJobAdapter.java:69)
  at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
  at 
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  at java.util.concurrent.FutureTask.run(FutureTask.java:262)
  at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:745)

 Issue occurs when it tries to pop elements from carbonContextDataHolders 
 stack when it is empty.

 Is it a possible scenario that this stack being empty and shouldn't it be 
 handled at CarbonContextDataHolder (check isEmpty in stack)?

 [1] - https://wso2.org/jira/browse/ESBJAVA-3832

 Thanks

 --
 Jagath Ariyarathne
 Technical Lead
 WSO2 Inc.  http://wso2.com/
 Email: jaga...@wso2.com
 Mob  : +94 77 386 7048




 --

 Best Regards,

 Malaka Silva
 Senior Tech Lead
 M: +94 777 219 791
 Tel : 94 11 214 5345
 Fax :94 11 2145300
 Skype : malaka.sampath.silva
 LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77
 Blog : http://mrmalakasilva.blogspot.com/

 WSO2, Inc.
 lean . enterprise . middleware
 http://www.wso2.com/
 http://www.wso2.com/about/team/malaka-silva/
 http://wso2.com/about/team/malaka-silva/

 Save a tree -Conserve nature  Save the world for your future. Print this
 email only if it is absolutely necessary.

 ___
 Dev mailing list
 Dev@wso2.org
 http://wso2.org/cgi-bin/mailman/listinfo/dev




-- 
Kasun Indrasiri
Software Architect
WSO2, Inc.; http://wso2.com
lean.enterprise.middleware

cell: +94 77 556 5206
Blog : http://kasunpanorama.blogspot.com/
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

[Dev] [ESB] EmptyStackException when resuming a paused message processor

[Jagath Sisirakumara Ariyarathne]

Hi,

I am working on [1] and found that the cause of the exception mentioned
below is in the code segment in
org.wso2.carbon.context.internal.CarbonContextDataHolder
in carbon.utils.

public void endTenantFlow() {

StackCarbonContextDataHolder carbonContextDataHolders =
parentContextHolderStack.get();
if (carbonContextDataHolders != null) {
currentContextHolder.set(carbonContextDataHolders.pop());
}
}

*Exception :*

java.util.EmptyStackException
at java.util.Stack.peek(Stack.java:102)
at java.util.Stack.pop(Stack.java:84)
at 
org.wso2.carbon.context.internal.CarbonContextDataHolder.endTenantFlow(CarbonContextDataHolder.java:1291)
at 
org.wso2.carbon.context.PrivilegedCarbonContext.endTenantFlow(PrivilegedCarbonContext.java:75)
at 
org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute(TaskQuartzJobAdapter.java:69)
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Issue occurs when it tries to pop elements from
carbonContextDataHolders stack when it is empty.

Is it a possible scenario that this stack being empty and shouldn't it
be handled at CarbonContextDataHolder (check isEmpty in stack)?

[1] - https://wso2.org/jira/browse/ESBJAVA-3832

Thanks

-- 
Jagath Ariyarathne
Technical Lead
WSO2 Inc.  http://wso2.com/
Email: jaga...@wso2.com
Mob  : +94 77 386 7048
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [ESB] EmptyStackException when resuming a paused message processor

[Jagath Sisirakumara Ariyarathne]

Hi Carbon Team,

Please review and merge the PR[1] to fix the JIRA[2].

[1] - https://github.com/wso2/carbon4-kernel/pull/285
[2] - https://wso2.org/jira/browse/ESBJAVA-3832

Thanks.

On Tue, Jul 7, 2015 at 5:47 PM, Kasun Indrasiri ka...@wso2.com wrote:

 Yeah, we should always check for an empty stack.

 On Tue, Jul 7, 2015 at 5:17 PM, Malaka Silva mal...@wso2.com wrote:

 I think we need to check isEmpty as well.

 On Tue, Jul 7, 2015 at 3:41 PM, Jagath Sisirakumara Ariyarathne 
 jaga...@wso2.com wrote:

 Hi,

 I am working on [1] and found that the cause of the exception mentioned
 below is in the code segment in 
 org.wso2.carbon.context.internal.CarbonContextDataHolder
 in carbon.utils.

 public void endTenantFlow() {

 StackCarbonContextDataHolder carbonContextDataHolders = 
 parentContextHolderStack.get();
 if (carbonContextDataHolders != null) {
 currentContextHolder.set(carbonContextDataHolders.pop());
 }
 }

 *Exception :*

 java.util.EmptyStackException
 at java.util.Stack.peek(Stack.java:102)
 at java.util.Stack.pop(Stack.java:84)
 at 
 org.wso2.carbon.context.internal.CarbonContextDataHolder.endTenantFlow(CarbonContextDataHolder.java:1291)
 at 
 org.wso2.carbon.context.PrivilegedCarbonContext.endTenantFlow(PrivilegedCarbonContext.java:75)
 at 
 org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute(TaskQuartzJobAdapter.java:69)
 at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
 at 
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
 at java.util.concurrent.FutureTask.run(FutureTask.java:262)
 at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:745)

 Issue occurs when it tries to pop elements from carbonContextDataHolders 
 stack when it is empty.

 Is it a possible scenario that this stack being empty and shouldn't it be 
 handled at CarbonContextDataHolder (check isEmpty in stack)?

 [1] - https://wso2.org/jira/browse/ESBJAVA-3832

 Thanks

 --
 Jagath Ariyarathne
 Technical Lead
 WSO2 Inc.  http://wso2.com/
 Email: jaga...@wso2.com
 Mob  : +94 77 386 7048




 --

 Best Regards,

 Malaka Silva
 Senior Tech Lead
 M: +94 777 219 791
 Tel : 94 11 214 5345
 Fax :94 11 2145300
 Skype : malaka.sampath.silva
 LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77
 Blog : http://mrmalakasilva.blogspot.com/

 WSO2, Inc.
 lean . enterprise . middleware
 http://www.wso2.com/
 http://www.wso2.com/about/team/malaka-silva/
 http://wso2.com/about/team/malaka-silva/

 Save a tree -Conserve nature  Save the world for your future. Print this
 email only if it is absolutely necessary.

 ___
 Dev mailing list
 Dev@wso2.org
 http://wso2.org/cgi-bin/mailman/listinfo/dev




 --
 Kasun Indrasiri
 Software Architect
 WSO2, Inc.; http://wso2.com
 lean.enterprise.middleware

 cell: +94 77 556 5206
 Blog : http://kasunpanorama.blogspot.com/




-- 
Jagath Ariyarathne
Technical Lead
WSO2 Inc.  http://wso2.com/
Email: jaga...@wso2.com
Mob  : +94 77 386 7048
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [ESB] EmptyStackException when resuming a paused message processor

[Sameera Jayasoma]

I understand, but we need to understand why that stack becomes empty.
AFAIK, if we follow the proper APIs, stack should not become empty

On Tue, Jul 7, 2015 at 5:47 PM, Kasun Indrasiri ka...@wso2.com wrote:

 Yeah, we should always check for an empty stack.

 On Tue, Jul 7, 2015 at 5:17 PM, Malaka Silva mal...@wso2.com wrote:

 I think we need to check isEmpty as well.

 On Tue, Jul 7, 2015 at 3:41 PM, Jagath Sisirakumara Ariyarathne 
 jaga...@wso2.com wrote:

 Hi,

 I am working on [1] and found that the cause of the exception mentioned
 below is in the code segment in 
 org.wso2.carbon.context.internal.CarbonContextDataHolder
 in carbon.utils.

 public void endTenantFlow() {

 StackCarbonContextDataHolder carbonContextDataHolders = 
 parentContextHolderStack.get();
 if (carbonContextDataHolders != null) {
 currentContextHolder.set(carbonContextDataHolders.pop());
 }
 }

 *Exception :*

 java.util.EmptyStackException
 at java.util.Stack.peek(Stack.java:102)
 at java.util.Stack.pop(Stack.java:84)
 at 
 org.wso2.carbon.context.internal.CarbonContextDataHolder.endTenantFlow(CarbonContextDataHolder.java:1291)
 at 
 org.wso2.carbon.context.PrivilegedCarbonContext.endTenantFlow(PrivilegedCarbonContext.java:75)
 at 
 org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute(TaskQuartzJobAdapter.java:69)
 at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
 at 
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
 at java.util.concurrent.FutureTask.run(FutureTask.java:262)
 at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:745)

 Issue occurs when it tries to pop elements from carbonContextDataHolders 
 stack when it is empty.

 Is it a possible scenario that this stack being empty and shouldn't it be 
 handled at CarbonContextDataHolder (check isEmpty in stack)?

 [1] - https://wso2.org/jira/browse/ESBJAVA-3832

 Thanks

 --
 Jagath Ariyarathne
 Technical Lead
 WSO2 Inc.  http://wso2.com/
 Email: jaga...@wso2.com
 Mob  : +94 77 386 7048




 --

 Best Regards,

 Malaka Silva
 Senior Tech Lead
 M: +94 777 219 791
 Tel : 94 11 214 5345
 Fax :94 11 2145300
 Skype : malaka.sampath.silva
 LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77
 Blog : http://mrmalakasilva.blogspot.com/

 WSO2, Inc.
 lean . enterprise . middleware
 http://www.wso2.com/
 http://www.wso2.com/about/team/malaka-silva/
 http://wso2.com/about/team/malaka-silva/

 Save a tree -Conserve nature  Save the world for your future. Print this
 email only if it is absolutely necessary.

 ___
 Dev mailing list
 Dev@wso2.org
 http://wso2.org/cgi-bin/mailman/listinfo/dev




 --
 Kasun Indrasiri
 Software Architect
 WSO2, Inc.; http://wso2.com
 lean.enterprise.middleware

 cell: +94 77 556 5206
 Blog : http://kasunpanorama.blogspot.com/

 ___
 Dev mailing list
 Dev@wso2.org
 http://wso2.org/cgi-bin/mailman/listinfo/dev




-- 
Sameera Jayasoma,
Software Architect,

WSO2, Inc. (http://wso2.com)
email: same...@wso2.com
blog: http://blog.sameera.org
twitter: https://twitter.com/sameerajayasoma
flickr: http://www.flickr.com/photos/sameera-jayasoma/collections
Mobile: 0094776364456

Lean . Enterprise . Middleware
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [ESB] EmptyStackException when resuming a paused message processor

[Jagath Sisirakumara Ariyarathne]

Hi Sameera,

I will check it and update.

Thanks.

On Wed, Jul 8, 2015 at 11:10 AM, Sameera Jayasoma same...@wso2.com wrote:

 Hi Jagath,

 Can you debug and see whey the stack becomes empty? Thats a serious
 problem. Stack should be become empty here.

 Checking whether the stack is empty will stop the error log, but it
 doesn't fix the actual problem here.

 On Wed, Jul 8, 2015 at 10:50 AM, Sameera Jayasoma same...@wso2.com
 wrote:

 I understand, but we need to understand why that stack becomes empty.
 AFAIK, if we follow the proper APIs, stack should not become empty

 On Tue, Jul 7, 2015 at 5:47 PM, Kasun Indrasiri ka...@wso2.com wrote:

 Yeah, we should always check for an empty stack.

 On Tue, Jul 7, 2015 at 5:17 PM, Malaka Silva mal...@wso2.com wrote:

 I think we need to check isEmpty as well.

 On Tue, Jul 7, 2015 at 3:41 PM, Jagath Sisirakumara Ariyarathne 
 jaga...@wso2.com wrote:

 Hi,

 I am working on [1] and found that the cause of the exception
 mentioned below is in the code segment in org
 .wso2.carbon.context.internal.CarbonContextDataHolder in carbon.utils.

 public void endTenantFlow() {

 StackCarbonContextDataHolder carbonContextDataHolders = 
 parentContextHolderStack.get();
 if (carbonContextDataHolders != null) {
 currentContextHolder.set(carbonContextDataHolders.pop());
 }
 }

 *Exception :*

 java.util.EmptyStackException
   at java.util.Stack.peek(Stack.java:102)
   at java.util.Stack.pop(Stack.java:84)
   at 
 org.wso2.carbon.context.internal.CarbonContextDataHolder.endTenantFlow(CarbonContextDataHolder.java:1291)
   at 
 org.wso2.carbon.context.PrivilegedCarbonContext.endTenantFlow(PrivilegedCarbonContext.java:75)
   at 
 org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute(TaskQuartzJobAdapter.java:69)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
   at 
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:745)

 Issue occurs when it tries to pop elements from carbonContextDataHolders 
 stack when it is empty.

 Is it a possible scenario that this stack being empty and shouldn't it be 
 handled at CarbonContextDataHolder (check isEmpty in stack)?

 [1] - https://wso2.org/jira/browse/ESBJAVA-3832

 Thanks

 --
 Jagath Ariyarathne
 Technical Lead
 WSO2 Inc.  http://wso2.com/
 Email: jaga...@wso2.com
 Mob  : +94 77 386 7048




 --

 Best Regards,

 Malaka Silva
 Senior Tech Lead
 M: +94 777 219 791
 Tel : 94 11 214 5345
 Fax :94 11 2145300
 Skype : malaka.sampath.silva
 LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77
 Blog : http://mrmalakasilva.blogspot.com/

 WSO2, Inc.
 lean . enterprise . middleware
 http://www.wso2.com/
 http://www.wso2.com/about/team/malaka-silva/
 http://wso2.com/about/team/malaka-silva/

 Save a tree -Conserve nature  Save the world for your future. Print
 this email only if it is absolutely necessary.

 ___
 Dev mailing list
 Dev@wso2.org
 http://wso2.org/cgi-bin/mailman/listinfo/dev




 --
 Kasun Indrasiri
 Software Architect
 WSO2, Inc.; http://wso2.com
 lean.enterprise.middleware

 cell: +94 77 556 5206
 Blog : http://kasunpanorama.blogspot.com/

 ___
 Dev mailing list
 Dev@wso2.org
 http://wso2.org/cgi-bin/mailman/listinfo/dev




 --
 Sameera Jayasoma,
 Software Architect,

 WSO2, Inc. (http://wso2.com)
 email: same...@wso2.com
 blog: http://blog.sameera.org
 twitter: https://twitter.com/sameerajayasoma
 flickr: http://www.flickr.com/photos/sameera-jayasoma/collections
 Mobile: 0094776364456

 Lean . Enterprise . Middleware




 --
 Sameera Jayasoma,
 Software Architect,

 WSO2, Inc. (http://wso2.com)
 email: same...@wso2.com
 blog: http://blog.sameera.org
 twitter: https://twitter.com/sameerajayasoma
 flickr: http://www.flickr.com/photos/sameera-jayasoma/collections
 Mobile: 0094776364456

 Lean . Enterprise . Middleware


 ___
 Dev mailing list
 Dev@wso2.org
 http://wso2.org/cgi-bin/mailman/listinfo/dev




-- 
Jagath Ariyarathne
Technical Lead
WSO2 Inc.  http://wso2.com/
Email: jaga...@wso2.com
Mob  : +94 77 386 7048
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] [ESB] EmptyStackException when resuming a paused message processor

[Sameera Jayasoma]

Hi Jagath,

Can you debug and see whey the stack becomes empty? Thats a serious
problem. Stack should be become empty here.

Checking whether the stack is empty will stop the error log, but it doesn't
fix the actual problem here.

On Wed, Jul 8, 2015 at 10:50 AM, Sameera Jayasoma same...@wso2.com wrote:

 I understand, but we need to understand why that stack becomes empty.
 AFAIK, if we follow the proper APIs, stack should not become empty

 On Tue, Jul 7, 2015 at 5:47 PM, Kasun Indrasiri ka...@wso2.com wrote:

 Yeah, we should always check for an empty stack.

 On Tue, Jul 7, 2015 at 5:17 PM, Malaka Silva mal...@wso2.com wrote:

 I think we need to check isEmpty as well.

 On Tue, Jul 7, 2015 at 3:41 PM, Jagath Sisirakumara Ariyarathne 
 jaga...@wso2.com wrote:

 Hi,

 I am working on [1] and found that the cause of the exception mentioned
 below is in the code segment in 
 org.wso2.carbon.context.internal.CarbonContextDataHolder
 in carbon.utils.

 public void endTenantFlow() {

 StackCarbonContextDataHolder carbonContextDataHolders = 
 parentContextHolderStack.get();
 if (carbonContextDataHolders != null) {
 currentContextHolder.set(carbonContextDataHolders.pop());
 }
 }

 *Exception :*

 java.util.EmptyStackException
at java.util.Stack.peek(Stack.java:102)
at java.util.Stack.pop(Stack.java:84)
at 
 org.wso2.carbon.context.internal.CarbonContextDataHolder.endTenantFlow(CarbonContextDataHolder.java:1291)
at 
 org.wso2.carbon.context.PrivilegedCarbonContext.endTenantFlow(PrivilegedCarbonContext.java:75)
at 
 org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute(TaskQuartzJobAdapter.java:69)
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at 
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

 Issue occurs when it tries to pop elements from carbonContextDataHolders 
 stack when it is empty.

 Is it a possible scenario that this stack being empty and shouldn't it be 
 handled at CarbonContextDataHolder (check isEmpty in stack)?

 [1] - https://wso2.org/jira/browse/ESBJAVA-3832

 Thanks

 --
 Jagath Ariyarathne
 Technical Lead
 WSO2 Inc.  http://wso2.com/
 Email: jaga...@wso2.com
 Mob  : +94 77 386 7048




 --

 Best Regards,

 Malaka Silva
 Senior Tech Lead
 M: +94 777 219 791
 Tel : 94 11 214 5345
 Fax :94 11 2145300
 Skype : malaka.sampath.silva
 LinkedIn : http://www.linkedin.com/pub/malaka-silva/6/33/77
 Blog : http://mrmalakasilva.blogspot.com/

 WSO2, Inc.
 lean . enterprise . middleware
 http://www.wso2.com/
 http://www.wso2.com/about/team/malaka-silva/
 http://wso2.com/about/team/malaka-silva/

 Save a tree -Conserve nature  Save the world for your future. Print
 this email only if it is absolutely necessary.

 ___
 Dev mailing list
 Dev@wso2.org
 http://wso2.org/cgi-bin/mailman/listinfo/dev




 --
 Kasun Indrasiri
 Software Architect
 WSO2, Inc.; http://wso2.com
 lean.enterprise.middleware

 cell: +94 77 556 5206
 Blog : http://kasunpanorama.blogspot.com/

 ___
 Dev mailing list
 Dev@wso2.org
 http://wso2.org/cgi-bin/mailman/listinfo/dev




 --
 Sameera Jayasoma,
 Software Architect,

 WSO2, Inc. (http://wso2.com)
 email: same...@wso2.com
 blog: http://blog.sameera.org
 twitter: https://twitter.com/sameerajayasoma
 flickr: http://www.flickr.com/photos/sameera-jayasoma/collections
 Mobile: 0094776364456

 Lean . Enterprise . Middleware




-- 
Sameera Jayasoma,
Software Architect,

WSO2, Inc. (http://wso2.com)
email: same...@wso2.com
blog: http://blog.sameera.org
twitter: https://twitter.com/sameerajayasoma
flickr: http://www.flickr.com/photos/sameera-jayasoma/collections
Mobile: 0094776364456

Lean . Enterprise . Middleware
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev