Hi Nuwan, Here is the mail thread. [Dev][ESB] EmptyStackException when resuming a paused message processor
On Wed, Oct 7, 2015 at 5:22 PM, Nuwan Dias <[email protected]> wrote: > Hi Chanaka, > > Thanks for looking into this. Hope we can get it merged to the kernel soon > :). > > Thanks, > NuwanD. > > On Wed, Oct 7, 2015 at 5:19 PM, Chanaka Fernando <[email protected]> > wrote: > >> Hi NuwanD, >> >> I have found the root cause for this issue. As mentioned in [1], this >> error is coming from kernel level and that has been propagated to the >> synapse level. Actual root cause is mentioned below. >> >> [2015-10-07 17:13:38,135] ERROR - ServerWorker Error processing GET >> reguest for : /t/api.com/responseapi/weather?q=London. Error detail: >> null. >> java.util.EmptyStackException >> at java.util.Stack.peek(Stack.java:102) >> at java.util.Stack.pop(Stack.java:84) >> at >> org.wso2.carbon.context.internal.CarbonContextDataHolder.endTenantFlow(CarbonContextDataHolder.java:1291) >> at >> org.wso2.carbon.context.PrivilegedCarbonContext.endTenantFlow(PrivilegedCarbonContext.java:75) >> at >> org.wso2.carbon.core.multitenancy.MultitenantMessageReceiver.processRequest(MultitenantMessageReceiver.java:229) >> at >> org.wso2.carbon.core.multitenancy.MultitenantMessageReceiver.receive(MultitenantMessageReceiver.java:78) >> at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180) >> at >> org.apache.synapse.transport.passthru.ServerWorker.processNonEntityEnclosingRESTHandler(ServerWorker.java:317) >> at >> org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:149) >> at >> org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:745) >> >> >> There is a separate mail thread to discuss about this issue and >> previously Jagath has given a fix but that was not merged to the kernel. I >> will reply to that thread and continue the fix on that thread. >> >> [1] https://wso2.org/jira/browse/ESBJAVA-4142 >> >> Thanks, >> Chanaka >> >> On Tue, Oct 6, 2015 at 4:49 PM, Chanaka Fernando <[email protected]> >> wrote: >> >>> Hi NuwanD, >>> >>> I can reproduce the issue with the provided configuration. I have >>> already created a fix for this issue and currently testing it. I'll update >>> the thread once I finalized the fix and merge the PR. >>> >>> On Tue, Oct 6, 2015 at 2:59 PM, Nuwan Dias <[email protected]> wrote: >>> >>>> Attaching the two APIs I used for testing. To reproduce this issue, >>>> create a tenant with the domain as 'wso2.com' and deploy these two >>>> APIs within the tenant space and restart the server. >>>> >>>> Then access the API without the version using the following URL. >>>> >>>> http://localhost:8280/t/wso2.com/responseapi/weather?q=London >>>> >>>> Thanks, >>>> NuwanD. >>>> >>>> On Tue, Oct 6, 2015 at 2:53 PM, Chanaka Fernando <[email protected]> >>>> wrote: >>>> >>>>> Hi Nuwan, >>>>> >>>>> I had a look into the code which is causing this NPE. According to the >>>>> way it is written, this NPE can be occurred. I have tried to replicate >>>>> what >>>>> NuwanD has mentioned but couldn't reproduce the same NPE. This can be >>>>> occurred due to synapse configuration issue. >>>>> >>>>> @NuwanD: Can you share the configurations which you have used to >>>>> reproduce this issue? >>>>> >>>>> >>>>> Thanks, >>>>> Chanaka >>>>> >>>>> On Thu, Oct 1, 2015 at 10:41 AM, Chanaka Fernando <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi All, >>>>>> >>>>>> We are looking into this issue. >>>>>> >>>>>> On Thu, Oct 1, 2015 at 10:37 AM, Sandapa Handakumbura < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi Nuwan, >>>>>>> >>>>>>> I faced a similar issue during ESB 4.9.0 release testing. First >>>>>>> request for proxy services (after tenant unloading/ server restart) >>>>>>> failed >>>>>>> in tenant mode [1]. Unfortunately ESB team could not reproduce it and I >>>>>>> could not identify the exact steps to reproduce it either. >>>>>>> >>>>>>> Hope the team will look in to this since you have identified steps >>>>>>> to reproduce the issue. >>>>>>> >>>>>>> [1] https://wso2.org/jira/browse/ESBJAVA-4142 >>>>>>> >>>>>>> Thanks, >>>>>>> Sandapa >>>>>>> >>>>>>> >>>>>>> On Thu, Oct 1, 2015 at 4:47 AM, Nuwan Dias <[email protected]> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I have a scenario on the ESB where I have created two APIs with the >>>>>>>> same context in the tenant space. The first API doesn't have a version >>>>>>>> and >>>>>>>> the second API has a version. The first API receives the request and >>>>>>>> forwards to the second API. See illustration below. >>>>>>>> >>>>>>>> Client (/twitter) --> Twitter_API (/twitter/1.0.0) --> >>>>>>>> Twitter_API_v1.0.0 >>>>>>>> >>>>>>>> This scenario however causes the following [1] NPE on the first >>>>>>>> request that's sent to the server. The subsequent requests pass until a >>>>>>>> server restart happens. >>>>>>>> >>>>>>>> I tried logging into the management console of the ESB using tenant >>>>>>>> credentials to check whether it was related to tenant loading. The >>>>>>>> first >>>>>>>> request still failed with the same error. Any idea why this is >>>>>>>> happening? >>>>>>>> This issue first occurred to me in API Manager after migrating to the >>>>>>>> latest released version of Synapse. And is causing test failures in the >>>>>>>> product. It worked fine in the older versions of Synapse (2.1.2- >>>>>>>> version >>>>>>>> range). >>>>>>>> >>>>>>>> [1] >>>>>>>> >>>>>>>> [2015-10-01 04:41:39,519] ERROR - PassThroughHttpSender Failed to >>>>>>>> submit the response >>>>>>>> java.lang.NullPointerException >>>>>>>> at >>>>>>>> org.apache.synapse.transport.passthru.util.SourceResponseFactory.create(SourceResponseFactory.java:64) >>>>>>>> at >>>>>>>> org.apache.synapse.transport.passthru.PassThroughHttpSender.submitResponse(PassThroughHttpSender.java:462) >>>>>>>> at >>>>>>>> org.apache.synapse.transport.passthru.PassThroughHttpSender.invoke(PassThroughHttpSender.java:267) >>>>>>>> at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442) >>>>>>>> at >>>>>>>> org.wso2.carbon.core.multitenancy.transports.TenantTransportSender.invoke(TenantTransportSender.java:231) >>>>>>>> at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442) >>>>>>>> at >>>>>>>> org.apache.synapse.core.axis2.Axis2Sender.sendBack(Axis2Sender.java:212) >>>>>>>> at >>>>>>>> org.apache.synapse.core.axis2.Axis2SynapseEnvironment.send(Axis2SynapseEnvironment.java:444) >>>>>>>> at >>>>>>>> org.apache.synapse.mediators.builtin.SendMediator.mediate(SendMediator.java:102) >>>>>>>> at >>>>>>>> org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:81) >>>>>>>> at >>>>>>>> org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:48) >>>>>>>> at >>>>>>>> org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:149) >>>>>>>> at org.apache.synapse.rest.Resource.process(Resource.java:297) >>>>>>>> at org.apache.synapse.rest.API.process(API.java:335) >>>>>>>> at >>>>>>>> org.apache.synapse.rest.RESTRequestHandler.dispatchToAPI(RESTRequestHandler.java:97) >>>>>>>> at >>>>>>>> org.apache.synapse.rest.RESTRequestHandler.process(RESTRequestHandler.java:52) >>>>>>>> at >>>>>>>> org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:266) >>>>>>>> at >>>>>>>> org.apache.synapse.core.axis2.SynapseCallbackReceiver.handleMessage(SynapseCallbackReceiver.java:529) >>>>>>>> at >>>>>>>> org.apache.synapse.core.axis2.SynapseCallbackReceiver.receive(SynapseCallbackReceiver.java:172) >>>>>>>> at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180) >>>>>>>> at >>>>>>>> org.wso2.carbon.core.multitenancy.MultitenantMessageReceiver.processResponse(MultitenantMessageReceiver.java:166) >>>>>>>> at >>>>>>>> org.wso2.carbon.core.multitenancy.MultitenantMessageReceiver.receive(MultitenantMessageReceiver.java:82) >>>>>>>> at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180) >>>>>>>> at >>>>>>>> org.apache.synapse.transport.passthru.ClientWorker.run(ClientWorker.java:247) >>>>>>>> at >>>>>>>> org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172) >>>>>>>> at >>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>>> at >>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>> [2015-10-01 04:41:39,521] ERROR - Axis2Sender >>>>>>>> Access-Control-Allow-Credentials:true,Access-Control-Allow-Methods:GET, >>>>>>>> POST,Access-Control-Allow-Origin:*,Content-Type:application/json; >>>>>>>> charset=utf-8,X-Source:redis,<?xml version='1.0' >>>>>>>> encoding='utf-8'?><soapenv:Envelope xmlns:soapenv=" >>>>>>>> http://www.w3.org/2003/05/soap-envelope";><soapenv:Body/></soapenv:Envelope> >>>>>>>> Unexpected error sending message back >>>>>>>> org.apache.axis2.AxisFault: Failed to submit the response >>>>>>>> at >>>>>>>> org.apache.synapse.transport.passthru.PassThroughHttpSender.handleException(PassThroughHttpSender.java:610) >>>>>>>> at >>>>>>>> org.apache.synapse.transport.passthru.PassThroughHttpSender.invoke(PassThroughHttpSender.java:269) >>>>>>>> at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442) >>>>>>>> at >>>>>>>> org.wso2.carbon.core.multitenancy.transports.TenantTransportSender.invoke(TenantTransportSender.java:231) >>>>>>>> at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442) >>>>>>>> at >>>>>>>> org.apache.synapse.core.axis2.Axis2Sender.sendBack(Axis2Sender.java:212) >>>>>>>> at >>>>>>>> org.apache.synapse.core.axis2.Axis2SynapseEnvironment.send(Axis2SynapseEnvironment.java:444) >>>>>>>> at >>>>>>>> org.apache.synapse.mediators.builtin.SendMediator.mediate(SendMediator.java:102) >>>>>>>> at >>>>>>>> org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:81) >>>>>>>> at >>>>>>>> org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:48) >>>>>>>> at >>>>>>>> org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:149) >>>>>>>> at org.apache.synapse.rest.Resource.process(Resource.java:297) >>>>>>>> at org.apache.synapse.rest.API.process(API.java:335) >>>>>>>> at >>>>>>>> org.apache.synapse.rest.RESTRequestHandler.dispatchToAPI(RESTRequestHandler.java:97) >>>>>>>> at >>>>>>>> org.apache.synapse.rest.RESTRequestHandler.process(RESTRequestHandler.java:52) >>>>>>>> at >>>>>>>> org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:266) >>>>>>>> at >>>>>>>> org.apache.synapse.core.axis2.SynapseCallbackReceiver.handleMessage(SynapseCallbackReceiver.java:529) >>>>>>>> at >>>>>>>> org.apache.synapse.core.axis2.SynapseCallbackReceiver.receive(SynapseCallbackReceiver.java:172) >>>>>>>> at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180) >>>>>>>> at >>>>>>>> org.wso2.carbon.core.multitenancy.MultitenantMessageReceiver.processResponse(MultitenantMessageReceiver.java:166) >>>>>>>> at >>>>>>>> org.wso2.carbon.core.multitenancy.MultitenantMessageReceiver.receive(MultitenantMessageReceiver.java:82) >>>>>>>> at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180) >>>>>>>> at >>>>>>>> org.apache.synapse.transport.passthru.ClientWorker.run(ClientWorker.java:247) >>>>>>>> at >>>>>>>> org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172) >>>>>>>> at >>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>>> at >>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>> Caused by: java.lang.NullPointerException >>>>>>>> at >>>>>>>> org.apache.synapse.transport.passthru.util.SourceResponseFactory.create(SourceResponseFactory.java:64) >>>>>>>> at >>>>>>>> org.apache.synapse.transport.passthru.PassThroughHttpSender.submitResponse(PassThroughHttpSender.java:462) >>>>>>>> at >>>>>>>> org.apache.synapse.transport.passthru.PassThroughHttpSender.invoke(PassThroughHttpSender.java:267) >>>>>>>> ... 25 more >>>>>>>> >>>>>>>> Thanks, >>>>>>>> NuwanD. >>>>>>>> >>>>>>>> -- >>>>>>>> Nuwan Dias >>>>>>>> >>>>>>>> Technical Lead - WSO2, Inc. http://wso2.com >>>>>>>> email : [email protected] >>>>>>>> Phone : +94 777 775 729 >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Dev mailing list >>>>>>>> [email protected] >>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Sandapa Handakumbura >>>>>>> Associate Technical Lead >>>>>>> WSO2.Inc. ; http://wso2.com/ >>>>>>> M : +94777115866 >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Dev mailing list >>>>>>> [email protected] >>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> -- >>>>>> Chanaka Fernando >>>>>> Senior Technical Lead >>>>>> WSO2, Inc.; http://wso2.com >>>>>> lean.enterprise.middleware >>>>>> >>>>>> mobile: +94 773337238 >>>>>> Blog : http://soatutorials.blogspot.com >>>>>> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0 >>>>>> Twitter:https://twitter.com/chanakaudaya >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> -- >>>>> Chanaka Fernando >>>>> Senior Technical Lead >>>>> WSO2, Inc.; http://wso2.com >>>>> lean.enterprise.middleware >>>>> >>>>> mobile: +94 773337238 >>>>> Blog : http://soatutorials.blogspot.com >>>>> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0 >>>>> Twitter:https://twitter.com/chanakaudaya >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Nuwan Dias >>>> >>>> Technical Lead - WSO2, Inc. http://wso2.com >>>> email : [email protected] >>>> Phone : +94 777 775 729 >>>> >>> >>> >>> >>> -- >>> -- >>> Chanaka Fernando >>> Senior Technical Lead >>> WSO2, Inc.; http://wso2.com >>> lean.enterprise.middleware >>> >>> mobile: +94 773337238 >>> Blog : http://soatutorials.blogspot.com >>> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0 >>> Twitter:https://twitter.com/chanakaudaya >>> >>> >>> >>> >>> >> >> >> -- >> -- >> Chanaka Fernando >> Senior Technical Lead >> WSO2, Inc.; http://wso2.com >> lean.enterprise.middleware >> >> mobile: +94 773337238 >> Blog : http://soatutorials.blogspot.com >> LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0 >> Twitter:https://twitter.com/chanakaudaya >> >> >> >> >> > > > -- > Nuwan Dias > > Technical Lead - WSO2, Inc. http://wso2.com > email : [email protected] > Phone : +94 777 775 729 > -- -- Chanaka Fernando Senior Technical Lead WSO2, Inc.; http://wso2.com lean.enterprise.middleware mobile: +94 773337238 Blog : http://soatutorials.blogspot.com LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0 Twitter:https://twitter.com/chanakaudaya
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
