As Amila has pointed out, the issue happens in the tenant case due because HTTP_HEADERS property is not copied in the TenantTransportSender. Is there any reason why this is not done in TenantTransportSender ?
Regards Nandika On Tue, May 20, 2014 at 11:08 AM, Hasitha Aravinda <[email protected]> wrote: > > > Hi, > > On Tue, May 20, 2014 at 11:02 AM, Asanthi Kulasinghe <[email protected]> > wrote: > >> Hi, >> >> There were 2 scenarios reproducing this error initially. >> >> 1. When completing workflow tasks ( >> https://wso2.org/jira/browse/APIMANAGER-2360) - FIXED >> 2. When invoking workflows >> >> I can reproduce the following error intermittently on BPS for scenario 2. >> Tasks do not get created when this is reproduced. >> >> >> [2014-05-20 10:16:50,054] ERROR >> {org.wso2.carbon.bpel.b4p.extension.PeopleActivity} - >> org.apache.axis2.AxisFault: Timeout waiting for connection >> org.apache.axis2.AxisFault: Timeout waiting for connection >> > > Increase HTTP connection as mentioned on jira BPS-507. This will solve > this problem. Also refer [1] for BPS performance tuning. > > [1] - > http://tryitnw.blogspot.com/2014/05/wso2-bps-320-performance-tuning-know.html > > Thanks, > Hasitha. > > >> at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430) >> at >> org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:197) >> at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:75) >> at >> org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:451) >> at >> org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:278) >> at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442) >> at >> org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:430) >> at >> org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225) >> >> at >> org.apache.axis2.client.OperationClient.execute(OperationClient.java:149) >> at >> org.wso2.carbon.bpel.core.ode.integration.utils.AxisServiceUtils.invokeService(AxisServiceUtils.java:305) >> at >> org.wso2.carbon.bpel.b4p.extension.PeopleActivity.invoke(PeopleActivity.java:734) >> at >> org.wso2.carbon.bpel.b4p.extension.BPEL4PeopleExtensionOperation.runAsync(BPEL4PeopleExtensionOperation.java:66) >> at >> org.apache.ode.bpel.runtime.extension.AbstractLongRunningExtensionOperation.run(AbstractLongRunningExtensionOperation.java:95) >> at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> >> at >> org.apache.ode.jacob.vpu.JacobVPU$JacobThreadImpl.run(JacobVPU.java:451) >> at org.apache.ode.jacob.vpu.JacobVPU.execute(JacobVPU.java:139) >> at >> org.apache.ode.bpel.engine.BpelRuntimeContextImpl.execute(BpelRuntimeContextImpl.java:898) >> at >> org.apache.ode.bpel.engine.PartnerLinkMyRoleImpl.invokeNewInstance(PartnerLinkMyRoleImpl.java:208) >> at >> org.apache.ode.bpel.engine.BpelProcess$1.invoke(BpelProcess.java:283) >> >> at >> org.apache.ode.bpel.engine.BpelProcess.invokeProcess(BpelProcess.java:224) >> at >> org.apache.ode.bpel.engine.BpelProcess.invokeProcess(BpelProcess.java:279) >> at >> org.apache.ode.bpel.engine.BpelProcess.handleJobDetails(BpelProcess.java:434) >> at >> org.apache.ode.bpel.engine.BpelEngineImpl.onScheduledJob(BpelEngineImpl.java:558) >> at >> org.apache.ode.bpel.engine.BpelServerImpl.onScheduledJob(BpelServerImpl.java:467) >> at >> org.apache.ode.scheduler.simple.SimpleScheduler$RunJob$1.call(SimpleScheduler.java:536) >> at >> org.apache.ode.scheduler.simple.SimpleScheduler$RunJob$1.call(SimpleScheduler.java:530) >> at >> org.apache.ode.scheduler.simple.SimpleScheduler.execTransaction(SimpleScheduler.java:280) >> at >> org.apache.ode.scheduler.simple.SimpleScheduler.execTransaction(SimpleScheduler.java:235) >> at >> org.apache.ode.scheduler.simple.SimpleScheduler$RunJob.call(SimpleScheduler.java:530) >> at >> org.apache.ode.scheduler.simple.SimpleScheduler$RunJob.call(SimpleScheduler.java:514) >> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:744) >> Caused by: org.apache.commons.httpclient.ConnectionPoolTimeoutException: >> Timeout waiting for connection >> at >> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:497) >> at >> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:416) >> at >> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:153) >> >> at >> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) >> at >> org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:622) >> at >> org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:193) >> ... 34 more >> >> >> >> On Thu, May 15, 2014 at 10:41 PM, Amila De Silva <[email protected]> wrote: >> >>> Hi Charitha, >>> Do you get this for the super tenant or for a tenant? If it is for the >>> super tenant, can you try by changing the application/json message builder >>> to org.apache.axis2.json.JSONBuilder? >>> >>> >>> On Fri, May 9, 2014 at 10:04 PM, Charitha Kankanamge <[email protected]> >>> wrote: >>> >>>> Hi Amila, >>>> Seems we are still seeing this error. I observe the following error >>>> frequently in BPS-3.2.0 when working with AM workflow scenarios. Everything >>>> running locally so not sure readtimeout makes sense. >>>> https://wso2.org/jira/browse/BPS-507 >>>> >>>> >>>> [2014-05-09 21:36:44,857] ERROR >>>> {org.wso2.carbon.bpel.core.ode.integration.PartnerService} - Error >>>> sending message to Axis2 for ODE mex >>>> {PartnerRoleMex#hqejbhcnphr99p0v6qjh9q [PID {http://workflow.ap >>>> plication.apimgt.carbon.wso2.org}ApplicationApprovalWorkFlowProcess-1] >>>> calling [email protected](...) >>>> Status REQUEST} >>>> org.apache.axis2.AxisFault: Read timed out >>>> at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430) >>>> at >>>> org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:197) >>>> at >>>> org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:75) >>>> at >>>> org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(Common >>>> sHTTPTransportSender.java:451) >>>> at >>>> org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSen >>>> der.java:278) >>>> at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442) >>>> at >>>> org.apache.axis2.description.OutOnlyAxisOperationClient.executeImpl(OutOnlyAxisOperation. >>>> java:297) >>>> at >>>> org.apache.axis2.client.OperationClient.execute(OperationClient.java:149) >>>> at >>>> org.wso2.carbon.bpel.core.ode.integration.utils.AxisServiceUtils.invokeService(AxisServic >>>> eUtils.java:305) >>>> at >>>> org.wso2.carbon.bpel.core.ode.integration.PartnerService.invoke(PartnerService.java:324) >>>> at >>>> org.wso2.carbon.bpel.core.ode.integration.BPELMessageExchangeContextImpl.invokePartner(BP >>>> ELMessageExchangeContextImpl.java:43) >>>> at >>>> org.apache.ode.bpel.engine.BpelRuntimeContextImpl.invoke(BpelRuntimeContextImpl.java:793) >>>> at org.apache.ode.bpel.runtime.INVOKE.run(INVOKE.java:130) >>>> at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source) >>>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> at java.lang.reflect.Method.invoke(Method.java:601) >>>> at >>>> org.apache.ode.jacob.vpu.JacobVPU$JacobThreadImpl.run(JacobVPU.java:451) >>>> at org.apache.ode.jacob.vpu.JacobVPU.execute(JacobVPU.java:139) >>>> at >>>> org.apache.ode.bpel.engine.BpelRuntimeContextImpl.execute(BpelRuntimeContextImpl.java:898 >>>> ) >>>> at >>>> org.apache.ode.bpel.engine.PartnerLinkMyRoleImpl.invokeInstance(PartnerLinkMyRoleImpl.jav >>>> a:250) >>>> at >>>> org.apache.ode.bpel.engine.BpelProcess$1.invoke(BpelProcess.java:288) >>>> at >>>> org.apache.ode.bpel.engine.BpelProcess.invokeProcess(BpelProcess.java:224) >>>> at >>>> org.apache.ode.bpel.engine.BpelProcess.invokeProcess(BpelProcess.java:279) >>>> at >>>> org.apache.ode.bpel.engine.BpelProcess.handleJobDetails(BpelProcess.java:434) >>>> at >>>> org.apache.ode.bpel.engine.BpelEngineImpl.onScheduledJob(BpelEngineImpl.java:558) >>>> at >>>> org.apache.ode.bpel.engine.BpelServerImpl.onScheduledJob(BpelServerImpl.java:467) >>>> at >>>> org.apache.ode.scheduler.simple.SimpleScheduler$RunJob$1.call(SimpleScheduler.java:536) >>>> at >>>> org.apache.ode.scheduler.simple.SimpleScheduler$RunJob$1.call(SimpleScheduler.java:530) >>>> at >>>> org.apache.ode.scheduler.simple.SimpleScheduler.execTransaction(SimpleScheduler.java:280) >>>> >>>> at >>>> org.apache.ode.scheduler.simple.SimpleScheduler.execTransaction(SimpleScheduler.java:235) >>>> >>>> at >>>> org.apache.ode.scheduler.simple.SimpleScheduler$RunJob.call(SimpleScheduler.java:530) >>>> at >>>> org.apache.ode.scheduler.simple.SimpleScheduler$RunJob.call(SimpleScheduler.java:514) >>>> at >>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >>>> at java.lang.Thread.run(Thread.java:722) >>>> Caused by: java.net.SocketTimeoutException: Read timed out >>>> at java.net.SocketInputStream.socketRead0(Native Method) >>>> at java.net.SocketInputStream.read(SocketInputStream.java:150) >>>> at java.net.SocketInputStream.read(SocketInputStream.java:121) >>>> at sun.security.ssl.InputRecord.readFully(InputRecord.java:312) >>>> at sun.security.ssl.InputRecord.read(InputRecord.java:350) >>>> at >>>> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:893) >>>> at >>>> sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:850) >>>> at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) >>>> at >>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:235) >>>> at >>>> java.io.BufferedInputStream.read(BufferedInputStream.java:254) >>>> at >>>> org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) >>>> at >>>> org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) >>>> at >>>> org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) >>>> at >>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.re >>>> adLine(MultiThreadedHttpConnectionManager.java:1413) >>>> at >>>> org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) >>>> at >>>> org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) >>>> at >>>> org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) >>>> at >>>> org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java >>>> :398) >>>> at >>>> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:17 >>>> 1) >>>> at >>>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) >>>> at >>>> org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java: >>>> 622) >>>> at >>>> org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:193) >>>> ... 35 more >>>> >>>> >>>> Charitha Kankanamge >>>> cell: +1 812-391-7730 >>>> http://charithaka.blogspot.com >>>> *SOA and Testing without nonsense* >>>> >>>> >>>> On Mon, Mar 17, 2014 at 7:30 PM, Amila De Silva <[email protected]> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> Got the following error[1] when invoking a BPEL workflow hosted by a >>>>> tenant. >>>>> When the BPEL workflow is invoked , it creates a HumanTask instance >>>>> and suspends until the task is completed. Upon completing the task, BPEL >>>>> resumes its execution and sends the outcome of the task to a callback >>>>> service. >>>>> The call back service is secured with Basic Authentication and the >>>>> credentials to invoke the service is provided in an .epr file which gets >>>>> deployed with the BPEL . While investigating the error it was found that >>>>> the Authorization header is not sent when the BPEL resumes and calls the >>>>> callback service. >>>>> >>>>> As per the current investigations the problem occurs due to the >>>>> following reasons. >>>>> >>>>> 1. When resuming the BPEL, at some point, UnifiedEndpointHandler reads >>>>> the .epr file get the credentials and create the Basic Authentication >>>>> header. This header is set to the messageContext using, >>>>> >>>>> msgContext.getOptions().setProperty(HTTPConstants.HTTP_HEADERS, >>>>> headers); >>>>> >>>>> 2. Before sending the request out, CommonsHTTPTransportSender reads >>>>> this header from the message Context. >>>>> >>>>> Object httpHeadersObj = >>>>> msgContext.getProperty(HTTPConstants.HTTP_HEADERS); >>>>> >>>>> 3. If the BPEL is deployed in super.tenants space, by the time >>>>> execution reaches the TransportSender, the Header gets properly set into a >>>>> property in the messageContext. But in case of a tenant, this doesn't >>>>> happen. >>>>> >>>>> One way to fix this would be to populate this Header within the >>>>> TenantTransportSender. The other way would be to locate the point where >>>>> this is done in the super tenants flow and incorporate that when invoking >>>>> a >>>>> tenant's flow. >>>>> >>>>> [1] [2014-03-17 17:00:51,520] ERROR >>>>> {org.wso2.carbon.bpel.core.ode.integration.PartnerService} - Error >>>>> sending >>>>> message to Axis2 for ODE mex {PartnerRoleMex#hqejbhcnphr94qgl0ib2e4 [PID { >>>>> http://workflow.application.apimgt.carbon.wso2.org}ApplicationApprovalWorkFlowProcess-2] >>>>> calling [email protected](...) >>>>> Status REQUEST} >>>>> org.apache.axis2.AxisFault: Transport error: 401 Error: Unauthorized >>>>> at >>>>> org.apache.axis2.transport.http.HTTPSender.handleResponse(HTTPSender.java:311) >>>>> at >>>>> org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:194) >>>>> at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:75) >>>>> at >>>>> org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:451) >>>>> at >>>>> org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:278) >>>>> at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442) >>>>> at >>>>> org.wso2.carbon.core.multitenancy.transports.TenantTransportSender.invoke(TenantTransportSender.java:198) >>>>> at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442) >>>>> at >>>>> org.apache.axis2.description.OutOnlyAxisOperationClient.executeImpl(OutOnlyAxisOperation.java:297) >>>>> at >>>>> org.apache.axis2.client.OperationClient.execute(OperationClient.java:149) >>>>> at >>>>> org.wso2.carbon.bpel.core.ode.integration.utils.AxisServiceUtils.invokeService(AxisServiceUtils.java:305) >>>>> at >>>>> org.wso2.carbon.bpel.core.ode.integration.PartnerService.invoke(PartnerService.java:324) >>>>> at >>>>> org.wso2.carbon.bpel.core.ode.integration.BPELMessageExchangeContextImpl.invokePartner(BPELMessageExchangeContextImpl.java:43) >>>>> at >>>>> org.apache.ode.bpel.engine.BpelRuntimeContextImpl.invoke(BpelRuntimeContextImpl.java:793) >>>>> at org.apache.ode.bpel.runtime.INVOKE.run(INVOKE.java:130) >>>>> at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) >>>>> at >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>> at >>>>> org.apache.ode.jacob.vpu.JacobVPU$JacobThreadImpl.run(JacobVPU.java:451) >>>>> at org.apache.ode.jacob.vpu.JacobVPU.execute(JacobVPU.java:139) >>>>> at >>>>> org.apache.ode.bpel.engine.BpelRuntimeContextImpl.execute(BpelRuntimeContextImpl.java:898) >>>>> at >>>>> org.apache.ode.bpel.engine.PartnerLinkMyRoleImpl.invokeInstance(PartnerLinkMyRoleImpl.java:250) >>>>> at >>>>> org.apache.ode.bpel.engine.BpelProcess$1.invoke(BpelProcess.java:288) >>>>> at >>>>> org.apache.ode.bpel.engine.BpelProcess.invokeProcess(BpelProcess.java:224) >>>>> at >>>>> org.apache.ode.bpel.engine.BpelProcess.invokeProcess(BpelProcess.java:279) >>>>> at >>>>> org.apache.ode.bpel.engine.BpelProcess.handleJobDetails(BpelProcess.java:434) >>>>> at >>>>> org.apache.ode.bpel.engine.BpelEngineImpl.onScheduledJob(BpelEngineImpl.java:558) >>>>> at >>>>> org.apache.ode.bpel.engine.BpelServerImpl.onScheduledJob(BpelServerImpl.java:467) >>>>> at >>>>> org.apache.ode.scheduler.simple.SimpleScheduler$RunJob$1.call(SimpleScheduler.java:536) >>>>> at >>>>> org.apache.ode.scheduler.simple.SimpleScheduler$RunJob$1.call(SimpleScheduler.java:530) >>>>> at >>>>> org.apache.ode.scheduler.simple.SimpleScheduler.execTransaction(SimpleScheduler.java:280) >>>>> at >>>>> org.apache.ode.scheduler.simple.SimpleScheduler.execTransaction(SimpleScheduler.java:235) >>>>> at >>>>> org.apache.ode.scheduler.simple.SimpleScheduler$RunJob.call(SimpleScheduler.java:530) >>>>> at >>>>> org.apache.ode.scheduler.simple.SimpleScheduler$RunJob.call(SimpleScheduler.java:514) >>>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> -- >>>>> *Amila De Silva* >>>>> >>>>> *Software Engineer* >>>>> WSO2 Inc. >>>>> mobile :(+94) 775119302 >>>>> >>>>> >>>>> _______________________________________________ >>>>> Dev mailing list >>>>> [email protected] >>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>> >>>>> >>>> >>> >>> >>> -- >>> *Amila De Silva* >>> >>> WSO2 Inc. >>> mobile :(+94) 775119302 >>> >>> >>> _______________________________________________ >>> Dev mailing list >>> [email protected] >>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>> >>> >> >> >> -- >> *Asanthi Kulasinghe* >> WSO2 Inc; http://www.wso2.com/. >> Mobile: +94777355522 >> >> >> >> _______________________________________________ >> Dev mailing list >> [email protected] >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > Hasitha Aravinda, > Software Engineer, > WSO2 Inc. > Email: [email protected] > Mobile: +94 71 8 210 200 > > > _______________________________________________ > Dev mailing list > [email protected] > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- Nandika Jayawardana Senior Technical Lead WSO2 Inc ; http://wso2.com lean.enterprise.middleware
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
