Hello,
Our team is currently using Axis2 1.2 in our application and we are
running some performance and stress tests with JMeter.
While running these stress tests we have come across two issues that we
hope someone can help us with.
The first issue relates to temporary files. In our stress testing we were
using the URLBasedAxisConfigurator to load .mar files on the client side,
but got the error below with 1 user and 1000 iterations:
java.lang.RuntimeException: java.io.IOException: Too many open files;;
at
org.apache.axis2.deployment.util.Utils.getURLsForAllJars(Utils.java(Compiled
Code));;
at
org.apache.axis2.deployment.util.Utils.createClassLoader(Utils.java:684);;
at
au.gov.centrelink.jfs.proxy.axis2.ClientURLBasedAxisConfigurator.loadRepositoryFromURL(ClientURLBasedAxisConfigurator.java(Compiled
Code));;
at
org.apache.axis2.deployment.URLBasedAxisConfigurator.getAxisConfiguration(URLBasedAxisConfigurator.java:72);;
at
org.apache.axis2.context.ConfigurationContextFactory.createConfigurationContext(ConfigurationContextFactory.java:60);;
at
au.gov.centrelink.jfs.proxy.ProxyFactory.createStub(ProxyFactory.java:699);;
at
au.gov.centrelink.jfs.proxy.ProxyFactory.createProxy(ProxyFactory.java:440);;
at
au.gov.centrelink.jfs.proxy.ProxyFactory.createProxy(ProxyFactory.java:395);;
at
au.gov.centrelink.jsf.pst.core.CDCCustomerEmailAddressTester2.execute(CDCCustomerEmailAddressTester2.java:64);;
at
org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:421);;
at
org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:226);;
at
org.apache.struts.action.ActionServlet.process(ActionServlet.java:1164);;
at
org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:415);;
at javax.servlet.http.HttpServlet.service(HttpServlet.java:763);;
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856);;
at
com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1572);;
at
com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1521);;
at
com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:113);;
at
com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:82);;
at
com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:759);;
at
com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.forward(WebAppRequestDispatcher.java:334);;
at
org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1056);;
at
org.apache.struts.action.RequestProcessor.processForwardConfig(RequestProcessor.java:388);;
at
org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:231);;
at
org.apache.struts.action.ActionServlet.process(ActionServlet.java:1164);;
at
org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:415);;
at javax.servlet.http.HttpServlet.service(HttpServlet.java:763);;
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856);;
at
com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1572);;
at
com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1521);;
at
com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:136);;
at
au.gov.centrelink.itsecurity.acs.web.SecureRequestFilter.doFilter(SecureRequestFilter.java:60);;
at
com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:142);;
at
com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:121);;
at
au.gov.centrelink.jaf.liser.api.WebRequestLiser.doFilter(WebRequestLiser.java:267);;
at
au.gov.centrelink.jaf.interceptor.core.JAFFilter.doFilter(JAFFilter.java:99);;
at
com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:142);;
at
com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:121);;
at
com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:82);;
at
com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:759);;
at
com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:89);;
at
com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1924);;
at
com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:98);;
at
com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:472);;
at
com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:411);;
at
com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:288);;
at
com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminaters(NewConnectionInitialReadCallback.java:207);;
at
com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:109);;
at
com.ibm.ws.tcp.channel.impl.WorkQueueManager.requestComplete(WorkQueueManager.java:566);;
at
com.ibm.ws.tcp.channel.impl.WorkQueueManager.attemptIO(WorkQueueManager.java:619);;
at
com.ibm.ws.tcp.channel.impl.WorkQueueManager.workerRun(WorkQueueManager.java:952);;
at
com.ibm.ws.tcp.channel.impl.WorkQueueManager$Worker.run(WorkQueueManager.java:1039);;
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1471);;
Caused by: java.io.IOException: Too many open files;;
at java.io.WinNTFileSystem.createFileExclusively(Native Method);;
at java.io.File.checkAndCreate(File.java(Compiled Code));;
at java.io.File.createTempFile(File.java(Compiled Code));;
at java.io.File.createTempFile(File.java(Inlined Compiled Code));;
at
org.apache.axis2.deployment.util.Utils.createTempFile(Utils.java(Compiled
Code));; ... 53 more;; ]]
At that time we were loading the axis2 Configuration context each time the
Stub was instantiated. Looking at the code for the method
getURLsForAllJars() in org.apache.axis2.deployment.util.Utils, it looks
like a temp file is always created for the URL that is passed in. Is it
possible to change the method so it checks whether the URL uses the 'file'
protocol, and if so, read from the file directly, instead of creating a
temp file and reading from it? When we made this change we found that the
"Too many open files" problem seemed to go away. Although we have
switched to caching the configuration context rather than loading it every
time, we still think the above change would be a good one to make.
Our other issue is a WstxUnexpectedCharException that seems to occur
intermittently - it has occurred only once in our last two runs.
We were running JMeter with 20 concurrent users, each of which makes
10,000 requests (200,000 requests made in total) to a web application. The
web application acts as the web service client and makes service calls to
a servlet. The servlet returns essentially the same SOAP response for
every request it receives (the only change is in the text content of an
xml element in the SOAP body of the response, which changes slightly each
time). In the last two runs, we got one instance of the following
exception, which seems to happen in the web app while parsing the SOAP
response:
[29/06/07 13:27:14:469 EST] 0000004a SystemErr R
au.gov.centrelink.jfs.proxy.envlookup.EnvironmentLookupException:
RemoteException occurred while doing an environment lookup for CRN
200393036S
at
au.gov.centrelink.jfs.proxy.envlookup.impl.DefaultCustomerEnvironmentLookupImpl.findEnvironment(DefaultCustomerEnvironmentLookupImpl.java(Compiled
Code))
at
au.gov.centrelink.jfs.proxy.impl.AbstractCustomerServiceClient.doRouting(AbstractCustomerServiceClient.java(Compiled
Code))
at
au.gov.centrelink.jfs.proxy.impl.AbstractCustomerServiceClient$$EnhancerByCGLIB$$cdba8c46.CGLIB$doRouting$0(<generated>(Inlined
Compiled Code))
at
au.gov.centrelink.jfs.proxy.impl.AbstractCustomerServiceClient$$EnhancerByCGLIB$$cdba8c46$$FastClassByCGLIB$$183db92b.invoke(<generated>(Compiled
Code))
at
net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java(Inlined
Compiled Code))
at
au.gov.centrelink.jfs.proxy.ServiceClientMetaClass.intercept(ServiceClientMetaClass.java(Compiled
Code))
at
au.gov.centrelink.jfs.proxy.impl.AbstractCustomerServiceClient$$EnhancerByCGLIB$$cdba8c46.doRouting(<generated>(Compiled
Code))
at
au.gov.centrelink.jfs.proxy.ServiceClientMetaClass.intercept(ServiceClientMetaClass.java(Compiled
Code))
at
au.gov.centrelink.jfs.proxy.impl.AbstractCustomerServiceClient$$EnhancerByCGLIB$$cdba8c46.retrieveEmailAddress(<generated>(Compiled
Code))
at
au.gov.centrelink.jsf.pst.core.CDCCustomerEmailAddressTester2.execute(CDCCustomerEmailAddressTester2.java(Compiled
Code))
at
org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java(Inlined
Compiled Code))
at
org.apache.struts.action.RequestProcessor.process(RequestProcessor.java(Compiled
Code))
at
org.apache.struts.action.ActionServlet.process(ActionServlet.java(Inlined
Compiled Code))
at
org.apache.struts.action.ActionServlet.doPost(ActionServlet.java(Compiled
Code))
at
javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled Code))
at
javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled Code))
at
com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java(Compiled
Code))
at
com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java(Compiled
Code))
at
com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java(Compiled
Code))
at
com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java(Compiled
Code))
at
com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java(Compiled
Code))
at
com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.forward(WebAppRequestDispatcher.java(Compiled
Code))
at
org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java(Compiled
Code))
at
org.apache.struts.action.RequestProcessor.processForwardConfig(RequestProcessor.java(Compiled
Code))
at
org.apache.struts.action.RequestProcessor.process(RequestProcessor.java(Compiled
Code))
at
org.apache.struts.action.ActionServlet.process(ActionServlet.java(Inlined
Compiled Code))
at
org.apache.struts.action.ActionServlet.doPost(ActionServlet.java(Compiled
Code))
at
javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled Code))
at
javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled Code))
at
com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java(Compiled
Code))
at
com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java(Compiled
Code))
at
com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java(Compiled
Code))
at
au.gov.centrelink.itsecurity.acs.web.SecureRequestFilter.doFilter(SecureRequestFilter.java(Compiled
Code))
at
com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java(Compiled
Code))
at
com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java(Compiled
Code))
at
au.gov.centrelink.jaf.liser.api.WebRequestLiser.doFilter(WebRequestLiser.java(Compiled
Code))
at
au.gov.centrelink.jaf.interceptor.core.JAFFilter.doFilter(JAFFilter.java(Compiled
Code))
at
com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java(Compiled
Code))
at
com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java(Compiled
Code))
at
com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java(Compiled
Code))
at
com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java(Compiled
Code))
at
com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java(Compiled
Code))
at
com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java(Compiled
Code))
at
com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java(Compiled
Code))
at
com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java(Compiled
Code))
at
com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java(Compiled
Code))
at
com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java(Compiled
Code))
at
com.ibm.ws.tcp.channel.impl.WorkQueueManager.requestComplete(WorkQueueManager.java(Compiled
Code))
at
com.ibm.ws.tcp.channel.impl.WorkQueueManager.attemptIO(WorkQueueManager.java(Compiled
Code))
at
com.ibm.ws.tcp.channel.impl.WorkQueueManager.workerRun(WorkQueueManager.java(Compiled
Code))
at
com.ibm.ws.tcp.channel.impl.WorkQueueManager$Worker.run(WorkQueueManager.java(Compiled
Code))
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java(Compiled
Code))
Caused by: org.apache.axis2.AxisFault:
com.ctc.wstx.exc.WstxUnexpectedCharException: Unexpected character 'E'
(code 69) in prolog; expected '<'
at [row,col {unknown-source}]: [1,1]
at
org.apache.axis2.transport.TransportUtils.createSOAPMessage(TransportUtils.java(Compiled
Code))
at
org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java(Compiled
Code))
at
org.apache.axis2.description.OutInAxisOperationClient.execute(OutInAxisOperation.java(Compiled
Code))
at
au.gov.centrelink.jfs.isislocation.service.IsisLocationStub.envLookup(IsisLocationStub.java(Compiled
Code))
at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java(Compiled
Code))
at java.lang.reflect.Method.invoke(Method.java(Compiled Code))
at
au.gov.centrelink.jfs.proxy.ServiceClientMetaClass.intercept(ServiceClientMetaClass.java(Compiled
Code))
at
au.gov.centrelink.jfs.proxy.impl.AbstractDirectServiceClient$$EnhancerByCGLIB$$b007db1b.envLookup(<generated>(Compiled
Code))
... 52 more
Caused by: org.apache.axiom.om.OMException:
com.ctc.wstx.exc.WstxUnexpectedCharException: Unexpected character 'E'
(code 69) in prolog; expected '<'
at [row,col {unknown-source}]: [1,1]
at
org.apache.axiom.om.impl.builder.StAXOMBuilder.next(StAXOMBuilder.java(Compiled
Code))
at
org.apache.axiom.soap.impl.builder.StAXSOAPModelBuilder.getSOAPEnvelope(StAXSOAPModelBuilder.java(Inlined
Compiled Code))
at
org.apache.axiom.soap.impl.builder.StAXSOAPModelBuilder.<init>(StAXSOAPModelBuilder.java(Inlined
Compiled Code))
at
org.apache.axis2.builder.BuilderUtil.getSOAPBuilder(BuilderUtil.java(Inlined
Compiled Code))
at
org.apache.axis2.transport.TransportUtils.createSOAPMessage(TransportUtils.java(Compiled
Code))
... 61 more
Caused by: com.ctc.wstx.exc.WstxUnexpectedCharException: Unexpected
character 'E' (code 69) in prolog; expected '<'
at [row,col {unknown-source}]: [1,1]
at
com.ctc.wstx.sr.StreamScanner.throwUnexpectedChar(StreamScanner.java:623)
at
com.ctc.wstx.sr.BasicStreamReader.nextFromProlog(BasicStreamReader.java(Compiled
Code))
at
com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java(Compiled
Code))
... 66 more
It is strange that this exception occurs only once given that the servlet
returns essentially the same SOAP response every time.
Has anyone encountered this error before in a stress testing scenario, or
has any idea what might cause it to occur intermittently?
Thanks for your ongoing assistance,
Greg
**********************************************************************
IMPORTANT: This e-mail is intended for the use of the addressee and may
contain information that is confidential, commercially valuable or subject to
legal or parliamentary privilege. If you are not the intended recipient you
are notified that any review, re-transmission, disclosure, use or dissemination
of this communication is strictly prohibited by several Commonwealth Acts of
Parliament. If you have received this communication in error please notify the
sender immediately and delete all copies of this transmission together with any
attachments.
**********************************************************************
