subject:"Performance degrades on tomcat7 for the same runs of a sample performance 60 user test"

Re: Performance degrades on tomcat7 for the same runs of a sample performance 60 user test

2015-11-09 Thread Christopher Schultz

Dimple,

On 11/9/15 3:46 AM, dimple ranka wrote:
> Some help is really appreciated, atleast it will be good to hear if some
> one else is also facing slowness with the tomcat7 server.
> 
> I did some more investigation here and as mentioned earlier saw the
> slowness with tomcat7 is more reproducible when there are some
> cryptographic signature verifications.

I was going to ask about the crypto. One of the only threads actually
doing anything in the thread dump you showed was showing a TLS
handshake. Java's crypto is not very fast compared to native crypto. IF
you are handling TLS from Tomcat, you might want to consider a faster
solution if you do a lot of transactions.

If you have a shortage of entropy on the server, generating new crypto
keys can really slow things down. That could certainly explain the
"first run is fast, later runs are slower" observation. But this has
absolutely nothing to do with Tomcat -- it's all within the JVM so you
shouldn't have noticed a change with a Tomcat upgrade.

> Turning on the StuckThreadDetectionValve with 1 second threshold as
> shown below revealed that there are warning logs with threads stuck
> showing up in the next run of the test. This clearly shows that the
> slowness is due to the threads getting into the blocked state. More
> are in the signature verification but few are in other areas too.
> 
>   className="org.apache.catalina.valves.StuckThreadDetectionValve"
> threshold="1" />
> 
> NOTE - the above test was performed on a low end windows machine. Since the
> granularity of the valve threshold is in seconds was finding it hard to
> reproduce the valve to spit out some logs on the high end server. But the
> test shows that in the second run of the performance test which was run on
> another machine, only then i see the warning logs on the tomcat server.
>
> [snip]
>

> WARNING: Thread "http-nio-8443-exec-55" (id=83) has been active for 1,397
> milliseconds (since 11/7/15 11:06 AM) to serve the same request for
> https://10.55.198.52:8443/Signature/authenticate and may be stuck
> (configured threshold for this StuckThreadDetectionValve is 1 seconds).
> There is/are 1 thread(s) in total that are monitored by this Valve and may
> be stuck.
> java.lang.Throwable
> at sun.security.ec.ECDSASignature.verifySignedDigest(Native Method)
> at sun.security.ec.ECDSASignature.engineVerify(ECDSASignature.java:309)
> at java.security.Signature$Delegate.engineVerify(Signature.java:1192)
> at java.security.Signature.verify(Signature.java:626)
> at signature.SignatureEndPoint.verifyECDSASignature(Unknown Source)
> at signature.SignatureEndPoint.verifySignature(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1511)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1442)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1391)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1381)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at
>

Re: Performance degrades on tomcat7 for the same runs of a sample performance 60 user test

2015-11-09 Thread Dimple Ranka

On Mon, Nov 9, 2015 at 8:34 PM, Christopher Schultz <
ch...@christopherschultz.net> wrote:

> Dimple,
>
> On 11/9/15 3:46 AM, dimple ranka wrote:
> > Some help is really appreciated, atleast it will be good to hear if some
> > one else is also facing slowness with the tomcat7 server.
> >
> > I did some more investigation here and as mentioned earlier saw the
> > slowness with tomcat7 is more reproducible when there are some
> > cryptographic signature verifications.
>
> I was going to ask about the crypto. One of the only threads actually
> doing anything in the thread dump you showed was showing a TLS
> handshake. Java's crypto is not very fast compared to native crypto. IF
> you are handling TLS from Tomcat, you might want to consider a faster
> solution if you do a lot of transactions.
>
> If you have a shortage of entropy on the server, generating new crypto
> keys can really slow things down. That could certainly explain the
> "first run is fast, later runs are slower" observation. But this has
> absolutely nothing to do with Tomcat -- it's all within the JVM so you
> shouldn't have noticed a change with a Tomcat upgrade.
>
> ==> We have configured the secure random source as below to avoid the
entropy issue -
securerandom.source=file:/dev/./urandom

> Turning on the StuckThreadDetectionValve with 1 second threshold as
> > shown below revealed that there are warning logs with threads stuck
> > showing up in the next run of the test. This clearly shows that the
> > slowness is due to the threads getting into the blocked state. More
> > are in the signature verification but few are in other areas too.
> >
> >   > className="org.apache.catalina.valves.StuckThreadDetectionValve"
> > threshold="1" />
> >
> > NOTE - the above test was performed on a low end windows machine. Since
> the
> > granularity of the valve threshold is in seconds was finding it hard to
> > reproduce the valve to spit out some logs on the high end server. But the
> > test shows that in the second run of the performance test which was run
> on
> > another machine, only then i see the warning logs on the tomcat server.
> >
> > [snip]
> >
>
> > WARNING: Thread "http-nio-8443-exec-55" (id=83) has been active for 1,397
> > milliseconds (since 11/7/15 11:06 AM) to serve the same request for
> > https://10.55.198.52:8443/Signature/authenticate and may be stuck
> > (configured threshold for this StuckThreadDetectionValve is 1 seconds).
> > There is/are 1 thread(s) in total that are monitored by this Valve and
> may
> > be stuck.
> > java.lang.Throwable
> > at sun.security.ec.ECDSASignature.verifySignedDigest(Native Method)
> > at sun.security.ec.ECDSASignature.engineVerify(ECDSASignature.java:309)
> > at java.security.Signature$Delegate.engineVerify(Signature.java:1192)
> > at java.security.Signature.verify(Signature.java:626)
> > at signature.SignatureEndPoint.verifyECDSASignature(Unknown Source)
> > at signature.SignatureEndPoint.verifySignature(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at
> >
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> > at
> >
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
> > at
> >
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> > at
> >
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
> > at
> >
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> > at
> >
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> > at
> >
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> > at
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1511)
> > at
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1442)
> > at
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1391)
> > at
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1381)
> > at
> >
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> > at
> >
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538)
> > at
> >
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716)
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
> > at
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
>

Re: Performance degrades on tomcat7 for the same runs of a sample performance 60 user test

2015-11-09 Thread dimple ranka

Hi all,

Some help is really appreciated, atleast it will be good to hear if some
one else is also facing slowness with the tomcat7 server.

I did some more investigation here and as mentioned earlier saw the
slowness with tomcat7 is more reproducible when there are some
cryptographic signature verifications.  Turning on the
StuckThreadDetectionValve with 1 second threshold as shown below revealed
that there are warning logs with threads stuck showing up in the next run
of the test. This clearly shows that the slowness is due to the threads
getting into the blocked state. More are in the signature verification but
few are in other areas too.

 


NOTE - the above test was performed on a low end windows machine. Since the
granularity of the valve threshold is in seconds was finding it hard to
reproduce the valve to spit out some logs on the high end server. But the
test shows that in the second run of the performance test which was run on
another machine, only then i see the warning logs on the tomcat server.

CATALINA.out contents showing the stuck threads ===>

Nov 07, 2015 11:01:13 AM org.apache.catalina.startup.HostConfig undeploy
INFO: Undeploying context [/host-manager]
Nov 07, 2015 11:01:13 AM org.apache.catalina.startup.HostConfig undeploy
INFO: Undeploying context [/docs]
Nov 07, 2015 11:01:13 AM org.apache.catalina.startup.HostConfig undeploy
INFO: Undeploying context []
Nov 07, 2015 11:01:13 AM org.apache.catalina.startup.HostConfig undeploy
INFO: Undeploying context [/Signature]
Nov 07, 2015 11:01:13 AM org.apache.catalina.startup.HostConfig undeploy
INFO: Undeploying context [/examples]
Nov 07, 2015 11:01:13 AM org.apache.catalina.startup.HostConfig undeploy
INFO: Undeploying context [/manager]
Nov 07, 2015 11:01:13 AM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive
D:\apache-tomcat-7.0.64\webapps\Signature.war
Nov 07, 2015 11:01:16 AM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable
debug logging for this logger for a complete list of JARs that were scanned
but no TLDs were found in them. Skipping unneeded JARs during scanning can
improve startup time and JSP compilation time.
Nov 07, 2015 11:01:17 AM
com.sun.jersey.api.core.servlet.WebAppResourceConfig init
INFO: Scanning for root resource and provider classes in the Web app
resource paths:
  /WEB-INF/lib
  /WEB-INF/classes
Nov 07, 2015 11:01:18 AM com.sun.jersey.api.core.ScanningResourceConfig
logClasses
INFO: Root resource classes found:
  class signature.NullEndPoint
  class signature.SignatureEndPoint
Nov 07, 2015 11:01:18 AM com.sun.jersey.api.core.ScanningResourceConfig
logClasses
INFO: Provider classes found:
  class org.codehaus.jackson.jaxrs.JsonMappingExceptionMapper
  class org.codehaus.jackson.jaxrs.JacksonJaxbJsonProvider
  class org.codehaus.jackson.jaxrs.JacksonJsonProvider
  class org.codehaus.jackson.jaxrs.JsonParseExceptionMapper
Nov 07, 2015 11:01:18 AM
com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.17 01/17/2013 03:31
PM'
Nov 07, 2015 11:01:19 AM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deployment of web application archive
D:\apache-tomcat-7.0.64\webapps\Signature.war has finished in 5,871 ms
Nov 07, 2015 11:01:19 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deploying web application directory
D:\apache-tomcat-7.0.64\webapps\docs
Nov 07, 2015 11:01:19 AM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable
debug logging for this logger for a complete list of JARs that were scanned
but no TLDs were found in them. Skipping unneeded JARs during scanning can
improve startup time and JSP compilation time.
Nov 07, 2015 11:01:19 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deployment of web application directory
D:\apache-tomcat-7.0.64\webapps\docs has finished in 140 ms
Nov 07, 2015 11:01:19 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deploying web application directory
D:\apache-tomcat-7.0.64\webapps\examples
Nov 07, 2015 11:01:19 AM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable
debug logging for this logger for a complete list of JARs that were scanned
but no TLDs were found in them. Skipping unneeded JARs during scanning can
improve startup time and JSP compilation time.
Nov 07, 2015 11:01:19 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deployment of web application directory
D:\apache-tomcat-7.0.64\webapps\examples has finished in 445 ms
Nov 07, 2015 11:01:19 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deploying web application directory
D:\apache-tomcat-7.0.64\webapps\host-manager
Nov 07, 2015 11:01:19 AM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for

Re: Performance degrades on tomcat7 for the same runs of a sample performance 60 user test

2015-11-04 Thread dimple ranka

Looks like attachments are not allowed so only sharing the contents of
threadDump3.out

2015-11-04 22:55:58
Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.79-b02 mixed mode):

"http-nio-8443-exec-147" daemon prio=10 tid=0x7f5c1c005000 nid=0x6241
runnable [0x7f5bd30ef000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
- locked <0xfc2d9498> (a java.lang.Object)
at
org.apache.tomcat.util.net.SecureNioChannel.read(SecureNioChannel.java:443)
at
org.apache.coyote.http11.InternalNioInputBuffer.readSocket(InternalNioInputBuffer.java:436)
at
org.apache.coyote.http11.InternalNioInputBuffer.fill(InternalNioInputBuffer.java:795)
at
org.apache.coyote.http11.InternalNioInputBuffer.parseRequestLine(InternalNioInputBuffer.java:227)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:993)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1760)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1719)
- locked <0x875077f0> (a
org.apache.tomcat.util.net.SecureNioChannel)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
- <0x88a613f0> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"http-nio-8443-exec-146" daemon prio=10 tid=0x7f5c18020800 nid=0x6240
waiting on condition [0x7f5c6e271000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x86b7a088> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
- None

"http-nio-8443-exec-145" daemon prio=10 tid=0x7f5c18037000 nid=0x623f
waiting on condition [0x7f5c6e372000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x86b7a088> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
- None

"http-nio-8443-exec-144" daemon prio=10 tid=0x7f5c1802f800 nid=0x623e
waiting on condition [0x7f5c6c89c000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x86b7a088> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at

Re: Performance degrades on tomcat7 for the same runs of a sample performance 60 user test

2015-11-04 Thread dimple ranka

Hi Chris,

I captured 3 threadumps with the below requests/second when the server is
already slow. We were using BIO before and had observed slowness even with
that. We switched to using NIO since it seemed a recommendation for
production environment.
--
*WITH NIO*
*--*
*RUN1 (threadDump1.out)*
Requests/second = 18849.1606557377

*RUN2 **(threadDump2.out)*
Requests/second = 18943.43606557377

*RUN3 **(threadDump3.out)*
Requests/second = 18894.21241830065
--

The issue we are trying to address why does performance degrade by 50% with
subsequent runs.

Thanks,
Dimple

On Thu, Nov 5, 2015 at 1:31 AM, Christopher Schultz <
ch...@christopherschultz.net> wrote:

> Dimple,
>
> On 11/3/15 11:24 PM, dimple ranka wrote:
> > Also forgot to mention that setting maxKeepAliveRequests to -1 did not
> > help. As expected from the connector documentation the default value for
> > this attribute is 100 and we have a 60 user test set up.
> >
> > On Wed, Nov 4, 2015 at 8:18 AM, dimple ranka 
> wrote:
> >
> >> Hi Mark,
> >>
> >> Another observation to be noted here is that system CPU usage shoots up
> >> for subsequent runs, especially for later runs.
> >>
> >> We have been looking into this issue for couple of weeks now and it is
> >> clear in the same environment with the same setup it doesn't reproduce
> on
> >> tomcat6. The moment we deploy the web application in a tomcat7 container
> >> the slowness with subsequent runs shows up.
>
> Can you take some thread dumps to find out what Tomcat is doing? High
> CPU usage and lower request throughput sounds like a Poller problem
> since you are using NIO.
>
> -chris
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Performance degrades on tomcat7 for the same runs of a sample performance 60 user test

2015-11-04 Thread Christopher Schultz

Dimple,

On 11/3/15 11:24 PM, dimple ranka wrote:
> Also forgot to mention that setting maxKeepAliveRequests to -1 did not
> help. As expected from the connector documentation the default value for
> this attribute is 100 and we have a 60 user test set up.
> 
> On Wed, Nov 4, 2015 at 8:18 AM, dimple ranka  wrote:
> 
>> Hi Mark,
>>
>> Another observation to be noted here is that system CPU usage shoots up
>> for subsequent runs, especially for later runs.
>>
>> We have been looking into this issue for couple of weeks now and it is
>> clear in the same environment with the same setup it doesn't reproduce on
>> tomcat6. The moment we deploy the web application in a tomcat7 container
>> the slowness with subsequent runs shows up.

Can you take some thread dumps to find out what Tomcat is doing? High
CPU usage and lower request throughput sounds like a Poller problem
since you are using NIO.

-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Performance degrades on tomcat7 for the same runs of a sample performance 60 user test

2015-11-03 Thread Mark Thomas

On 03/11/2015 06:17, dimple ranka wrote:

> ##
> On tomcat7 number of requests fullfilled per second
> ##
> * RUN1 ** ==> **  38128.97704918033 runs/sec*
>  *RUN2 ==> **  19040.35947712418 runs/sec*
>  *RUN3 ==> ** 19043.7908496732**  runs/sec*
> * RUN4 ==> ** 19001.71568627451 runs/sec*
> ##

Every performance test I have ever done with Tomcat, the first run has
had poorer performance while the system warms up (threads started,
caches filled, session ID generators initialised etc.) and subsequent
tests have performed better.

The results above suggest something about the test is broken.

First of all, thank you for the detailed information. It helps a lot.

Your server side test code won't run on Tomcat without additional
libraries and configuration that you haven't specified. That makes it
harder for people to reproduce your results. I'd recommend using a
simple servlet for testing in the first instance and only when any
issues have been resolved, move to your test.

20k requests/s is rather low. I'd expect those sorts of numbers from a
single threaded test on reasonable hardware. With 8*2 threads to play
with on the server I'd expect that number to be a lot higher.

My experience of JMeter is that the results are unreliable as you
approach the point where the overhead per request for JMeter approaches
the overhead per request for Tomcat. You would probably get better
results if you switched to a lower overhead test client. I tend to use
ab for these sorts of tests.

It is not clear from your original e-mail if the test client is running
on the same machine as the server or not.
If not, the network can have a significant impact. With this test a
gigabit network should be fine but we have seen network saturation in
performance tests for larger static files so do keep an eye on the
network utilisation just to be safe.

The higher the throughput, the greater the impact of HTTP keep-alive. I
have frequently observed strange test results caused by the system
running out of free ports due to the churn rate of connections. netstat
is your friend. I'd recommend exploring the impact of different settings
for maxKeepAliveRequests. You almost certainly want something
significantly higher than the default of 100. A quick test would be to
set it to -1 and see what impact that has.

HTH,

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Performance degrades on tomcat7 for the same runs of a sample performance 60 user test

2015-11-03 Thread dimple ranka

Hi Mark,

The test client is running on another machine.

The server side test code is written using jersey 1.17.

Will try out with playing with maxKeepAliveRequests.

My only concern is that if jmeter is doing spoofy things then why it doesnt
show up on tomcat6.

Thanks,
Dimple

On Tue, Nov 3, 2015 at 2:27 PM, Mark Thomas  wrote:

> On 03/11/2015 06:17, dimple ranka wrote:
>
> 
>
> > ##
> > On tomcat7 number of requests fullfilled per second
> > ##
> > * RUN1 ** ==> **  38128.97704918033 runs/sec*
> >  *RUN2 ==> **  19040.35947712418 runs/sec*
> >  *RUN3 ==> ** 19043.7908496732**  runs/sec*
> > * RUN4 ==> ** 19001.71568627451 runs/sec*
> > ##
>
> Every performance test I have ever done with Tomcat, the first run has
> had poorer performance while the system warms up (threads started,
> caches filled, session ID generators initialised etc.) and subsequent
> tests have performed better.
>
> The results above suggest something about the test is broken.
>
> First of all, thank you for the detailed information. It helps a lot.
>
> Your server side test code won't run on Tomcat without additional
> libraries and configuration that you haven't specified. That makes it
> harder for people to reproduce your results. I'd recommend using a
> simple servlet for testing in the first instance and only when any
> issues have been resolved, move to your test.
>
> 20k requests/s is rather low. I'd expect those sorts of numbers from a
> single threaded test on reasonable hardware. With 8*2 threads to play
> with on the server I'd expect that number to be a lot higher.
>
> My experience of JMeter is that the results are unreliable as you
> approach the point where the overhead per request for JMeter approaches
> the overhead per request for Tomcat. You would probably get better
> results if you switched to a lower overhead test client. I tend to use
> ab for these sorts of tests.
>
> It is not clear from your original e-mail if the test client is running
> on the same machine as the server or not.
> If not, the network can have a significant impact. With this test a
> gigabit network should be fine but we have seen network saturation in
> performance tests for larger static files so do keep an eye on the
> network utilisation just to be safe.
>
> The higher the throughput, the greater the impact of HTTP keep-alive. I
> have frequently observed strange test results caused by the system
> running out of free ports due to the churn rate of connections. netstat
> is your friend. I'd recommend exploring the impact of different settings
> for maxKeepAliveRequests. You almost certainly want something
> significantly higher than the default of 100. A quick test would be to
> set it to -1 and see what impact that has.
>
> HTH,
>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Performance degrades on tomcat7 for the same runs of a sample performance 60 user test

2015-11-03 Thread dimple ranka

Hi Mark,

Another observation to be noted here is that system CPU usage shoots up for
subsequent runs, especially for later runs.

We have been looking into this issue for couple of weeks now and it is
clear in the same environment with the same setup it doesn't reproduce on
tomcat6. The moment we deploy the web application in a tomcat7 container
the slowness with subsequent runs shows up.

Thanks,
Dimple

On Tue, Nov 3, 2015 at 2:36 PM, dimple ranka  wrote:

> Hi Mark,
>
> The test client is running on another machine.
>
> The server side test code is written using jersey 1.17.
>
> Will try out with playing with maxKeepAliveRequests.
>
> My only concern is that if jmeter is doing spoofy things then why it
> doesnt show up on tomcat6.
>
> Thanks,
> Dimple
>
> On Tue, Nov 3, 2015 at 2:27 PM, Mark Thomas  wrote:
>
>> On 03/11/2015 06:17, dimple ranka wrote:
>>
>> 
>>
>> > ##
>> > On tomcat7 number of requests fullfilled per second
>> > ##
>> > * RUN1 ** ==> **  38128.97704918033 runs/sec*
>> >  *RUN2 ==> **  19040.35947712418 runs/sec*
>> >  *RUN3 ==> ** 19043.7908496732**  runs/sec*
>> > * RUN4 ==> ** 19001.71568627451 runs/sec*
>> > ##
>>
>> Every performance test I have ever done with Tomcat, the first run has
>> had poorer performance while the system warms up (threads started,
>> caches filled, session ID generators initialised etc.) and subsequent
>> tests have performed better.
>>
>> The results above suggest something about the test is broken.
>>
>> First of all, thank you for the detailed information. It helps a lot.
>>
>> Your server side test code won't run on Tomcat without additional
>> libraries and configuration that you haven't specified. That makes it
>> harder for people to reproduce your results. I'd recommend using a
>> simple servlet for testing in the first instance and only when any
>> issues have been resolved, move to your test.
>>
>> 20k requests/s is rather low. I'd expect those sorts of numbers from a
>> single threaded test on reasonable hardware. With 8*2 threads to play
>> with on the server I'd expect that number to be a lot higher.
>>
>> My experience of JMeter is that the results are unreliable as you
>> approach the point where the overhead per request for JMeter approaches
>> the overhead per request for Tomcat. You would probably get better
>> results if you switched to a lower overhead test client. I tend to use
>> ab for these sorts of tests.
>>
>> It is not clear from your original e-mail if the test client is running
>> on the same machine as the server or not.
>> If not, the network can have a significant impact. With this test a
>> gigabit network should be fine but we have seen network saturation in
>> performance tests for larger static files so do keep an eye on the
>> network utilisation just to be safe.
>>
>> The higher the throughput, the greater the impact of HTTP keep-alive. I
>> have frequently observed strange test results caused by the system
>> running out of free ports due to the churn rate of connections. netstat
>> is your friend. I'd recommend exploring the impact of different settings
>> for maxKeepAliveRequests. You almost certainly want something
>> significantly higher than the default of 100. A quick test would be to
>> set it to -1 and see what impact that has.
>>
>> HTH,
>>
>> Mark
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>>
>

Re: Performance degrades on tomcat7 for the same runs of a sample performance 60 user test

2015-11-03 Thread dimple ranka

Also forgot to mention that setting maxKeepAliveRequests to -1 did not
help. As expected from the connector documentation the default value for
this attribute is 100 and we have a 60 user test set up.

On Wed, Nov 4, 2015 at 8:18 AM, dimple ranka  wrote:

> Hi Mark,
>
> Another observation to be noted here is that system CPU usage shoots up
> for subsequent runs, especially for later runs.
>
> We have been looking into this issue for couple of weeks now and it is
> clear in the same environment with the same setup it doesn't reproduce on
> tomcat6. The moment we deploy the web application in a tomcat7 container
> the slowness with subsequent runs shows up.
>
> Thanks,
> Dimple
>
> On Tue, Nov 3, 2015 at 2:36 PM, dimple ranka 
> wrote:
>
>> Hi Mark,
>>
>> The test client is running on another machine.
>>
>> The server side test code is written using jersey 1.17.
>>
>> Will try out with playing with maxKeepAliveRequests.
>>
>> My only concern is that if jmeter is doing spoofy things then why it
>> doesnt show up on tomcat6.
>>
>> Thanks,
>> Dimple
>>
>> On Tue, Nov 3, 2015 at 2:27 PM, Mark Thomas  wrote:
>>
>>> On 03/11/2015 06:17, dimple ranka wrote:
>>>
>>> 
>>>
>>> > ##
>>> > On tomcat7 number of requests fullfilled per second
>>> > ##
>>> > * RUN1 ** ==> **  38128.97704918033 runs/sec*
>>> >  *RUN2 ==> **  19040.35947712418 runs/sec*
>>> >  *RUN3 ==> ** 19043.7908496732**  runs/sec*
>>> > * RUN4 ==> ** 19001.71568627451 runs/sec*
>>> > ##
>>>
>>> Every performance test I have ever done with Tomcat, the first run has
>>> had poorer performance while the system warms up (threads started,
>>> caches filled, session ID generators initialised etc.) and subsequent
>>> tests have performed better.
>>>
>>> The results above suggest something about the test is broken.
>>>
>>> First of all, thank you for the detailed information. It helps a lot.
>>>
>>> Your server side test code won't run on Tomcat without additional
>>> libraries and configuration that you haven't specified. That makes it
>>> harder for people to reproduce your results. I'd recommend using a
>>> simple servlet for testing in the first instance and only when any
>>> issues have been resolved, move to your test.
>>>
>>> 20k requests/s is rather low. I'd expect those sorts of numbers from a
>>> single threaded test on reasonable hardware. With 8*2 threads to play
>>> with on the server I'd expect that number to be a lot higher.
>>>
>>> My experience of JMeter is that the results are unreliable as you
>>> approach the point where the overhead per request for JMeter approaches
>>> the overhead per request for Tomcat. You would probably get better
>>> results if you switched to a lower overhead test client. I tend to use
>>> ab for these sorts of tests.
>>>
>>> It is not clear from your original e-mail if the test client is running
>>> on the same machine as the server or not.
>>> If not, the network can have a significant impact. With this test a
>>> gigabit network should be fine but we have seen network saturation in
>>> performance tests for larger static files so do keep an eye on the
>>> network utilisation just to be safe.
>>>
>>> The higher the throughput, the greater the impact of HTTP keep-alive. I
>>> have frequently observed strange test results caused by the system
>>> running out of free ports due to the churn rate of connections. netstat
>>> is your friend. I'd recommend exploring the impact of different settings
>>> for maxKeepAliveRequests. You almost certainly want something
>>> significantly higher than the default of 100. A quick test would be to
>>> set it to -1 and see what impact that has.
>>>
>>> HTH,
>>>
>>> Mark
>>>
>>> -
>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>>
>>>
>>
>

Performance degrades on tomcat7 for the same runs of a sample performance 60 user test

2015-11-02 Thread dimple ranka

Hi all,

I am doing some performance data collection on tomcat7 on a 8 processor
high end machine with CentOS release 6.7 (Final) operating system and i am
observing slowness with subsequent runs of the same jmeter test. Here are
the version and specfication details followed by the endpoint source and
the problem statement finally.

##
Tomcat7 Version  ==>
##
Server version: Apache Tomcat/7.0.64
Server built:   Aug 19 2015 17:18:06 UTC
Server number:*  7.0.64.0*
OS Name:Linux
OS Version: 2.6.32-573.3.1.el6.x86_64
Architecture:   amd64
JVM Version:1.7.0_79-b15
JVM Vendor: Oracle Corporation
##


##
Connector configurations
##

##


##
Machine Specifications
##
Architecture:  x86_64
CPU op-mode(s):32-bit, 64-bit
Byte Order:Little Endian
CPU(s):8
On-line CPU(s) list:   0-7
Thread(s) per core:2
Core(s) per socket:4
Socket(s): 1
NUMA node(s):  1
Vendor ID: GenuineIntel
CPU family:6
Model: 58
Stepping:  9
CPU MHz:   1600.000
BogoMIPS:  7000.55
Virtualization:VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache:  256K
L3 cache:  8192K
NUMA node0 CPU(s): 0-7
##


##
Source code of the endpoint being performance tested
##
import javax.ws.rs.Consumes;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;


@Path("/null")
public class NullEndPoint {
public static final String UTF_8 = "UTF-8";
public static final String JSON_CONTENT_TYPE = MediaType.APPLICATION_JSON +
"; charset=" + UTF_8;
@POST
@Consumes(JSON_CONTENT_TYPE)
@Produces(JSON_CONTENT_TYPE)
public Response verify(String params) {
return Response.ok("{ \"status\":\"success\"}").build();
}

}
##


##
Jmeter Test Plan - 60 users test for 1200 seconds duration
##


  

  
  false
  false
  

  
  


  
continue
${__P(users,60)}
0
10
0
30
10
1
${__P(duration,1200)}
5

  false
  -1

  
  

  

  Content-Type
  application/json

  



  

  userNameUUID
  user_${__UUID
  =


  host
  ${__P(myhost,XXX.XX.XX.XX)}
  =


  port
  8443
  =


  protocol
  https
  =


  pathPrefix
  /null
  =

  



  true
  

  
false
{
}
=
  

  
  ${host}
  ${port}
  
  
  ${protocol}
  
  ${pathPrefix}
  POST
  true
  false
  true
  false
  false
  



  false
  
saveConfig

  true
  true
  true
  true
  true
  true
  true
  true
  true
  false
  true
  true
  false
  false
  false
  false
  false
  false
  false

false
  0
  true

  
  


  

  

##


##
On tomcat7 number of requests fullfilled per second
##
* RUN1 ** ==> **  38128.97704918033 runs/sec*
 *RUN2 ==> **  19040.35947712418 runs/sec*
 *RUN3 ==> ** 19043.7908496732**  runs/sec*
* RUN4 ==> ** 19001.71568627451 runs/sec*
##

##
*PROBLEM*
##
It can be clearly seen that the

Re: Performance degrades on tomcat7 for the same runs of a sample performance 60 user test

Re: Performance degrades on tomcat7 for the same runs of a sample performance 60 user test

Re: Performance degrades on tomcat7 for the same runs of a sample performance 60 user test

Re: Performance degrades on tomcat7 for the same runs of a sample performance 60 user test

Re: Performance degrades on tomcat7 for the same runs of a sample performance 60 user test

Re: Performance degrades on tomcat7 for the same runs of a sample performance 60 user test

Re: Performance degrades on tomcat7 for the same runs of a sample performance 60 user test

Re: Performance degrades on tomcat7 for the same runs of a sample performance 60 user test

Re: Performance degrades on tomcat7 for the same runs of a sample performance 60 user test

Re: Performance degrades on tomcat7 for the same runs of a sample performance 60 user test

Performance degrades on tomcat7 for the same runs of a sample performance 60 user test

11 matches

Site Navigation

Mail list logo

Footer information