We had seen similar behavior when we used the SelectChannelConnector. In our situation, we found that using the BlockingChannelConnector was a better fit. That might be worth a try in your situation.
On Wed, Apr 18, 2012 at 4:41 PM, Devon Lazarus <[email protected]>wrote: > Hi,**** > > ** ** > > I’ve been trying to get to the bottom of what I believe is a performance > problem in a Jetty application. Unfortunately, I’ve hit a road block and > could use some help. Googling hasn’t shed any light on the issue. Neither > has stackoverflow.**** > > ** ** > > Basically, I’m seeing very high CPU load without a lot of throughput. To > quantify that, we have a c1.medium (dual core) running Ubuntu 10.04LTS in > EC2 with Jetty 7.6.2 configured at 50 min/200 max threads, a single > acceptor, and the Oracle JDK 1.7_03 JVM configured with -mx2048m –ms512m, > and some GC tuning.**** > > ** ** > > We’re using a custom load generation tool to find the point at which the > server falls over. However, we’re hitting 90% CPU burn with loads well over > 2 while handling ~25 concurrent transactions/second (throughput was 63 tps > with ~400ms response times). That just doesn’t seem right (way too low).** > ** > > ** ** > > In profiling the JVM regardless of load (low or under siege), we can see > clearly that 99% of our CPU time is spent in one of two places:**** > > ** ** > > sun.nio.ch.SelectorImpl.select() at 66%**** > > sun.nio.ch.ServerSocketChannelImpl.accept() at 33%**** > > ** ** > > with all of that time being consumed by the acceptor/selector threads > dumped below in equal parts.**** > > ** ** > > In looking at the thread dump, we have many threads that look like this:** > ** > > ** ** > > "qtp665633643-88" - Thread t@88**** > > java.lang.Thread.State: TIMED_WAITING**** > > at sun.misc.Unsafe.park(Native Method)**** > > - parking to wait for <2ef1b67e> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)**** > > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)**** > > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > **** > > at > org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:337) > **** > > at > org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:517) > **** > > at > org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:39) > **** > > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:563) > **** > > at java.lang.Thread.run(Thread.java:722)**** > > ** ** > > Locked ownable synchronizers:**** > > - None**** > > ** ** > > And I see the acceptor/selectors like this:**** > > ** ** > > "qtp665633643-83 Selector0" - Thread t@83**** > > java.lang.Thread.State: RUNNABLE**** > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)** > ** > > at > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)**** > > at > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)**** > > at > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)**** > > - locked <7b18010e> (a sun.nio.ch.Util$2)**** > > - locked <23bb2f20> (a > java.util.Collections$UnmodifiableSet)**** > > - locked <7557d06a> (a sun.nio.ch.EPollSelectorImpl)**** > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)*** > * > > at > org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:564) > **** > > at > org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:285)** > ** > > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) > **** > > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) > **** > > at java.lang.Thread.run(Thread.java:722)**** > > ** ** > > Locked ownable synchronizers:**** > > - None**** > > ** ** > > "qtp1586177080-28 Acceptor0 [email protected]:80" - Thread > t@28**** > > java.lang.Thread.State: RUNNABLE**** > > at sun.nio.ch.ServerSocketChannelImpl.accept0(Native > Method)**** > > at > sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226) > **** > > - locked <58341b55> (a java.lang.Object)**** > > at > org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104) > **** > > at > org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933) > **** > > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) > **** > > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) > **** > > at java.lang.Thread.run(Thread.java:722)**** > > ** ** > > Locked ownable synchronizers:**** > > - None**** > > ** ** > > "qtp1586177080-27 Selector0" - Thread t@27**** > > java.lang.Thread.State: RUNNABLE**** > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)** > ** > > at > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)**** > > at > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)**** > > at > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)**** > > - locked <be7202b> (a sun.nio.ch.Util$2)**** > > - locked <2761d665> (a > java.util.Collections$UnmodifiableSet)**** > > - locked <4b49db96> (a sun.nio.ch.EPollSelectorImpl)**** > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)*** > * > > at > org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:564) > **** > > at > org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:285)** > ** > > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) > **** > > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) > **** > > at java.lang.Thread.run(Thread.java:722)**** > > ** ** > > Locked ownable synchronizers:**** > > - None**** > > ** ** > > Not a single thread in BLOCKED state.**** > > ** ** > > I know I only have a single acceptor configured when I have two > processors, but when I do configure two acceptors, I see the two acceptor > threads blocking each other (one RUNNABLE while the other BLOCKED and > switch back and forth after 10s of seconds).**** > > ** ** > > Any help is appreciated, and I apologize for the long email.**** > > ** ** > > Regards,**** > > ** ** > > Devon Lazarus, Internet Services Group**** > > Sonos, Inc.**** > > ** ** > > _______________________________________________ > jetty-users mailing list > [email protected] > https://dev.eclipse.org/mailman/listinfo/jetty-users > > -- Neil Bliss | Senior Engineer | 3Crowd Technologies, Inc. c: +1.408.838.9632 | o: +1.415.445.4500
_______________________________________________ jetty-users mailing list [email protected] https://dev.eclipse.org/mailman/listinfo/jetty-users
