First, and most important, your thread pool size is far too low for a production machine.
Some napkin math for you to work this out. maximum thread in thread pool = 16 + ((num of cpu cores) * 4) + (peak concurrent connections) Next, update your Java VM. Your JVM is officially expired and out of date. Would start with using official oracle JVMs, not the OpenJDKs series (at least in the Java 1.6 class). 1.6_29 fixed a number of JIT issues that caused problematic looping that caused a *wide* range of problems. 1.6_30 fixed an SSL regression introduced in u29 1.6_31 was a minor update (mainly for Timezone updates) 1.6_32 contained important networking and garbage collection bug fixes 1.6_33 had many updates, the only one impacting a server would be the Timezone updates. 1.6_34 had a bunch of networking and util/collections bug fixes 1.6_35 emergency patch for CVE-2012-4681 (not server affecting) and more Timezone updates 1.6_37 more timezone updates (and other non-server impacting updates) 1.6_38 networking fixes (for windows) and SSL close issue 1.6_39 more timezone updates 1.6_41 more timezone updates 1.6_43 announced end for Java 6, plus emergency patch for CVE-2013-1493 (not server affecting) 1.6_45 fix for Runtime.exec & RMI bug fixes along with (yup you guessed it) a Timezone update At least get past update 38. Also, if you are using SSL in production, it would be highly advised that you upgrade to Java 1.7, as many of the SSL fixes have not been put into Java 1.6. -- Joakim Erdfelt <[email protected]> webtide.com <http://www.webtide.com/> - intalio.com/jetty Expert advice, services and support from from the Jetty & CometD experts eclipse.org/jetty - cometd.org On Tue, Sep 30, 2014 at 1:44 PM, Jeff Smyth <[email protected]> wrote: > The Java Version is OpenJDK 1.6.0_27 > > The App is running on Heroku and uses Ubuntu 10.04 > > Jetty is running embedded, and the ExecutorThreadPool is being invoked > with the following parameters: > > corePoolSize=5 > > MaximumPoolSize=20 > > keepAliveTime=5000 ms > > The LinkedBlockingQueue that is passed to the ExecutorThreadPool has a > capacity of 20. > > > - Jeff > ------------------------------ > *From:* [email protected] <[email protected]> > on behalf of Joakim Erdfelt <[email protected]> > *Sent:* Tuesday, September 30, 2014 4:01 PM > *To:* JETTY user mailing list > *Subject:* Re: [jetty-users] Getting a > java.util.concurrent.RejectedExecutionException > > What specific version of Java 1.6? (some versions of Java 1.6 have > problems) > On what environment/platform/OS? (some platforms have shown problems) > Also, what are your threadpool settings? (low threadpool settings can > cause problems) > > > -- > Joakim Erdfelt <[email protected]> > webtide.com <http://www.webtide.com/> - intalio.com/jetty > Expert advice, services and support from from the Jetty & CometD experts > eclipse.org/jetty - cometd.org > > On Tue, Sep 30, 2014 at 12:46 PM, Jeff Smyth <[email protected]> wrote: > >> Jetty Users Group, >> >> I have recently been facing an issue with my Production Server which >> runs on Jetty. I hope this is the correct location to ask for >> assistance. The following is the error that is being thrown: >> >> app/web.4 >> <https://papertrailapp.com/events?centered_on_id=453512210627715097&q=program%3Aapp%2Fweb.4> >> : 2014-09-30 03:42:36,794 [pool-2-thread-1 Selector0] WARN >> org.eclipse.jetty.io.nio - Dispatched Failed! SCEP@5ebab0e9{l(/ >> 172.16.96.233:57773)<->r(/172.16.96.234:24302 >> ),s=-1,open=true,ishut=false,oshut=false,rb=false,wb=false,w=true,i=1r}-{AsyncHttpConnection@32b4c195,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=0},r=0} >> to >> org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager@45544e03 >> >> app/web.4 >> <https://papertrailapp.com/events?centered_on_id=453512210627715099&q=program%3Aapp%2Fweb.4> >> : 2014-09-30 03:42:36,794 [pool-2-thread-1 Selector0] WARN >> org.eclipse.jetty.util.thread.ExecutorThreadPool - >> app/web.4 >> <https://papertrailapp.com/events?centered_on_id=453512210627715100&q=program%3Aapp%2Fweb.4> >> : java.util.concurrent.RejectedExecutionException >> app/web.4 >> <https://papertrailapp.com/events?centered_on_id=453512210627715102&q=program%3Aapp%2Fweb.4> >> : at >> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1992) >> app/web.4 >> <https://papertrailapp.com/events?centered_on_id=453512210627776512&q=program%3Aapp%2Fweb.4> >> : at >> org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager.dispatch(SelectChannelConnector.java:300) >> app/web.4 >> <https://papertrailapp.com/events?centered_on_id=453512210627776513&q=program%3Aapp%2Fweb.4> >> : at >> org.eclipse.jetty.io.nio.SelectChannelEndPoint.dispatch(SelectChannelEndPoint.java:236) >> app/web.4 >> <https://papertrailapp.com/events?centered_on_id=453512210627776514&q=program%3Aapp%2Fweb.4> >> : at >> org.eclipse.jetty.io.nio.SelectChannelEndPoint.schedule(SelectChannelEndPoint.java:195) >> app/web.4 >> <https://papertrailapp.com/events?centered_on_id=453512210627776515&q=program%3Aapp%2Fweb.4> >> : at >> org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:619) >> app/web.4 >> <https://papertrailapp.com/events?centered_on_id=453512210627776516&q=program%3Aapp%2Fweb.4> >> : at >> org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290) >> app/web.4 >> <https://papertrailapp.com/events?centered_on_id=453512210627776518&q=program%3Aapp%2Fweb.4> >> : at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) >> app/web.4 >> <https://papertrailapp.com/events?centered_on_id=453512210627776520&q=program%3Aapp%2Fweb.4> >> : at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> app/web.4 >> <https://papertrailapp.com/events?centered_on_id=453512210627776521&q=program%3Aapp%2Fweb.4> >> : at java.lang.Thread.run(Thread.java:679) >> >> This error spits out over 100 times a second. When the server is >> brought up, it runs fine for about 4-8 hours until this error starts >> showing, and the application is soon unresponsive. >> >> The application runs on a Heroku server, it is on Java 1.6, using Jetty >> Version 7.6.16.v20140903 >> >> I also found this (seemingly) related bug: >> <https://jira.codehaus.org/browse/JETTY-1423,> >> https://jira.codehaus.org/browse/JETTY-1423 , however, since I am using >> Jetty 1.6 I am not using an affected version. >> >> Any help or insight into what may be causing this issue would be >> appreciated. >> >> Thanks! >> >> >> _______________________________________________ >> jetty-users mailing list >> [email protected] >> To change your delivery options, retrieve your password, or unsubscribe >> from this list, visit >> https://dev.eclipse.org/mailman/listinfo/jetty-users >> > > > _______________________________________________ > jetty-users mailing list > [email protected] > To change your delivery options, retrieve your password, or unsubscribe > from this list, visit > https://dev.eclipse.org/mailman/listinfo/jetty-users >
_______________________________________________ jetty-users mailing list [email protected] To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/jetty-users
