Hi, It's CentOS.
# /usr/sbin/sestatus SELinux status: disabled 2014/1/18 Joakim Erdfelt <[email protected]> > What linux distribution are you running? Curious if selinux is active on > your system. > > -- > Joakim Erdfelt <[email protected]> > webtide.com <http://www.webtide.com/> - intalio.com/jetty > Expert advice, services and support from from the Jetty & CometD experts > eclipse.org/jetty - cometd.org > > > On Thu, Jan 16, 2014 at 7:45 PM, Zen Zhong <[email protected]> wrote: > >> Hi, Simone >> >> I enabled debug logging for HttpInput and HttpInputOverHttp. I found java >> process was killed by kernel today, I'm not sure whether the issue was >> reproduced or not this time, but Scheduler thread really cost much time to >> read data from one connection, some logs as following: >> >> In /var/log/messages, there're many "Out of socket memory" and "possible >> SYN flooding on port 80. Sending cookies.", and one "Jan 16 19:35:01 >> <hostname> kernel: Out of memory: Kill process 30279 (java) score 367 or >> sacrifice child". >> >> tail -n 10000000 /data/jetty/logs/2014_01_16.stderrout.log > temp >> less -N temp >> 1 2014-01-16 >> 19:29:56.442:DBUG:oejs.HttpInputOverHTTP:Scheduler-1456460106: >> HttpInputOverHTTP@fa92e29 filled 0 >> 2 2014-01-16 >> 19:29:56.442:DBUG:oejs.HttpInputOverHTTP:Scheduler-1456460106: >> HttpInputOverHTTP@fa92e29 filled 0 >> 3 2014-01-16 >> 19:29:56.442:DBUG:oejs.HttpInputOverHTTP:Scheduler-1456460106: >> HttpInputOverHTTP@fa92e29 filled 0 >> ...<repeat> >> 3868890 2014-01-16 >> 19:31:25.810:DBUG:oejs.HttpInputOverHTTP:Scheduler-1456460106: >> HttpInputOverHTTP@713f72e1 filled 0 >> 3868891 2014-01-16 >> 19:31:25.810:DBUG:oejs.HttpInputOverHTTP:Scheduler-1456460106: >> HttpInputOverHTTP@713f72e1 filled -1 >> 3868892 2014-01-16 19:31:25.810:DBUG:oejs.HttpInput:Scheduler-1456460106: >> HttpInputOverHTTP@713f72e1 early EOF >> 3868893 2014-01-16 19:31:25.810:DBUG:oejs.HttpInput:Scheduler-1456460106: >> HttpInputOverHTTP@713f72e1 eof EARLY_EOF >> ...<ignore some timeout exception> >> 3868952 2014-01-16 >> 19:31:27.076:DBUG:oejs.HttpInputOverHTTP:Scheduler-1456460106: >> HttpInputOverHTTP@5a1809ad filled 0 >> 3868953 2014-01-16 >> 19:31:27.076:WARN:oejs.HttpConnection:Scheduler-1456460106: req=[POST >> /api/oauth2/token]@955987031 >> org.eclipse.jetty.server.Request@38fb34538689537, >> _parser=HttpParser{s=CONTENT,0 of 61}, _parser.eof=false >> 3868954 java.lang.Exception: testparserstate >> 3868955 at >> org.eclipse.jetty.server.HttpConnection.completed(HttpConnection.java:326) >> 3868956 at >> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:401) >> 3868957 at >> org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:238) >> 3868958 at >> org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1159) >> 3868959 at >> org.eclipse.jetty.server.HttpChannelState.complete(HttpChannelState.java:440) >> 3868960 at >> org.eclipse.jetty.server.AsyncContextState.complete(AsyncContextState.java:92) >> 3868961 at >> my.web.async.BasicAsyncListener.onTimeout(BasicAsyncListener.java:36) >> 3868962 at >> org.eclipse.jetty.server.HttpChannelState.expired(HttpChannelState.java:396) >> 3868963 at >> org.eclipse.jetty.server.HttpChannelState$AsyncTimeout.run(HttpChannelState.java:699) >> 3868964 at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> 3868965 at >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >> 3868966 at >> java.util.concurrent.FutureTask.run(FutureTask.java:166) >> 3868967 at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) >> 3868968 at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) >> 3868969 at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> 3868970 at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> 3868971 at java.lang.Thread.run(Thread.java:724) >> 3868972 2014-01-16 >> 19:31:27.076:DBUG:oejs.HttpInputOverHTTP:Scheduler-1456460106: >> HttpInputOverHTTP@5a1809ad filled 0 >> 3868973 2014-01-16 >> 19:31:27.076:DBUG:oejs.HttpInputOverHTTP:Scheduler-1456460106: >> HttpInputOverHTTP@5a1809ad filled 0 >> 3868974 2014-01-16 >> 19:31:27.076:DBUG:oejs.HttpInputOverHTTP:Scheduler-1456460106: >> HttpInputOverHTTP@5a1809ad filled 0 >> ...<repeat> >> 4151735 2014-01-16 >> 19:31:33.536:DBUG:oejs.HttpInputOverHTTP:Scheduler-1456460106: >> HttpInputOverHTTP@5a1809ad filled 0 >> 4151736 2014-01-16 >> 19:31:33.536:DBUG:oejs.HttpInputOverHTTP:Scheduler-1456460106: >> HttpInputOverHTTP@5a1809ad filled 0 >> 4151737 2014-01-16 >> 19:31:33.536:DBUG:oejs.HttpInputOverHTTP:Scheduler-1456460106: >> HttpInputOverHTTP@5a1809ad filled -1 >> 4151738 2014-01-16 19:31:33.536:DBUG:oejs.HttpInput:Scheduler-1456460106: >> HttpInputOverHTTP@5a1809ad early EOF >> 4151739 2014-01-16 19:31:33.536:DBUG:oejs.HttpInput:Scheduler-1456460106: >> HttpInputOverHTTP@5a1809ad eof EARLY_EOF >> 4151740 2014-01-16 >> 19:31:33.640:DBUG:oejs.HttpInputOverHTTP:Scheduler-1456460106: >> HttpInputOverHTTP@e2db5db filled 0 >> 4151741 2014-01-16 >> 19:31:33.640:WARN:oejs.HttpConnection:Scheduler-1456460106: req=[POST >> /api/oauth2/token]@1603375770 >> org.eclipse.jetty.server.Request@5f919241517419a, >> _parser=HttpParser{s=CONTENT,0 of 126}, _parser.eof=false >> 4151742 java.lang.Exception: testparserstate >> 4151743 at >> org.eclipse.jetty.server.HttpConnection.completed(HttpConnection.java:326) >> 4151744 at >> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:401) >> 4151745 at >> org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:238) >> 4151746 at >> org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1159) >> 4151747 at >> org.eclipse.jetty.server.HttpChannelState.complete(HttpChannelState.java:440) >> 4151748 at >> org.eclipse.jetty.server.AsyncContextState.complete(AsyncContextState.java:92) >> 4151749 at >> my.web.async.BasicAsyncListener.onTimeout(BasicAsyncListener.java:36) >> 4151750 at >> org.eclipse.jetty.server.HttpChannelState.expired(HttpChannelState.java:396) >> 4151751 at >> org.eclipse.jetty.server.HttpChannelState$AsyncTimeout.run(HttpChannelState.java:699) >> 4151752 at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> 4151753 at >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >> 4151754 at >> java.util.concurrent.FutureTask.run(FutureTask.java:166) >> 4151755 at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) >> 4151756 at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) >> 4151757 at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> 4151758 at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> 4151759 at java.lang.Thread.run(Thread.java:724) >> 4151760 2014-01-16 >> 19:31:33.641:DBUG:oejs.HttpInputOverHTTP:Scheduler-1456460106: >> HttpInputOverHTTP@e2db5db filled 0 >> 4151761 2014-01-16 >> 19:31:33.641:DBUG:oejs.HttpInputOverHTTP:Scheduler-1456460106: >> HttpInputOverHTTP@e2db5db filled 0 >> 4151762 2014-01-16 >> 19:31:33.641:DBUG:oejs.HttpInputOverHTTP:Scheduler-1456460106: >> HttpInputOverHTTP@e2db5db filled 0 >> ...<repeat> >> 9999998 2014-01-16 >> 19:35:01.223:DBUG:oejs.HttpInputOverHTTP:Scheduler-1456460106: >> HttpInputOverHTTP@e2db5db filled 0 >> 9999999 2014-01-16 >> 19:35:01.223:DBUG:oejs.HttpInputOverHTTP:Scheduler-1456460106: >> HttpInputOverHTTP@e2db5db filled 0 >> 10000000 2014-01-16 >> 19:35:01.223:DBUG:oejs.HttpInputOverHTTP:Scheduler-1456460106: >> HttpInputOverHTTP@e2db5db filled 0 >> >> Thanks >> >> >> 2014/1/15 Simone Bordet <[email protected]> >> >>> Hi, >>> >>> On Tue, Jan 14, 2014 at 2:13 PM, Zen Zhong <[email protected]> >>> wrote: >>> > Hi, >>> > >>> > I found there's connections leak in jetty. Environment: CentOS 6 >>> x86_64, JDK >>> > 7u25 x86_64, jetty-9.1.0. >>> >>> You have 3 stack dumps that look almost the same. >>> Since reads are non blocking, the timer thread was caught at that >>> point in all 3 times. >>> Did you experience 100% CPU usage for 1 core ? >>> >>> Otherwise it can be explained with a really slow client. >>> Method consumedAll() reads from the client, and it always reads >>> something. Had it read 0 bytes, you would have seen a stack dump >>> waiting in blockForContent(). >>> >>> If you can reproduce this problem, can you enable debug logging for >>> HttpInput and HttpInputOverHTTP ? >>> That should tell us what's going on. >>> >>> Thanks ! >>> >>> -- >>> Simone Bordet >>> ---- >>> http://cometd.org >>> http://webtide.com >>> http://intalio.com >>> Developer advice, training, services and support >>> from the Jetty & CometD experts. >>> Intalio, the modern way to build business applications. >>> _______________________________________________ >>> jetty-users mailing list >>> [email protected] >>> https://dev.eclipse.org/mailman/listinfo/jetty-users >>> >> >> >> _______________________________________________ >> jetty-users mailing list >> [email protected] >> https://dev.eclipse.org/mailman/listinfo/jetty-users >> >> > > _______________________________________________ > jetty-users mailing list > [email protected] > https://dev.eclipse.org/mailman/listinfo/jetty-users > >
_______________________________________________ jetty-users mailing list [email protected] https://dev.eclipse.org/mailman/listinfo/jetty-users
