Hi,
My jenkins instance was a little slower to respond this morning, I thought
maybe a bunch of jobs were running at the time, but the run queue was
empty. top showed the process running at 700%, basically 7 threads pegging
a core. Each was hung in the following code:
"RequestHandlerThread[#112]" daemon prio=10 tid=0x00002aaab80c7000
nid=0x721e runnable [0x00000000413c8000]
java.lang.Thread.State: RUNNABLE
at java.util.zip.Deflater.deflateBytes(Native Method)
at java.util.zip.Deflater.deflate(Deflater.java:306)
- locked <0x00000000e194f3c0> (a java.util.zip.ZStreamRef)
at
java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:159)
at
java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:118)
at java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:72)
- locked <0x00000000e194f5e8> (a java.util.zip.GZIPOutputStream)
at
org.kohsuke.stapler.compression.FilterServletOutputStream.write(FilterServletOutputStream.java:31)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:297)
at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:130)
- locked <0x00000000e194f640> (a java.io.OutputStreamWriter)
at java.io.OutputStreamWriter.close(OutputStreamWriter.java:216)
at java.io.PrintWriter.close(PrintWriter.java:295)
- locked <0x00000000e194f640> (a java.io.OutputStreamWriter)
at
org.kohsuke.stapler.compression.CompressionFilter.reportException(CompressionFilter.java:77)
at
org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53)
at
winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at
hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
at
winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
at
winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215)
at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
I thought maybe I had 7 users streaming job output, but netstat showed no
open connections.
The following error was in the log 7 times:
WARNING: Untrapped servlet exception
winstone.ClientSocketException: Failed to write to client
at winstone.ClientOutputStream.write(ClientOutputStream.java:41)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at
winstone.WinstoneOutputStream.commit(WinstoneOutputStream.java:167)
at
winstone.WinstoneOutputStream.flush(WinstoneOutputStream.java:219)
at
winstone.WinstoneOutputStream.close(WinstoneOutputStream.java:229)
at
java.util.zip.DeflaterOutputStream.close(DeflaterOutputStream.java:149)
at
org.kohsuke.stapler.compression.FilterServletOutputStream.close(FilterServletOutputStream.java:36)
at org.kohsuke.stapler.Stapler.serveStaticResource(Stapler.java:422)
at org.kohsuke.stapler.Stapler.serveStaticResource(Stapler.java:249)
at org.kohsuke.stapler.Stapler.service(Stapler.java:152)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
at
winstone.ServletConfiguration.execute(ServletConfiguration.java:248)
at winstone.RequestDispatcher.forward(RequestDispatcher.java:333)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376)
at
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)
at
hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)
at
winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47)
at
winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
at
hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
at
winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at
org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
at
winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at
hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
at
winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
at
winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215)
at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketException: Connection reset
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at winstone.ClientOutputStream.write(ClientOutputStream.java:39)
... 42 more
I can try upgrading to the latest, I'm currently running 1.477, not that
far behind, and didn't think stapler changed that much
