El mié 15 ene 2014 21:06:41 CET, R P Herrold escribió: > On Wed, 15 Jan 2014, R P Herrold wrote: > >> Nagios reports it disappeared at 13:28:44 >> and re-appeared at 13:48:34 > > Hi, David > > I see you said in IRC > > 13:42 < dcaro> ecohen: apuimedo|away yes, gerrit web is having > issues, it started too many threads > > What jobs were then running that you saw this? Did you deduce > this from the process table, the dmesg, or some log file? > > -- Russ herrold
Hi Russ,
Well, more or less, I saw that the process count for gerrit was a
number really close to 1024, and that it was not increasing or
decreasing. Then I looked at the logs and sow a coupple of entries like
this:
[2014-01-15 13:21:50,584] ERROR com.google.gerrit.pgm.Daemon : Thread
HTTP-23 threw exception
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:657)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.startThread(QueuedThreadPool.java:441)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.dispatch(QueuedThreadPool.java:366)
at
org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager.dispatch(SelectChannelConnector.java:300)
at
org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:708)
at
org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:679)
So yes, that seems to be the problem. I raised the limit for user
processes from 1024 to 10240 for now, but I have to check what is
making those processes get stuck.
I'm not sure of the jobs that were running at that moment. But there
were a few and all get stuck trying to get the changes through http
protocol.
--
David Caro
Red Hat S.L.
Continuous Integration Engineer - EMEA ENG Virtualization R&D
Email: [email protected]
Web: www.redhat.com
RHT Global #: 82-62605
signature.asc
Description: OpenPGP digital signature
_______________________________________________ Infra mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/infra
