https://bugzilla.wikimedia.org/show_bug.cgi?id=47284
Web browser: ---
Bug ID: 47284
Summary: Gerrit/Jetty flooding logs with "Dispatched Failed!"
messages and becoming unresponsive
Product: Wikimedia
Version: wmf-deployment
Hardware: All
OS: All
Status: NEW
Severity: critical
Priority: Unprioritized
Component: Git/Gerrit
Assignee: wikibugs-l@lists.wikimedia.org
Reporter: christ...@quelltextlich.at
CC: christ...@quelltextlich.at, innocentkil...@gmail.com,
marcin.cies...@gmail.com, rlan...@gmail.com
Classification: Unclassified
Mobile Platform: ---
Gerrit with the shipped Jetty servlet container causes problems, when
the client connection pool fills up.
Then gerrit's log get spammed with messages like
[2013-02-19 14:55:54,592] WARN org.eclipse.jetty.io.nio : Dispatched Failed!
SCEP@44fb799e{l(/127.0.0.1:47357)<->r(/127.0.0.1:8080),d=false,open=true,ishut=false,oshut=false,rb=false,wb=false,w=true,i=1r}-{AsyncHttpConnection@59ed9773,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=0},r=0}
to
org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager@3699d37e
This spamming renders Gerrit extremely unresponsive. When the problem
hit us around 2013-02-19 12:55:22,464 UTC, spamming through above
messages amounted to about 1Gb of logs in ~2 hours.
The problem is known and documented upstream:
http://code.google.com/p/gerrit/wiki/Scaling#Jetty
The problem hit us again at 2013-02-21 ~10:50 UTC, and also in one of
the last days.
Possible mitigation strategies are:
* hand connection limiting off to apache
https://gerrit.wikimedia.org/r/#/c/50591
* Trading Jetty for another container (e.g.: Tomcat)
--
You are receiving this mail because:
You are the assignee for the bug.
You are on the CC list for the bug.
_______________________________________________
Wikibugs-l mailing list
Wikibugs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
