One of the problems that I've seen is that xinetd is configured to a low limit of simultaneous connections and it's dropping them when a few jenkins jobs are running in parallel.
Changing/adding these options in the /etc/xinit.d/git file should be
enough (those are the actual values):
# Connection per second and time to wait when reached
cps = 50 10
# Max number of parallel instances
instances = 50
# Max number of connection per source
per_source = 10
Here is the trace of one of the failures:
Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit
from=::ffff:XX.XX.XX.X1
Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit
from=::ffff:XX.XX.XX.X2
Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit
from=::ffff:XX.XX.XX.X2
Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit
from=::ffff:XX.XX.XX.X2
Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit
from=::ffff:XX.XX.XX.X2
Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit
from=::ffff:XX.XX.XX.X2
Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit
from=::ffff:XX.XX.XX.X3
Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit
from=::ffff:XX.XX.XX.X4
Jun 6 10:30:07 gerrit xinetd[1153]: FAIL: git service_limit
from=::ffff:XX.XX.XX.X2
Jun 6 10:30:08 gerrit xinetd[1153]: FAIL: git service_limit
from=::ffff:XX.XX.XX.X5
Jun 6 10:30:08 gerrit xinetd[1153]: EXIT: git status=0 pid=340
duration=1(sec)
### The XX.XX.XX.XX are obfuscated ips
Also I've seen that the number of open files is stable and between
reasonable values, that's good :)
And memory and load seem fine too.
I'll keep monitoring it and see if I see something more, but I recommend
raising the values and see if it's enough of there's another underlying
problem.
--
David Caro
Red Hat Czech s.r.o.
Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605
Email: [email protected]
Web: www.cz.redhat.com
Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno, Czech Republic
RHT Global #: 82-62605
signature.asc
Description: OpenPGP digital signature
_______________________________________________ Infra mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/infra
