![]() |
|
|
|
|
Change By:
|
James Noonan
(11/Aug/14 2:23 PM)
|
|
Attachment:
|
thread_growth.png
|
|
Description:
|
The Jenkins Master reports that all its JNLP (and all our nodes are such) are offline.
On the nodes, they report that they are connected. The only way out is to restart the Master. Of the 6 (or so) times this has occurred, 1/2 the time all the slaves need to have their slave process restarted to recover.
We also see cases where after a restarting Jenkins, it recovers for a short time. Then the problem re-occurs. However, if it's running 10-minutes after a restart, we seem to be fine for 3-4 days.
We were running on version 565 when this first occurred. We ran fine for 3-months. What changed for us is that we increased the number of nodes. We now have 93 nodes, up from about 50. There was also an increase in the number of jobs.
We use the vSphere Cloud Plugin. However, we changed one slave to use ssh instead of jnlp. The problem was resolved for this slave, and it is not disconnected when the problem occurs. We did not find the same for a vsphere/jnlp slave where we removed the vsphere configuration. (Well, recreated the slave without vsphere).
This seems to be similar to:
https://issues.jenkins-ci.org/browse/JENKINS-24155
https://issues.jenkins-ci.org/browse/JENKINS-24050
https://issues.jenkins-ci.org/browse/JENKINS-22714
https://issues.jenkins-ci.org/browse/JENKINS-22932
https://issues.jenkins-ci.org/browse/JENKINS-23384
We have examined the VM logs, the network logs and the firewalls. There is no obvious issue.
I'm attaching the err.log of one of the incidents. Though it is clear that there is a problem with the slave connections, there is no clear 'cause'.
I've attached a thread graph of the problem.
Normally, Jenkins runs at about 200 threads. _AFTER_ the problem occurs, thread growth occurs linearly until reboot. In the graph, we see there was a problem on Friday night, as activity died off. After a Sat service restart, we see the problem occur again 6-hours later, with corresponding thread growth.
We suspect that if only the Jenkins Service is restarted, the time to next occurrence is lower.
Also, we configure some nodes to turn off when idle. However, though we originally suspected this to be a possible cause, we have not found any thing to further collaborate this theory.
|
|
|
|
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira
|
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
[email protected].
For more options, visit
https://groups.google.com/d/optout.
