|
After upgrading to Jenkins v1.549, ssh-slaves v1.6, git plugin v2.0.1, git-client plugin v1.6.2, we began observing many errors and job failures when running builds on our slave nodes. The errors were typically of the form:
In general, we saw these errors mainly during two stages of the build process: during SCM step (git checkout), and during transfer of a file specified as a job parameter to the slave node. Some examples of these errors are below; however, it is worth noting that we did observe errors at other phases of the build as well (i.e. was not limited to these two phases).
These errors were not dependably reproducible, and were nearly impossible to duplicate in isolation (i.e. could not reproduce when just running jobs on a single slave node).
After some time debugging and ruling out some things, we determined that perhaps the user nproc limit (ulimit -u) might be too low, based on the errors that w were seeing and what we were able to rule out after debugging. We increased this limit on our slave nodes and...things went back to normal.
None of our build and test jobs changed across the update, meaning that any changes we observed were introduced as a result of the upgrade. My question is then: what has changed recently with slave management that has increased the number of per node threads required? We don't yet have solid watermark data on how high it was actually going, but it was high enough to hit the ceiling for system default (1024).
|
