Hi Chris, thank you for troubleshooting this, can you clarify one thing for me, when you mention log files, are these the `.log` files generated during the test phase of the build or the XML/TXT files with JUnit reports?
I would think that the job type being Maven and the automatic gathering of JUnit test reports is the culprit but would like a confirmation. If it is so, I think we one possible solution is to migrate to Pipeline/Freestyle job type as, as far as I'm aware, there is no way to prevent Maven job type from gathering JUnit reports. zoran On Fri, Feb 16, 2018 at 2:37 AM, Chris Lambertus <c...@apache.org> wrote: > > Hi Camel PMC, > > We have been having an ongoing problem with Jenkins for quite some time, > where the CPU usage and IOPS skyrocket on the master. Each time this has > happened, the jenkins build nodes lose all of their associated labels, and > all new builds are unable to start. > > In the times I’ve been able to investigate this, there has in each case been > several Camel builds running, and in each case, the builds are opening > somewhere between 1500 and 2200 log files, which seems to be killing the > jenkins master. For comparison, some very large build jobs for Hadoop only > open ~15 or so log files. > > I have had to take the rather drastic step of disabling the Camel Jenkins > jobs (many of which have been failing for awhile now) while we continue to > investigate this issue. Before we re-enable the jobs, we’re going to have to > figure out how to get your builds to open a sane number of log files — the > current situation where the builds are creating thousands of log files is not > sustainable, and we believe this may be one of the causative factors of the > ongoing jenkins outages. While I cannot say with any certainty that this is > what’s been killing the master, it’s far enough out of the norm that I need > to rule it out. > > Please do not re-enable any of the disabled builds until we have had a chance > to work on this together. Can you please identify someone from the project to > act as a liaison with Infra to troubleshoot the issues with these builds? > > Since I have sent this to a private list, you have my permission to forward > this message on to your devs or other public lists as you deem appropriate. > > > Thanks, > > -Chris > ASF Infra > -- Zoran Regvart
