Re: camel builds causing problems with jenkins
Hi Zoran Thanks for your endless effort on the Jenkins Pipeline effort. Much appreciated to get the CI servers to be more reliable and useable for us at our big Apache Camel project. Keep it up. On Mon, Feb 19, 2018 at 10:44 AM, Zoran Regvart wrote: > Hi Cameleers, > I've created the initial Jenkinsfile and created multibranch pipeline > job at builds.apache.org[1]. I'll continue to refine it on the > `jenkins-pipeline` branch and then move that over to `master` when it > stabilizes. > > Things still left to do: > - e-mail notifications > - cherry picking Jenkinsfile changes to 2.19 and 2.20 branches > - Java 9 build > - run integration tests with karaf, osgi and springboot > > Anyone wishing to contribute to this can experiment on a branch pushed > to ASF/github mirror and see that branch being built in the same job, > when you get it stable enough just cherry pick the changes to > `master`. > > zoran > > [1] https://builds.apache.org/job/Camel/ > -- > Zoran Regvart -- Claus Ibsen - http://davsclaus.com @davsclaus Camel in Action 2: https://www.manning.com/ibsen2
Re: camel builds causing problems with jenkins
Hi Great work Zoran. Just wanted to say that back porting to 2.19.x branch is not really needed, as this branch is going to be EOL soon when we have 2.21.0 released, as then 2.19.x would be the last release. On Mon, Feb 19, 2018 at 10:44 AM, Zoran Regvart wrote: > Hi Cameleers, > I've created the initial Jenkinsfile and created multibranch pipeline > job at builds.apache.org[1]. I'll continue to refine it on the > `jenkins-pipeline` branch and then move that over to `master` when it > stabilizes. > > Things still left to do: > - e-mail notifications > - cherry picking Jenkinsfile changes to 2.19 and 2.20 branches > - Java 9 build > - run integration tests with karaf, osgi and springboot > > Anyone wishing to contribute to this can experiment on a branch pushed > to ASF/github mirror and see that branch being built in the same job, > when you get it stable enough just cherry pick the changes to > `master`. > > zoran > > [1] https://builds.apache.org/job/Camel/ > -- > Zoran Regvart -- Claus Ibsen - http://davsclaus.com @davsclaus Camel in Action 2: https://www.manning.com/ibsen2
Re: camel builds causing problems with jenkins
Hi Cameleers, I've created the initial Jenkinsfile and created multibranch pipeline job at builds.apache.org[1]. I'll continue to refine it on the `jenkins-pipeline` branch and then move that over to `master` when it stabilizes. Things still left to do: - e-mail notifications - cherry picking Jenkinsfile changes to 2.19 and 2.20 branches - Java 9 build - run integration tests with karaf, osgi and springboot Anyone wishing to contribute to this can experiment on a branch pushed to ASF/github mirror and see that branch being built in the same job, when you get it stable enough just cherry pick the changes to `master`. zoran [1] https://builds.apache.org/job/Camel/ -- Zoran Regvart
Re: camel builds causing problems with jenkins
The problem is entirely in the per module files on the master - with Pipeline, you're only going to call junit once, which will read all the relevant files on the agent and write just one file on the master. It'll definitely work much, much better. Feel free to reach out to me (my day job is working on Jenkins Pipeline internals and usability) if you need any help with Pipeline. A. On Feb 16, 2018 15:22, "Zoran Regvart" wrote: > Thank Chris you for clarifying, > those log files are created by Jenkins Maven support[1], and they are > created for each Maven module of the project that's being built, with > Camel having 840 maven modules. I don't think this approach scales > with such big number of Maven modules. > > So Camel devs, seems like we need to switch to a different Jenkins job > type, I propose that we switch to Jenkins pipeline job type. At some > point I've been experimenting with that[2], and I can lead this > effort. > > I wonder, though, if the same issue will pop up with the JUnit test > reports archiving in the pipeline job, there will still be roughly the > same amount of files transferred to master, it could be that there has > been some work on Jenkins side to optimize that with pipeline. > > zoran > > [1] https://github.com/jenkinsci/maven-plugin/blob/ > [2] https://github.com/zregvart/camel/blob/jenkinsfile/Jenkinsfile > > On Fri, Feb 16, 2018 at 6:39 PM, Chris Lambertus wrote: > > Here is a small sample of the types of logs we’re seeing: > > > > cml@jenkins-master:~$ cat camel-20180215.lsof | grep camel | grep log | > wc -l > > 2277 > > > > > > java18713 jenkins 4931w REG 8,17 0 > 215356692 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/ > modules/org.apache.camel.example$camel-example-twitter- > salesforce/builds/1328/log > > java18713 jenkins 4932w REG 8,17 0 > 215356694 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/ > modules/org.apache.camel.example$camel-example-twitter- > websocket/builds/1328/log > > java18713 jenkins 4933w REG 8,17 0 > 215356696 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/ > modules/org.apache.camel.example$camel-example-twitter- > websocket-blueprint/builds/1328/log > > java18713 jenkins 4934w REG 8,17 0 > 215356698 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/ > modules/org.apache.camel.example$camel-example- > validator-spring-boot/builds/1328/log > > java18713 jenkins 4935w REG 8,17 0 > 215356701 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/ > modules/org.apache.camel.example$camel-example-widget- > gadget-cdi/builds/1328/log > > java18713 jenkins 4936w REG 8,17 0 > 215356706 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/ > modules/org.apache.camel.example$camel-example-widget- > gadget-java/builds/1328/log > > java18713 jenkins 4937w REG 8,17 0 > 215356709 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/ > modules/org.apache.camel.example$camel-example-widget- > gadget-xml/builds/1328/log > > java18713 jenkins 4938w REG 8,17 0 > 215356714 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/ > modules/org.apache.camel.example$camel-example-zipkin/builds/1328/log > > java18713 jenkins 4939w REG 8,17 0 > 215356725 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/ > modules/org.apache.camel.example$camel-example-zipkin- > client/builds/1328/log > > > > > > > >> On Feb 16, 2018, at 3:56 AM, Zoran Regvart wrote: > >> > >> Hi Chris, > >> thank you for troubleshooting this, can you clarify one thing for me, > >> when you mention log files, are these the `.log` files generated > >> during the test phase of the build or the XML/TXT files with JUnit > >> reports? > >> > >> I would think that the job type being Maven and the automatic > >> gathering of JUnit test reports is the culprit but would like a > >> confirmation. > >> > >> If it is so, I think we one possible solution is to migrate to > >> Pipeline/Freestyle job type as, as far as I'm aware, there is no way > >> to prevent Maven job type from gathering JUnit reports. > >> > >> zoran > >> > >> On Fri, Feb 16, 2018 at 2:37 AM, Chris Lambertus > wrote: > >>> > >>> Hi Camel PMC, > >>> > >>> We have been having an ongoing problem with Jenkins for quite some > time, where the CPU usage and IOPS skyrocket on the master. Each time this > has happened, the jenkins build nodes lose all of their associated labels, > and all new builds are unable to start. > >>> > >>> In the times I’ve been able to investigate this, there has in each > case been several Camel builds running, and in each case, the builds are > opening somewhere between 1500 and 2200 log files, which seems to be > killing the jenkins master. For comparison, some very large build jobs for > Hadoop only open
Re: camel builds causing problems with jenkins
Thank Chris you for clarifying, those log files are created by Jenkins Maven support[1], and they are created for each Maven module of the project that's being built, with Camel having 840 maven modules. I don't think this approach scales with such big number of Maven modules. So Camel devs, seems like we need to switch to a different Jenkins job type, I propose that we switch to Jenkins pipeline job type. At some point I've been experimenting with that[2], and I can lead this effort. I wonder, though, if the same issue will pop up with the JUnit test reports archiving in the pipeline job, there will still be roughly the same amount of files transferred to master, it could be that there has been some work on Jenkins side to optimize that with pipeline. zoran [1] https://github.com/jenkinsci/maven-plugin/blob/ [2] https://github.com/zregvart/camel/blob/jenkinsfile/Jenkinsfile On Fri, Feb 16, 2018 at 6:39 PM, Chris Lambertus wrote: > Here is a small sample of the types of logs we’re seeing: > > cml@jenkins-master:~$ cat camel-20180215.lsof | grep camel | grep log | wc -l > 2277 > > > java18713 jenkins 4931w REG 8,17 0 215356692 > /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/modules/org.apache.camel.example$camel-example-twitter-salesforce/builds/1328/log > java18713 jenkins 4932w REG 8,17 0 215356694 > /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/modules/org.apache.camel.example$camel-example-twitter-websocket/builds/1328/log > java18713 jenkins 4933w REG 8,17 0 215356696 > /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/modules/org.apache.camel.example$camel-example-twitter-websocket-blueprint/builds/1328/log > java18713 jenkins 4934w REG 8,17 0 215356698 > /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/modules/org.apache.camel.example$camel-example-validator-spring-boot/builds/1328/log > java18713 jenkins 4935w REG 8,17 0 215356701 > /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/modules/org.apache.camel.example$camel-example-widget-gadget-cdi/builds/1328/log > java18713 jenkins 4936w REG 8,17 0 215356706 > /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/modules/org.apache.camel.example$camel-example-widget-gadget-java/builds/1328/log > java18713 jenkins 4937w REG 8,17 0 215356709 > /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/modules/org.apache.camel.example$camel-example-widget-gadget-xml/builds/1328/log > java18713 jenkins 4938w REG 8,17 0 215356714 > /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/modules/org.apache.camel.example$camel-example-zipkin/builds/1328/log > java18713 jenkins 4939w REG 8,17 0 215356725 > /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/modules/org.apache.camel.example$camel-example-zipkin-client/builds/1328/log > > > >> On Feb 16, 2018, at 3:56 AM, Zoran Regvart wrote: >> >> Hi Chris, >> thank you for troubleshooting this, can you clarify one thing for me, >> when you mention log files, are these the `.log` files generated >> during the test phase of the build or the XML/TXT files with JUnit >> reports? >> >> I would think that the job type being Maven and the automatic >> gathering of JUnit test reports is the culprit but would like a >> confirmation. >> >> If it is so, I think we one possible solution is to migrate to >> Pipeline/Freestyle job type as, as far as I'm aware, there is no way >> to prevent Maven job type from gathering JUnit reports. >> >> zoran >> >> On Fri, Feb 16, 2018 at 2:37 AM, Chris Lambertus wrote: >>> >>> Hi Camel PMC, >>> >>> We have been having an ongoing problem with Jenkins for quite some time, >>> where the CPU usage and IOPS skyrocket on the master. Each time this has >>> happened, the jenkins build nodes lose all of their associated labels, and >>> all new builds are unable to start. >>> >>> In the times I’ve been able to investigate this, there has in each case >>> been several Camel builds running, and in each case, the builds are opening >>> somewhere between 1500 and 2200 log files, which seems to be killing the >>> jenkins master. For comparison, some very large build jobs for Hadoop only >>> open ~15 or so log files. >>> >>> I have had to take the rather drastic step of disabling the Camel Jenkins >>> jobs (many of which have been failing for awhile now) while we continue to >>> investigate this issue. Before we re-enable the jobs, we’re going to have >>> to figure out how to get your builds to open a sane number of log files — >>> the current situation where the builds are creating thousands of log files >>> is not sustainable, and we believe this may be one of the causative factors >>> of the ongoing jenkins outages. While I cannot say with any certainty
Re: camel builds causing problems with jenkins
Here is a small sample of the types of logs we’re seeing: cml@jenkins-master:~$ cat camel-20180215.lsof | grep camel | grep log | wc -l 2277 java18713 jenkins 4931w REG 8,17 0 215356692 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/modules/org.apache.camel.example$camel-example-twitter-salesforce/builds/1328/log java18713 jenkins 4932w REG 8,17 0 215356694 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/modules/org.apache.camel.example$camel-example-twitter-websocket/builds/1328/log java18713 jenkins 4933w REG 8,17 0 215356696 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/modules/org.apache.camel.example$camel-example-twitter-websocket-blueprint/builds/1328/log java18713 jenkins 4934w REG 8,17 0 215356698 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/modules/org.apache.camel.example$camel-example-validator-spring-boot/builds/1328/log java18713 jenkins 4935w REG 8,17 0 215356701 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/modules/org.apache.camel.example$camel-example-widget-gadget-cdi/builds/1328/log java18713 jenkins 4936w REG 8,17 0 215356706 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/modules/org.apache.camel.example$camel-example-widget-gadget-java/builds/1328/log java18713 jenkins 4937w REG 8,17 0 215356709 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/modules/org.apache.camel.example$camel-example-widget-gadget-xml/builds/1328/log java18713 jenkins 4938w REG 8,17 0 215356714 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/modules/org.apache.camel.example$camel-example-zipkin/builds/1328/log java18713 jenkins 4939w REG 8,17 0 215356725 /x1/jenkins/jenkins-home/jobs/Camel.trunk.notest.java9/modules/org.apache.camel.example$camel-example-zipkin-client/builds/1328/log > On Feb 16, 2018, at 3:56 AM, Zoran Regvart wrote: > > Hi Chris, > thank you for troubleshooting this, can you clarify one thing for me, > when you mention log files, are these the `.log` files generated > during the test phase of the build or the XML/TXT files with JUnit > reports? > > I would think that the job type being Maven and the automatic > gathering of JUnit test reports is the culprit but would like a > confirmation. > > If it is so, I think we one possible solution is to migrate to > Pipeline/Freestyle job type as, as far as I'm aware, there is no way > to prevent Maven job type from gathering JUnit reports. > > zoran > > On Fri, Feb 16, 2018 at 2:37 AM, Chris Lambertus wrote: >> >> Hi Camel PMC, >> >> We have been having an ongoing problem with Jenkins for quite some time, >> where the CPU usage and IOPS skyrocket on the master. Each time this has >> happened, the jenkins build nodes lose all of their associated labels, and >> all new builds are unable to start. >> >> In the times I’ve been able to investigate this, there has in each case been >> several Camel builds running, and in each case, the builds are opening >> somewhere between 1500 and 2200 log files, which seems to be killing the >> jenkins master. For comparison, some very large build jobs for Hadoop only >> open ~15 or so log files. >> >> I have had to take the rather drastic step of disabling the Camel Jenkins >> jobs (many of which have been failing for awhile now) while we continue to >> investigate this issue. Before we re-enable the jobs, we’re going to have to >> figure out how to get your builds to open a sane number of log files — the >> current situation where the builds are creating thousands of log files is >> not sustainable, and we believe this may be one of the causative factors of >> the ongoing jenkins outages. While I cannot say with any certainty that this >> is what’s been killing the master, it’s far enough out of the norm that I >> need to rule it out. >> >> Please do not re-enable any of the disabled builds until we have had a >> chance to work on this together. Can you please identify someone from the >> project to act as a liaison with Infra to troubleshoot the issues with these >> builds? >> >> Since I have sent this to a private list, you have my permission to forward >> this message on to your devs or other public lists as you deem appropriate. >> >> >> Thanks, >> >> -Chris >> ASF Infra >> > > > > -- > Zoran Regvart signature.asc Description: Message signed with OpenPGP
Re: camel builds causing problems with jenkins
Hi Chris, thank you for troubleshooting this, can you clarify one thing for me, when you mention log files, are these the `.log` files generated during the test phase of the build or the XML/TXT files with JUnit reports? I would think that the job type being Maven and the automatic gathering of JUnit test reports is the culprit but would like a confirmation. If it is so, I think we one possible solution is to migrate to Pipeline/Freestyle job type as, as far as I'm aware, there is no way to prevent Maven job type from gathering JUnit reports. zoran On Fri, Feb 16, 2018 at 2:37 AM, Chris Lambertus wrote: > > Hi Camel PMC, > > We have been having an ongoing problem with Jenkins for quite some time, > where the CPU usage and IOPS skyrocket on the master. Each time this has > happened, the jenkins build nodes lose all of their associated labels, and > all new builds are unable to start. > > In the times I’ve been able to investigate this, there has in each case been > several Camel builds running, and in each case, the builds are opening > somewhere between 1500 and 2200 log files, which seems to be killing the > jenkins master. For comparison, some very large build jobs for Hadoop only > open ~15 or so log files. > > I have had to take the rather drastic step of disabling the Camel Jenkins > jobs (many of which have been failing for awhile now) while we continue to > investigate this issue. Before we re-enable the jobs, we’re going to have to > figure out how to get your builds to open a sane number of log files — the > current situation where the builds are creating thousands of log files is not > sustainable, and we believe this may be one of the causative factors of the > ongoing jenkins outages. While I cannot say with any certainty that this is > what’s been killing the master, it’s far enough out of the norm that I need > to rule it out. > > Please do not re-enable any of the disabled builds until we have had a chance > to work on this together. Can you please identify someone from the project to > act as a liaison with Infra to troubleshoot the issues with these builds? > > Since I have sent this to a private list, you have my permission to forward > this message on to your devs or other public lists as you deem appropriate. > > > Thanks, > > -Chris > ASF Infra > -- Zoran Regvart
