So PL, these are just the slaves and not jenkins, right? If we can incorporate some monitoring, that would be nice. (no emotional attachements to my hack has formed yet)
On Thu, Oct 1, 2015 at 2:18 PM, Pierre-Luc Dion <[email protected]> wrote: > Daan: I like your style! > > I was looking at using Chef [1] to managed Jenkins slaves, I've started > deploying it as Proof of concept for some slaves I've recently deployed, > within this scope of idea was to deploy a monitoring tool at some point to > alerts on event such as disk full. > > I was not aware that puppet was already used for automated deployment of > Jenkins. I guest if we are going to managed Jenkins slaves, we all need to > agree on the tool to use with the majority of the community/dev right? > > [1] cookbooks repo: https://github.com/CloudStack-extras/ci-chefrepo > > > > On Wed, Sep 30, 2015 at 6:59 AM, Daan Hoogland <[email protected]> > wrote: > > > So I did, expressing my worst operator practices I created a small > script: > > > > <script> > > ! /bin/bash > > > > THRESHOLD=95 > > USAGE=`/bin/df -P /var/lib/jenkins/workspace | /bin/grep -v Capacity | > > /bin/cut -c 47- | /bin/cut -c 6-7 ` > > > > function maildev() { > > /bin/mail -s 'disk usage at jenkins.bac.o' [email protected] > << > > EOT > > `/bin/df -P /var/lib/jenkins/workspace` > > EOT > > } > > > > [[ $USAGE > $THRESHOLD ]] && maildev > > </script> > > > > and added it to cron: > > > > */10 * * * * ( cd /var/lib/jenkins && ./check_workspace.sh ) >/dev/null > > > > There I noticed some traces of puppet usage. Should I implement this in > > puppet somehow? > > > > willing to do some more, > > Daan > > > > On Wed, Sep 30, 2015 at 11:40 AM, David Nalley <[email protected]> wrote: > > > > > SGTM, Feel free to. > > > > > > > > > > > > On Wed, Sep 30, 2015 at 5:34 AM, Daan Hoogland < > [email protected]> > > > wrote: > > > > H David, I just tried and it works indeed. Forgot about that. Does it > > > make > > > > sense to have a script containing du mail to dev@ if something like > > > > /var/lib/jenkins/workspace or / exceeds a threshold? (yes me > > > volunteering) > > > > > > > > On Wed, Sep 30, 2015 at 10:48 AM, Raja Pullela < > > [email protected]> > > > > wrote: > > > > > > > >> Btw, edison, pradeep, rayeesn - are not active anymore - talluri may > > be? > > > >> I think. > > > >> > > > >> -----Original Message----- > > > >> From: David Nalley [mailto:[email protected]] > > > >> Sent: Wednesday, September 30, 2015 2:01 PM > > > >> To: [email protected] > > > >> Cc: Pierre-Luc Dion <[email protected]>; [email protected] > > > >> Subject: Re: jenkins disk full ? > > > >> > > > >> Right now these folks have access: > > > >> > > > >> ewanm:x:1002:1002::/home/ewanm:/bin/bash > > > >> prasanna:x:1003:1003::/home/prasanna:/bin/bash > > > >> salvatore:x:1004:1004::/home/salvatore:/bin/bash > > > >> edison:x:1005:1005::/home/edison:/bin/bash > > > >> pradeep:x:1006:1006::/home/pradeep:/bin/bash > > > >> rayeesn:x:1007:1007::/home/rayeesn:/bin/bash > > > >> hugot:x:1008:1008::/home/hugot:/bin/bash > > > >> talluri:x:1009:1009::/home/talluri:/bin/bash > > > >> dahn:x:1010:1010::/home/dahn:/bin/bash > > > >> > > > >> PL or other folks who want access, send me a ssh pub key off list. > > > >> > > > >> --David > > > >> > > > >> On Wed, Sep 30, 2015 at 4:20 AM, Rajani Karuturi < > > > >> [email protected]> wrote: > > > >> > Thanks david. Do you know if anyone else has access to this? if > not, > > > can > > > >> you give access to PL please? > > > >> > > > > >> > ~Rajani > > > >> > > > > >> > > > > >> > > > > >> > On 30-Sep-2015, at 1:13 pm, David Nalley <[email protected]> wrote: > > > >> > > > > >> >> The below is before I cleaned up a bit. We now have 3.2GB free. > > > >> >> > > > >> >> [root@domU-12-31-39-15-22-2D jobs]# du -h -d 1 1.3G > > > >> >> ./package-rhel63-master 200M > ./cloudstack-4.3-maven-build-noredist > > > >> >> 638M ./package-rhel63-4.5 > > > >> >> 1.1G ./cloudstack-4.4-systemvm64 > > > >> >> 1.3M ./build-4.5 > > > >> >> 8.0K ./cloudstack-master-with-patch > > > >> >> 186M ./build-test-archive-4.4 > > > >> >> 2.9M ./test-smoke-matrix-master > > > >> >> 100K ./Create-Release-Packages > > > >> >> 1.5M ./test-packaging-4.4 > > > >> >> 214M ./build-test-archive-master > > > >> >> 612K ./cloudstack.apt-get.eu-systemvm 88M ./built-test-archive > 4.2M > > > >> >> ./simulator-4.5-singlerun 424K ./test-tabnanny-4.4 2.2G > > > >> >> ./build-master 124M ./cloudstack-4.4-systemvm 331M > > > >> >> ./package-centos7-4.4.4-noredist 804K > ./cloudstack-4.4-maven-build > > > >> >> 3.4M ./cloudstack-4.3-maven-build 645M ./package-centos7-master > 80M > > > >> >> ./test-fetch-marvin-master 180M ./parameterized-slowbuild 508K > > > >> >> ./cloudstack.apt-get.eu-noredis-rhel > > > >> >> 104K ./dockerbuild-master-marvin > > > >> >> 9.1M ./test-regression-matrix-master > > > >> >> 345M ./cloudstack-4.3-package-deb > > > >> >> 3.8G ./parameterized-sytemvm > > > >> >> 26M ./HyperVAgent4.3 > > > >> >> 510M ./cloudstack-4.3-package-rpm > > > >> >> 1.5M ./test-packaging-master > > > >> >> 7.2M ./build-docs-admin-master > > > >> >> 8.3M ./cloudstack-marvin-master > > > >> >> 1013M ./cloudstack-4.3-systemvm > > > >> >> 131M ./run-checkin-tests > > > >> >> 2.5M ./simulator-gate > > > >> >> 145M ./cloudstack-master-twiceweekly-coverity > > > >> >> 3.2M ./cloudstack-4.4-hyperv-agent > > > >> >> 104K ./test-setup-advancedzone-master 1.1G ./master-slowbuild 44M > > > >> >> ./cloudstack-master 301M ./package-centos7-master-noredist 396K > > > >> >> ./test-tabnanny-master 816K ./simulator-build 20K ./test-libcloud > > > >> >> 236K ./test-smoke-matrix-4.3 4.9M ./dockerbuild-4.5-simulator > 331M > > > >> >> ./package-centos7-4.4-noredist 632K > > > >> >> ./cloudstack.apt-get.eu-noredis-deb > > > >> >> 144K ./dockerbuild-master-management_centos6 > > > >> >> 292M ./package-centos6-4.6 > > > >> >> 9.4M ./stop-jetty > > > >> >> 326M ./cloudstack-4.4-package-rpm > > > >> >> 40K ./mgmt-build-reviewboard-requests 20K > > ./build-test-centos-builtin > > > >> >> 76K ./simulator-deploydb 20K ./Master - Python Static Code > Analysis > > > >> >> 53M ./test-smoke-matrix-4.4 442M ./build-master-slowbuild 376K > > > >> >> ./build-docs-niciranvp-master 840K ./test-matrix-4.4 2.7M > > > >> >> ./dockerbuild-master-simulator 21M ./build-docs-install-master > 50M > > > >> >> ./test-fetch-marvin-4.4 789M > > > >> >> ./cloudstack-rpm-packages-with-branch-parameter > > > >> >> 5.3G ./build-systemvm-4.5 > > > >> >> 1.1G ./cloudstack-4.3-systemvm64 > > > >> >> 3.2M ./build-docs-release-notes-master 8.0K > > > >> >> ./cloudstack-master-codecoverage 121M ./fastsimulatorbuild 4.3M > > > >> >> ./cloudstack-4.4-marvin 352M ./build-master-noredist 7.2M > > > >> >> ./test-setup-advancedzone-4.4 3.3M > ./cloudstack-master-hyperv-agent > > > >> >> 1.9G ./build-systemvm64-master 29M ./start-jetty 3.4M > > > >> >> ./simulator-4.4-integration-tests 264M ./build-4.5-noredist 2.0G > > > >> >> ./package-deb-master 576K ./build-master-simulator 137M > > > >> >> ./mgmt-update-nonoss-libs 11M ./test-matrix-extended-master 235M > > > >> >> ./cloudstack-4.4-maven-build-noredist > > > >> >> 5.3M ./simulator-hotfix-trigger > > > >> >> 8.0K ./cloudstack-maven-build-with-branch-parameter > > > >> >> 2.7M ./test-yumrepo-refresh-master > > > >> >> 1.3G ./cloudstack-4.5-hyperv-agent > > > >> >> 1.6M ./Build-by-Branch > > > >> >> 3.5G ./build-systemvm-master > > > >> >> 369M ./package-deb-4.6 > > > >> >> 172K ./test-maintenance-operations-4.4 48K > > > >> >> ./cloudstack-4.3-package-rpm-noredist > > > >> >> 532K ./build-docs-vxlan-master > > > >> >> 18M ./build-docs-devguide-master > > > >> >> 456K ./test-environment-refresh-master 876K ./test-matrix-master > > 2.1M > > > >> >> ./test-yumrepo-refresh-4.4 51M ./test-environment-refresh-4.4 > 6.3M > > > >> >> ./simulator-singlerun 724M ./package-deb-4.5 16M > > > >> >> ./docs-4.3-gsoc-guide 384K ./test-matrix-extended-4.4 20K > > > >> >> ./pull-requests 204K ./build-docs-midonet-master 358M > > > >> >> ./cloudstack-4.4-package-deb 506M ./build-master-jdk18 2.3M > > > >> >> ./cloudstack.apt-get.eu-noredis-centos7 > > > >> >> 499M ./package-centos7-4.6 > > > >> >> 119M ./build-4.5-simulator > > > >> >> 12K ./cloudstack-4.2-systemvm_test > > > >> >> 1.6M ./test-regression-matrix-4.4 > > > >> >> 144K ./build-systemvm64-master-with-debian8 > > > >> >> 12K ./cloudstack-4.2-systemvm > > > >> >> 12K ./cloudstack-4.2-systemvm64 > > > >> >> 3.1M ./cloudstack-4.4-auto-marvin > > > >> >> 334M ./package-rhel63-master-noredist 6.8M > > > >> >> ./test-maintenance-operations-master > > > >> >> 36G . > > > >> >> > > > >> >> On Tue, Sep 29, 2015 at 9:04 AM, Pierre-Luc Dion < > > > [email protected]> > > > >> wrote: > > > >> >>> David, Do you think you could do another cleanup on the master > of > > > >> j.bac.o ? > > > >> >>> > > > >> >>> Could you list the jobs that consume most of the spaces? > > > >> >>> > > > >> >>> Thanks, > > > >> >>> > > > >> >>> > > > >> >>> On Wed, Sep 23, 2015 at 12:22 AM, Rajani Karuturi > > > >> >>> <[email protected]> wrote: > > > >> >>>> > > > >> >>>> Hi David, > > > >> >>>> disk space issue again. Can you please do a little more > cleanup? > > > >> >>>> > > > >> >>>> > > > >> >>>> > > > http://jenkins.buildacloud.org/computer/coohq-slave-01/executors/0/ > > > >> >>>> causeOfDeath > > > >> >>>> > > > >> >>>> ~Rajani > > > >> >>>> > > > >> >>>> > > > >> >>>> > > > >> >>>> On 18-Sep-2015, at 11:59 pm, David Nalley <[email protected]> > wrote: > > > >> >>>> > > > >> >>>>> I freed up a little bit of space, but we'll have to do > something > > > >> >>>>> different in the long term. > > > >> >>>>> > > > >> >>>>> --David > > > >> >>>>> > > > >> >>>>> On Fri, Sep 18, 2015 at 7:37 AM, Pierre-Luc Dion > > > >> >>>>> <[email protected]> > > > >> >>>>> wrote: > > > >> >>>>>> Hi, > > > >> >>>>>> > > > >> >>>>>> Srikanteswararao (or any one having access to jenkins > server), > > do > > > >> >>>>>> you think you could do some more magic here? We ran out of > > space > > > >> >>>>>> again on the master jenkins server for j.bac.o. > > > >> >>>>>> > > > >> >>>>>> Thanks, > > > >> >>>>>> > > > >> >>>>>> PL > > > >> >>>>>> > > > >> >>>>>> On Mon, Jun 1, 2015 at 4:26 PM, Pierre-Luc Dion > > > >> >>>>>> <[email protected]> > > > >> >>>>>> wrote: > > > >> >>>>>> > > > >> >>>>>>> thanks Srikanteswararao, it worked and we have centos63 > rpm > > > >> >>>>>>> now for master branch! > > > >> >>>>>>> > > > >> >>>>>>> > > > >> >>>>>>> > > > >> >>>>>>> *Pierre-Luc DION* > > > >> >>>>>>> Architecte de Solution Cloud | Cloud Solutions Architect t > > > >> >>>>>>> 855.652.5683 > > > >> >>>>>>> > > > >> >>>>>>> *CloudOps* Votre partenaire infonuagique* | *Cloud Solutions > > > >> >>>>>>> Experts > > > >> >>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w > > cloudops.com > > > >> >>>>>>> *|* tw @CloudOps_ > > > >> >>>>>>> > > > >> >>>>>>> On Mon, Jun 1, 2015 at 3:09 PM, Srikanteswararao talluri < > > > >> >>>>>>> [email protected]> wrote: > > > >> >>>>>>> > > > >> >>>>>>>> There is no space . > > > >> >>>>>>>> > > > >> >>>>>>>> /dev/xvdf1 50G 47G 0 100% > > > /var/lib/jenkins/workspace > > > >> >>>>>>>> > > > >> >>>>>>>> I have cleaned up builds for 'fastsimulatorbuild' job which > > > >> >>>>>>>> were using 9G. > > > >> >>>>>>>> Modified the job to delete old builds and artefacts after > 30 > > > >> >>>>>>>> builds/30days. > > > >> >>>>>>>> > > > >> >>>>>>>> > > > >> >>>>>>>> ~Talluri > > > >> >>>>>>>> > > > >> >>>>>>>> On Mon, Jun 1, 2015 at 8:14 PM, Pierre-Luc Dion > > > >> >>>>>>>> <[email protected]> > > > >> >>>>>>>> wrote: > > > >> >>>>>>>> > > > >> >>>>>>>>> Can someone that have access to the principal jenkins > server > > > >> >>>>>>>>> to check > > > >> >>>>>>>> disk > > > >> >>>>>>>>> space left? > > > >> >>>>>>>>> > > > >> >>>>>>>>> getting this error on few slave, the one I've check does > not > > > >> >>>>>>>>> have disk > > > >> >>>>>>>> full > > > >> >>>>>>>>> > > > >> >>>>>>>>> > > > >> >>>>>>>>> Thread has died > > > >> >>>>>>>>> > > > >> >>>>>>>>> java.io.IOException: No space left on device > > > >> >>>>>>>>> at java.io.FileOutputStream.writeBytes(Native > Method) > > > >> >>>>>>>>> at > > > >> java.io.FileOutputStream.write(FileOutputStream.java:318) > > > >> >>>>>>>>> at > > > >> >>>>>>>>> > sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) > > > >> >>>>>>>>> at > > > >> sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:316) > > > >> >>>>>>>>> at > > > sun.nio.cs.StreamEncoder.close(StreamEncoder.java:149) > > > >> >>>>>>>>> at > > > >> >>>>>>>>> > > java.io.OutputStreamWriter.close(OutputStreamWriter.java:233) > > > >> >>>>>>>>> at > > java.io.BufferedWriter.close(BufferedWriter.java:266) > > > >> >>>>>>>>> at > > > >> >>>>>>>>> > hudson.util.AtomicFileWriter.close(AtomicFileWriter.java:94) > > > >> >>>>>>>>> at > > > >> >>>>>>>> > > hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:109) > > > >> >>>>>>>>> at hudson.util.TextFile.write(TextFile.java:121) > > > >> >>>>>>>>> at > hudson.model.Job.saveNextBuildNumber(Job.java:274) > > > >> >>>>>>>>> at hudson.model.Job.assignBuildNumber(Job.java:332) > > > >> >>>>>>>>> at hudson.model.Run.<init>(Run.java:286) > > > >> >>>>>>>>> at > > > >> hudson.model.AbstractBuild.<init>(AbstractBuild.java:167) > > > >> >>>>>>>>> at hudson.model.Build.<init>(Build.java:92) > > > >> >>>>>>>>> at > > > >> hudson.model.FreeStyleBuild.<init>(FreeStyleBuild.java:34) > > > >> >>>>>>>>> at > > > >> >>>>>>>>> > > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > > > >> >>>>>>>>> Method) > > > >> >>>>>>>>> at > > > >> >>>>>>>>> > > > >> >>>>>>>> > > > >> >>>>>>>> > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeCon > > > >> >>>>>>>> structorAccessorImpl.java:57) > > > >> >>>>>>>>> at > > > >> >>>>>>>>> > > > >> >>>>>>>> > > > >> >>>>>>>> > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Deleg > > > >> >>>>>>>> atingConstructorAccessorImpl.java:45) > > > >> >>>>>>>>> at > > > >> >>>>>>>> > > java.lang.reflect.Constructor.newInstance(Constructor.java:525) > > > >> >>>>>>>>> at > > > >> >>>>>>>>> > > > >> jenkins.model.lazy.LazyBuildMixIn.newBuild(LazyBuildMixIn.java:186) > > > >> >>>>>>>>> at > > > >> >>>>>>>> > > hudson.model.AbstractProject.newBuild(AbstractProject.java:1000 > > > >> >>>>>>>> ) > > > >> >>>>>>>>> at > > > >> >>>>>>>>> > > > >> >>>>>>>>> > > > >> > > hudson.model.AbstractProject.createExecutable(AbstractProject.java:1193) > > > >> >>>>>>>>> at > > > >> >>>>>>>>> > > > >> >>>>>>>>> > > > >> > > hudson.model.AbstractProject.createExecutable(AbstractProject.java:143) > > > >> >>>>>>>>> at hudson.model.Executor.run(Executor.java:213) > > > >> >>>>>>>>> > > > >> >>>>>>>>> more info > > > >> >>>>>>>>> <http://wiki.jenkins-ci.org/display/JENKINS/Dead+Executor > > > > > >> >>>>>>>>> > > > >> >>>>>>>>> > > > >> >>>>>>>>> > > > >> >>>>>>>>> ref: > > > >> >>>>>>>>> > > > >> >>>>>>>> > > > >> >>>>>>>> > > > http://jenkins.buildacloud.org/computer/cca-slave-02/executors/ > > > >> >>>>>>>> 0/causeOfDeath > > > >> >>>>>>>>> > > > >> >>>>>>>>> > > > >> >>>>>>>>> > > > >> >>>>>>>>> Thanks! > > > >> >>>>>>>>> > > > >> >>>>>>>> > > > >> >>>>>>> > > > >> >>>>>>> > > > >> >>>> > > > >> >>> > > > >> > > > > >> > > > > > > > > > > > > > > > > -- > > > > Daan > > > > > > > > > > > -- > > Daan > > > -- Daan
