Are there any errors/exceptions in Jenkins log files or stdout/stderr? -- Sami
Andrew Melo <[email protected]> kirjoitti 6.2.2013 kello 22.13: > Hey everyone, > > Sorry for resurrecting that thread, but are there any solutions for > this problem? It's basically killing our instance often. > > best, > Andrew > > On Wed, Aug 1, 2012 at 11:47 AM, Andrew Melo <[email protected]> wrote: >> Oh, wow, I didn't notice, but jenkins has autopopulated a user for >> everyone that ever committed on the project. There's like 30 people, >> ~4000 commits, so I could see why that would take a while :) >> >> On Wed, Aug 1, 2012 at 11:42 AM, Andrew Melo <[email protected]> wrote: >>> On Wed, Aug 1, 2012 at 11:37 AM, Slide <[email protected]> wrote: >>>> Can you gist your global config.xml and something from one of your >>>> jobs as well? Please remember to sanitize it. >>> >>> We actually keep it stored in SCM. https://github.com/dmwm/jenkins/ >>> >>> And the following is the gist for the job we run each commit (didn't >>> make it in for some reason...) >>> >>> https://gist.github.com/3228599 >>> >>>> >>>> On Wed, Aug 1, 2012 at 9:34 AM, Andrew Melo <[email protected]> wrote: >>>>> On Wed, Aug 1, 2012 at 11:26 AM, Slide <[email protected]> wrote: >>>>>> No, because its only looking for the email address because it wants to >>>>>> send an email to that user. >>>>> >>>>> I don't know who's getting emailed. I don't remember setting it up for >>>>> anything, and we actually wrote some scripts that turn jenkins >>>>> success/failures into Github issues, so having jenkins also send >>>>> emails would be redundant. >>>>> >>>>> I don't supposed there's a global flag to disable email? (I don't see >>>>> one at "manage jenkins") >>>>> >>>>> -Andrew >>>>>> >>>>>> On Wed, Aug 1, 2012 at 9:24 AM, Andrew Melo <[email protected]> >>>>>> wrote: >>>>>>> On Wed, Aug 1, 2012 at 11:22 AM, Slide <[email protected]> wrote: >>>>>>>> This is a huge issue with the email-ext plugin as well when it does >>>>>>>> email address resolution. Quite a number of people have complained >>>>>>>> about how long it takes. I have yet to come up with a good solution. >>>>>>>> The perforce plugin has a similar issue. >>>>>>> >>>>>>> If I just stick a dummy address in every user's profile, will that work? >>>>>>> >>>>>>>> >>>>>>>> slide >>>>>>>> >>>>>>>> On Wed, Aug 1, 2012 at 9:17 AM, Vojtech Juranek <[email protected]> >>>>>>>> wrote: >>>>>>>>> Looks like you it does search for user's email: >>>>>>>>> hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor >>>>>>>>> and spends time parsing changelogs: >>>>>>>>> hudson.scm.SubversionChangeLogParser.parse >>>>>>>>> >>>>>>>>> I guess you have quite large instance, otherwise this operation would >>>>>>>>> be quite >>>>>>>>> fast. >>>>>>>>> If you have some job, which has set up option to send an email to >>>>>>>>> devs who >>>>>>>>> broke the build, if the user hasn't specified an email, Jenkins tries >>>>>>>>> to find it >>>>>>>>> e.g. in git or SVN changelogs and search all projects and builds so >>>>>>>>> if you >>>>>>>>> have large instance with several dozen thousands of builds if can >>>>>>>>> take pretty >>>>>>>>> long time. >>>>>>>>> >>>>>>>>> You can fix it by setting up correct email for the user. >>>>>>>>> If you have installed git plugin, make sure you have 1.1.16 (I hope >>>>>>>>> it was >>>>>>>>> fixed in this version) or higher. Git plugin made this search even if >>>>>>>>> the user >>>>>>>>> has set up email correctly >>>>>>>>> >>>>>>>>> On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: >>>>>>>>>> On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek >>>>>>>>>> <[email protected]> >>>>>>>>> wrote: >>>>>>>>>>> On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: >>>>>>>>>>>> On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek >>>>>>>>>>>> <[email protected]> >>>>>>>>> wrote: >>>>>>>>>>>>> quick way how to look what the thread consuming CPU is doing is >>>>>>>>>>>>> to do >>>>>>>>>>>>> thread dump (e.g. using jstack $PID) and use top with threads on >>>>>>>>>>>>> (H >>>>>>>>>>>>> option) and then look up, see e.g. >>>>>>>>>>>>> http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- >>>>>>>>>>>>> consuming-most-cpu/ >>>>>>>>>>>> >>>>>>>>>>>> I see. I apparently don't have jstack on this machine :/. Does it >>>>>>>>>>>> only >>>>>>>>>>>> come with the JDK, or can I find it somewhere on the JRE? Once I >>>>>>>>>>>> find >>>>>>>>>>>> the offending thread, should it be pretty obvious what it does? >>>>>>>>>>> >>>>>>>>>>> jstack is part of JDK >>>>>>>>>>> >>>>>>>>>>> you can see the stack trace via Jenkins UI navigating to >>>>>>>>>>> $JENKINS_URL/threadDump but not sure if your (or any) Jenkins >>>>>>>>>>> version >>>>>>>>>>> provides thread IDs. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Once you identify the offending thread, it should be obvious what >>>>>>>>>>> it does >>>>>>>>>>> (but it may not be obvious why it does what it does:-) >>>>>>>>>> >>>>>>>>>> Okay, I installed the jdk, and I looked some more. >>>>>>>>>> >>>>>>>>>> Using top, I see one jenkins thread taking the lionsshare of the >>>>>>>>>> time: >>>>>>>>>> >>>>>>>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >>>>>>>>>> 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java >>>>>>>>>> 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java >>>>>>>>>> 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java >>>>>>>>>> 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java >>>>>>>>>> 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java >>>>>>>>>> 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java >>>>>>>>>> 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java >>>>>>>>>> 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java >>>>>>>>>> 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java >>>>>>>>>> 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java >>>>>>>>>> 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java >>>>>>>>>> 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java >>>>>>>>>> 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java >>>>>>>>>> 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java >>>>>>>>>> 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java >>>>>>>>>> 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java >>>>>>>>>> 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java >>>>>>>>>> 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java >>>>>>>>>> 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java >>>>>>>>>> 24973 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:18.32 java >>>>>>>>>> 24983 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:07.91 java >>>>>>>>>> 24838 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:38.35 java >>>>>>>>>> 24845 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:32.56 java >>>>>>>>>> 25037 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:16.63 java >>>>>>>>>> 25038 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:07.00 java >>>>>>>>>> 24491 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:05.38 java >>>>>>>>>> 24611 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:02.82 java >>>>>>>>>> 24488 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:00.30 java >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Then if I run jstack, I get the following backtrace: >>>>>>>>>> >>>>>>>>>> https://gist.github.com/3228105 >>>>>>>>>> >>>>>>>>>> Does that look useful at all? >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Andrew >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Website: http://earl-of-code.com >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> -- >>>>>>> Andrew Melo >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Website: http://earl-of-code.com >>>>> >>>>> >>>>> >>>>> -- >>>>> -- >>>>> Andrew Melo >>>> >>>> >>>> >>>> -- >>>> Website: http://earl-of-code.com >>> >>> >>> >>> -- >>> -- >>> Andrew Melo >> >> >> >> -- >> -- >> Andrew Melo > > > > -- > -- > Andrew Melo > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
