I just see these: Feb 6, 2013 9:50:10 PM hudson.plugins.im.IMConnectionProvider$ConnectorRunnable run INFO: Reconnect failed. Next connection attempt in 16 minutes Feb 6, 2013 9:51:40 PM hudson.slaves.SlaveComputer tryReconnect INFO: Attempting to reconnect dmwm-agent-int-old Feb 6, 2013 10:01:53 PM winstone.Logger logInternal WARNING: Called getInputStream after getParameter ... error Feb 6, 2013 10:02:19 PM winstone.Logger logInternal WARNING: Called getInputStream after getParameter ... error Feb 6, 2013 10:02:21 PM winstone.Logger logInternal WARNING: Called getInputStream after getParameter ... error
(UTC timestamps) While running a 3 matrix jobs with 10 configurations each and 10 slaves (with the web browser open), I get the following utilization PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 15209 jenkins 18 0 3366m 690m 18m S 177.2 22.9 203:51.39 java and the web interface is painfully slow. thanks, Andrew On Wed, Feb 6, 2013 at 2:57 PM, Sami Tikka <[email protected]> wrote: > Are there any errors/exceptions in Jenkins log files or stdout/stderr? > > -- Sami > > Andrew Melo <[email protected]> kirjoitti 6.2.2013 kello 22.13: > >> Hey everyone, >> >> Sorry for resurrecting that thread, but are there any solutions for >> this problem? It's basically killing our instance often. >> >> best, >> Andrew >> >> On Wed, Aug 1, 2012 at 11:47 AM, Andrew Melo <[email protected]> wrote: >>> Oh, wow, I didn't notice, but jenkins has autopopulated a user for >>> everyone that ever committed on the project. There's like 30 people, >>> ~4000 commits, so I could see why that would take a while :) >>> >>> On Wed, Aug 1, 2012 at 11:42 AM, Andrew Melo <[email protected]> wrote: >>>> On Wed, Aug 1, 2012 at 11:37 AM, Slide <[email protected]> wrote: >>>>> Can you gist your global config.xml and something from one of your >>>>> jobs as well? Please remember to sanitize it. >>>> >>>> We actually keep it stored in SCM. https://github.com/dmwm/jenkins/ >>>> >>>> And the following is the gist for the job we run each commit (didn't >>>> make it in for some reason...) >>>> >>>> https://gist.github.com/3228599 >>>> >>>>> >>>>> On Wed, Aug 1, 2012 at 9:34 AM, Andrew Melo <[email protected]> wrote: >>>>>> On Wed, Aug 1, 2012 at 11:26 AM, Slide <[email protected]> wrote: >>>>>>> No, because its only looking for the email address because it wants to >>>>>>> send an email to that user. >>>>>> >>>>>> I don't know who's getting emailed. I don't remember setting it up for >>>>>> anything, and we actually wrote some scripts that turn jenkins >>>>>> success/failures into Github issues, so having jenkins also send >>>>>> emails would be redundant. >>>>>> >>>>>> I don't supposed there's a global flag to disable email? (I don't see >>>>>> one at "manage jenkins") >>>>>> >>>>>> -Andrew >>>>>>> >>>>>>> On Wed, Aug 1, 2012 at 9:24 AM, Andrew Melo <[email protected]> >>>>>>> wrote: >>>>>>>> On Wed, Aug 1, 2012 at 11:22 AM, Slide <[email protected]> wrote: >>>>>>>>> This is a huge issue with the email-ext plugin as well when it does >>>>>>>>> email address resolution. Quite a number of people have complained >>>>>>>>> about how long it takes. I have yet to come up with a good solution. >>>>>>>>> The perforce plugin has a similar issue. >>>>>>>> >>>>>>>> If I just stick a dummy address in every user's profile, will that >>>>>>>> work? >>>>>>>> >>>>>>>>> >>>>>>>>> slide >>>>>>>>> >>>>>>>>> On Wed, Aug 1, 2012 at 9:17 AM, Vojtech Juranek <[email protected]> >>>>>>>>> wrote: >>>>>>>>>> Looks like you it does search for user's email: >>>>>>>>>> hudson.scm.SubversionMailAddressResolverImpl.findMailAddressFor >>>>>>>>>> and spends time parsing changelogs: >>>>>>>>>> hudson.scm.SubversionChangeLogParser.parse >>>>>>>>>> >>>>>>>>>> I guess you have quite large instance, otherwise this operation >>>>>>>>>> would be quite >>>>>>>>>> fast. >>>>>>>>>> If you have some job, which has set up option to send an email to >>>>>>>>>> devs who >>>>>>>>>> broke the build, if the user hasn't specified an email, Jenkins >>>>>>>>>> tries to find it >>>>>>>>>> e.g. in git or SVN changelogs and search all projects and builds so >>>>>>>>>> if you >>>>>>>>>> have large instance with several dozen thousands of builds if can >>>>>>>>>> take pretty >>>>>>>>>> long time. >>>>>>>>>> >>>>>>>>>> You can fix it by setting up correct email for the user. >>>>>>>>>> If you have installed git plugin, make sure you have 1.1.16 (I hope >>>>>>>>>> it was >>>>>>>>>> fixed in this version) or higher. Git plugin made this search even >>>>>>>>>> if the user >>>>>>>>>> has set up email correctly >>>>>>>>>> >>>>>>>>>> On Wednesday 01 August 2012 10:56:43 Andrew Melo wrote: >>>>>>>>>>> On Wed, Aug 1, 2012 at 10:36 AM, Vojtech Juranek >>>>>>>>>>> <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>>>> On Wednesday 01 August 2012 10:07:15 Andrew Melo wrote: >>>>>>>>>>>>> On Wed, Aug 1, 2012 at 9:48 AM, Vojtech Juranek >>>>>>>>>>>>> <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>>>>>> quick way how to look what the thread consuming CPU is doing is >>>>>>>>>>>>>> to do >>>>>>>>>>>>>> thread dump (e.g. using jstack $PID) and use top with threads on >>>>>>>>>>>>>> (H >>>>>>>>>>>>>> option) and then look up, see e.g. >>>>>>>>>>>>>> http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is- >>>>>>>>>>>>>> consuming-most-cpu/ >>>>>>>>>>>>> >>>>>>>>>>>>> I see. I apparently don't have jstack on this machine :/. Does it >>>>>>>>>>>>> only >>>>>>>>>>>>> come with the JDK, or can I find it somewhere on the JRE? Once I >>>>>>>>>>>>> find >>>>>>>>>>>>> the offending thread, should it be pretty obvious what it does? >>>>>>>>>>>> >>>>>>>>>>>> jstack is part of JDK >>>>>>>>>>>> >>>>>>>>>>>> you can see the stack trace via Jenkins UI navigating to >>>>>>>>>>>> $JENKINS_URL/threadDump but not sure if your (or any) Jenkins >>>>>>>>>>>> version >>>>>>>>>>>> provides thread IDs. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Once you identify the offending thread, it should be obvious what >>>>>>>>>>>> it does >>>>>>>>>>>> (but it may not be obvious why it does what it does:-) >>>>>>>>>>> >>>>>>>>>>> Okay, I installed the jdk, and I looked some more. >>>>>>>>>>> >>>>>>>>>>> Using top, I see one jenkins thread taking the lionsshare of the >>>>>>>>>>> time: >>>>>>>>>>> >>>>>>>>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >>>>>>>>>>> 24580 jenkins 25 0 3246m 743m 18m R 88.6 24.7 790:53.39 java >>>>>>>>>>> 24591 jenkins 15 0 3246m 743m 18m S 0.0 24.7 40:14.51 java >>>>>>>>>>> 25163 jenkins 15 0 3246m 743m 18m S 0.0 24.7 28:21.42 java >>>>>>>>>>> 24601 jenkins 15 0 3246m 743m 18m S 0.0 24.7 27:24.95 java >>>>>>>>>>> 24581 jenkins 15 0 3246m 743m 18m S 0.0 24.7 26:39.58 java >>>>>>>>>>> 24589 jenkins 18 0 3246m 743m 18m S 0.2 24.7 24:41.60 java >>>>>>>>>>> 24604 jenkins 15 0 3246m 743m 18m S 0.0 24.7 23:47.46 java >>>>>>>>>>> 24603 jenkins 15 0 3246m 743m 18m S 0.6 24.7 17:05.23 java >>>>>>>>>>> 24484 jenkins 15 0 3246m 743m 18m S 0.4 24.7 14:45.39 java >>>>>>>>>>> 24612 jenkins 18 0 3246m 743m 18m S 0.0 24.7 11:50.45 java >>>>>>>>>>> 24610 jenkins 15 0 3246m 743m 18m S 0.0 24.7 10:34.41 java >>>>>>>>>>> 24564 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:56.60 java >>>>>>>>>>> 24602 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:30.98 java >>>>>>>>>>> 24565 jenkins 16 0 3246m 743m 18m S 11.5 24.7 8:22.85 java >>>>>>>>>>> 24609 jenkins 15 0 3246m 743m 18m S 0.0 24.7 8:12.30 java >>>>>>>>>>> 24582 jenkins 15 0 3246m 743m 18m S 0.6 24.7 3:48.67 java >>>>>>>>>>> 24590 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:24.27 java >>>>>>>>>>> 24579 jenkins 15 0 3246m 743m 18m S 0.0 24.7 3:22.16 java >>>>>>>>>>> 24486 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:33.77 java >>>>>>>>>>> 24973 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:18.32 java >>>>>>>>>>> 24983 jenkins 15 0 3246m 743m 18m S 0.0 24.7 2:07.91 java >>>>>>>>>>> 24838 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:38.35 java >>>>>>>>>>> 24845 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:32.56 java >>>>>>>>>>> 25037 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:16.63 java >>>>>>>>>>> 25038 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:07.00 java >>>>>>>>>>> 24491 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:05.38 java >>>>>>>>>>> 24611 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:02.82 java >>>>>>>>>>> 24488 jenkins 15 0 3246m 743m 18m S 0.0 24.7 1:00.30 java >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Then if I run jstack, I get the following backtrace: >>>>>>>>>>> >>>>>>>>>>> https://gist.github.com/3228105 >>>>>>>>>>> >>>>>>>>>>> Does that look useful at all? >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Andrew >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Website: http://earl-of-code.com >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> -- >>>>>>>> Andrew Melo >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Website: http://earl-of-code.com >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> -- >>>>>> Andrew Melo >>>>> >>>>> >>>>> >>>>> -- >>>>> Website: http://earl-of-code.com >>>> >>>> >>>> >>>> -- >>>> -- >>>> Andrew Melo >>> >>> >>> >>> -- >>> -- >>> Andrew Melo >> >> >> >> -- >> -- >> Andrew Melo >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > > -- -- Andrew Melo -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
