> On Mar 9, 2010, at 3:36 PM, Robert Joly wrote: > > >> > >> > >> On Mar 9, 2010, at 1:16 PM, Robert Joly wrote: > >> > >>> Mardy wrote: > >>>> > >>>> > >>>> On Mar 9, 2010, at 10:07 AM, Robert Joly wrote: > >>>> > >>>>> Hi guys, > >>>>> I have been investigating XX-7634 which reports high CPU > >>>> utilization > >>>>> by java processes after an ISO install of the commercial > >>>> version (SCS) > >>>>> which uses the IBM JVM. > >>>>> > >>>>> Basically, after an ISO install I'm seeing that *all* the > >>>> java-based > >>>>> processes chew up between 50% and 100% of one processor and > >>>> remained > >>>>> like that for as long as I kept the box up (few hours). > >>>> The processes > >>>>> in question are sipXpage, sipXivr, sipXrelay, sipXconfig, > >>>> sipXrest and > >>>>> sipXprovision. Using jconsole I was able to find that the > >>>> hot thread > >>>>> for each of these services is called 'Attach Handler' > >>>>> (com.ibm.tools.attach.javaSE.AttachHandler.run()) and I > >> also found > >>>>> that I can eliminate the high CPU condition completely > on a fresh > >>>>> install by hand-editing the launch command for each of > these java > >>>>> processes to add the following property: > >>>>> -Dcom.ibm.tools.attach.enable=no > >>>>> > >>>>> If I add this property then all the processes are > >>>> well-behaved but I > >>>>> do not understand the fundamental reason why the hot thread > >>>> is there > >>>>> in the first place. I'm therefore turning to the Java gods > >>>> that are > >>>>> tuned in to this list to see if they had previous > >>>> encounters with this. > >>>>> > >>>>> Thanks in advance, > >>>>> bob > >>>> > >>>> I would like to know why this issue has all of a sudden > >> shown up on > >>>> the radar. The Attach API and supporting AttachHandler > thread was > >>>> introduced, as a result of an upgrade to the IBM JVM, back on > >>>> 2009-11-14 Is it possible that the high CPU utilization > has been > >>>> there since then but no one had noticed it until now? > >>> > >>> Refresh my memory, will you? Why are we using IBM in the > >> first place? > >> > >> Initially because it was the only option for supporting one of our > >> customers. In addition, we have discovered that it offers > some very > >> attractive memory optimization features not offered by other JVM's > >> that we may need to employ as the number of Java based services > >> increase. > >> > >>> > >>> I do not know when the high CPU behavior started showing up > >> but it was > >>> first reported on 2010-02-10. Also, not every system > exhibits the > >>> behavior. For example, our friends at Qantom can I can > >> reproduce the > >>> problem but Al Campbell and Chris Parfitt cannot on their > systems. > >>> I'm using a Dell R300 and Qantom is using Dell Optiplex and > >> so do Al > >>> and Chris. I have not identified the ingredient that makes > >> this hot > >>> thread appear and as far as I can tell, IBM does not publish the > >>> source code for their attach implementation. > >>> > >>> The thread seems to be looping around waiting for a > >> semaphore. Please > >>> see the attached screenshot for visual of the stacktrace. > >>> > >> > >> Is this actually causing a problem or is it just a red > herring? If > >> it is in fact impacting the performance of the system, > then disable > >> it. > > > > > > Here's a sample of top running on a bad system. This is > running on a > > quad-core machine and 60% of it is spent in the kernel and > 17% spent > > in user space. I'm not sure which way the priorities go but I'm > > hoping that the processes with lower number have higher priority... > > > > Would you agree that this is a problematic case? > > Is that a view of an idle system or one that is under heavy > load? If the system is idle, then no conclusion can be drawn > from that data. > > I suggest that you take the safe route and just disable the > Attach API.
I agree with this suggestion. I could modify the startup script for each java process to add a -Dcom.ibm.tools.attach.enable=no argument but this would not be optimal as new java processes that get introduce later may forget to do this. Instead, I was toying with the idea of modifying the 'sipx-config' script that is used to generate the string to use to launch 'java' (i.e. /usr/bin/java). Every startup script for java processes invoke it and it appears to me that it would be a good central place to put my -Dcom.ibm.tools.attach.enable=no argument so that it gets applied to all present and future java processes. Comments? > -----Original Message----- > From: Mardy Marshall [mailto:[email protected]] > Sent: Tuesday, March 09, 2010 4:59 PM > To: Joly, Robert AVAYA (CAR:9D30) > Cc: sipX-dev > Subject: Re: [sipX-dev] High CPU utilization with Java - help required > > > -----Original Message----- > From: Mardy Marshall [mailto:[email protected]] > Sent: Tuesday, March 09, 2010 4:59 PM > To: Joly, Robert AVAYA (CAR:9D30) > Cc: sipX-dev > Subject: Re: [sipX-dev] High CPU utilization with Java - help required > > _______________________________________________ sipx-dev mailing list [email protected] List Archive: http://list.sipfoundry.org/archive/sipx-dev Unsubscribe: http://list.sipfoundry.org/mailman/listinfo/sipx-dev sipXecs IP PBX -- http://www.sipfoundry.org/
