Re: [Reactor For CF] Been out of touch, but trying to get back

Wed, 24 May 2006 14:12:37 -0700

Try <cfdump
> var="#GetMetricData('perf_monitor')#" /> (single quotes around perf_monitor)... or was that just a typo in your mail?

W.

On 5/24/06, Doug Hughes < [EMAIL PROTECTED]> wrote:
> Mike,
>
> FYI - I thought I turned on metrics but calling <cfdump
> var="#GetMetricData("perf_monitor")#" /> results in an error saying that the
> service isn't running:
>
> The metrics service is not available.
> This exception is usually caused by service startup failure. Please check
> your server configuration.
>
> The error occurred in
> C:\Inetpub\DougHughes\www.doughughes.net\live\www\test2.cfm: line 1
>
> 1 : <cfdump var="#GetMetricData("perf_monitor")#" />
>
> Would you be interested in me trying to figure out what's wrong with this
> and restarting the server?  This would restart our memory leak though.
>
> FYI, I downloaded the log-so-far and ran it through hpjtune.  All of the
> graphs had an obvious upward angle to them.  Attached is a screenshot of one
> that I think relates to this problem.
>
> I'm also trying to get jprobe working with JRun to see if maybe I can
> visualize the heap a bit.  (Not optimistic, but it's worth a shot).
>
> Thanks,
>
> Doug
>
>
>
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto: [EMAIL PROTECTED]] On Behalf
> Of Mike Brunt - Webapper Services, LLC
> Sent: Wednesday, May 24, 2006 2:17 PM
> To: [email protected]
> Subject: RE: [Reactor For CF] Been out of touch, but trying to get back
>
> Apologies on that Doug, typo.
>
> Kind Regards - Mike Brunt
> CTO and Lead Architect - Webapper Services LLC
> Tel 562.243.6255
> Web Site: http://www.webapper.com
> Blog: http://www.webapper.net
> Tools: http://www.seefusion.com
>
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf
> Of Doug Hughes
> Sent: Wednesday, May 24, 2006 11:05 AM
> To: [email protected]
> Subject: RE: [Reactor For CF] Been out of touch, but trying to get back
>
> I forgot to mention, I changed Xms-512m to -Xms512m.  JRun wouldn't start
> otherwise.
>
> Doug
>
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto: [EMAIL PROTECTED]] On Behalf
> Of Mike Brunt - Webapper Services, LLC
> Sent: Wednesday, May 24, 2006 12:42 PM
> To: [email protected]
> Subject: RE: [Reactor For CF] Been out of touch, but trying to get back
>
> When you get the logs either shoot me a copy or if they are too big I'll
> give you a private ftp server we have.  I'll tell you what we find.
>
> Kind Regards - Mike Brunt
> CTO and Lead Architect - Webapper Services LLC Tel 562.243.6255 Web Site:
> http://www.webapper.com
> Blog: http://www.webapper.net
> Tools: http://www.seefusion.com
>
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf
> Of Doug Hughes
> Sent: Wednesday, May 24, 2006 9:30 AM
> To: [email protected]
> Subject: RE: [Reactor For CF] Been out of touch, but trying to get back
>
> Mike -
>
> The server has 3.5 gigs of ram.
>
> I made the changes to the jvm.config as requested.  Note: I only have one
> jvm.config.  All of the other instances share the same jvm.config too.  I've
> only restarted my instances so hopefully that doesn't impact your research.
> I've never quite figured out how to give a particular instance its own
> jvm.config.
>
> I made the change to log metrics in the Jrun.xml file for the troubled CF
> instance.
>
> I presume I'll watch these logs?
>
> Doug
>
>
>
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf
> Of Mike Brunt - Webapper Services, LLC
> Sent: Wednesday, May 24, 2006 11:14 AM
> To: [email protected]
> Subject: RE: [Reactor For CF] Been out of touch, but trying to get back
>
> Ok Doug, we specialize in tracking down memory leaks along with other things
> and as I have been totally useless one way or another to your Reactor
> project being only a spectator; I think it is time I tried to be
> contributory, if I can.
>
> So I asked for you jvm.config to see what your settings are and if you would
> accept my help I need you first to do the following:
>
> Currently your Java Arguments are as follows:
>
> java.args=-server -Xmx512m -Dsun.io.useCanonCaches=false
> -XX:MaxPermSize=128m -XX:+UseParallelGC -DJINTEGRA_NATIVE_MODE
> -DJINTEGRA_PREFETCH_ENUMS -Dcoldfusion.rootDir={application.home}/
>
> My first suggestion is to change slightly to these:
>
> java.args=-server Xms-512m -Xmx512m -Dsun.io.useCanonCaches=false
> -XX:PermSize=128m -XX:MaxPermSize=128m -XX:+UseParallelGC
> -DJINTEGRA_NATIVE_MODE -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
> -XX:+PrintHeapAtGC -verbose:gc -Xloggc: reactorgc.log
> -DJINTEGRA_PREFETCH_ENUMS -Dcoldfusion.rootDir={application.home}/
>
> Adding Xms-512m and -XX:PermSize=128m will start the heap up with more
> breathing space.  Incidentally, how much RAM do you have on this box?
>
> Adding -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC
> -verbose:gc -Xloggc:reactorgc.log will print Garbage Collection details to
> the -out.log and to a separate reactorgc.log which I will look at to see
> what is going on with memory in the heap.
>
> I also recommend turning on metrics logging in Jrun.xml:
>
> Uncomment:
>
> <service class=" coldfusion.server.jrun4.metrics.MetricsServiceAdapter"
> name="MetricsService">
>     <attribute name="bindToJNDI">true</attribute>
>   </service>
>
> Add or change these settings:
>
> <attribute name="metricsEnabled">true</attribute>
>     <attribute name="metricsLogFrequency">60</attribute>
>     <attribute name="metricsFormat">Web threads
> (busy/total/listen/idle/delay/handled):
> {busyTh}/{totalTh}/{listenTh}/{idleTh}/{delayRq}/{handledRq} Sessions:
> {sessions} Total Memory={totalMemory} Free={freeMemory}</attribute>
>
> These logging changes are temporary till we track down the issue from
> looking at those logs.  These changes will need a CF-JRun restart by the
> way.
>
> Kind Regards - Mike Brunt
> CTO and Lead Architect - Webapper Services LLC Tel 562.243.6255 Web Site:
> http://www.webapper.com
> Blog: http://www.webapper.net
> Tools: http://www.seefusion.com
>
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf
> Of Doug Hughes
> Sent: Wednesday, May 24, 2006 7:47 AM
> To: [email protected]
> Subject: RE: [Reactor For CF] Been out of touch, but trying to get back
>
>
> Mike,
>
> Java -version returns...
>
> java version "1.4.2_05"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_05-b04) Java
> HotSpot(TM) Client VM (build 1.4.2_05-b04, mixed mode)
>
> The jvm.config is attached.  I assume you didn't want to see the
> wsconfig_jvm.config.
>
> What are you looking for?
>
> Doug
>
>
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf
> Of Mike Brunt - Webapper Services, LLC
> Sent: Wednesday, May 24, 2006 10:32 AM
> To: [email protected]
> Subject: RE: [Reactor For CF] Been out of touch, but trying to get back
>
> Can you let me know what JVM you are using - precise version and what memory
> settings are in your jvm.config file, in fact can you send me your
> jvm.config file so I can see what your current settings are.
>
> Kind Regards - Mike Brunt
> CTO and Lead Architect - Webapper Services LLC Tel 562.243.6255 Web Site:
> http://www.webapper.com
> Blog: http://www.webapper.net
> Tools: http://www.seefusion.com
>
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf
> Of Doug Hughes
> Sent: Wednesday, May 24, 2006 7:26 AM
> To: [email protected]
> Subject: RE: [Reactor For CF] Been out of touch, but trying to get back
>
> I'm a little perplexed as to why it would spike like that.  Aside from
> instance variables (and there are plenty of those) there's not much else
> that would be put in ram.
>
> I wonder if the situation is really that the objects are not getting garbage
> collected due to the app being under too much load all the time?  I suppose
> that's quite possible.  RAM usage DOES dip from time to time, but never as
> far down as you might expect.
>
> Maybe I should try forcing the garbage collector to run more often?  As a
> note, I did forcibly run the garbage collector like this:
>
> <cfset CreateObject("Java", "java.lang.System").gc() />
>
> This didn't make any difference.  However, the docs say this:
>
> "Calling the gc method suggests that the Java Virtual Machine expend effort
> toward recycling unused objects in order to make the memory they currently
> occupy available for quick reuse. When control returns from the method call,
> the Java Virtual Machine has made a best effort to reclaim space from all
> discarded objects."
>
> This implies that there's no guarantee that much of any garbage will be
> collected.
>
> I'm not fond of the idea of adding tomcat to the server where the problem is
> obviously occurring.  I might try it locally.  So far I'm not seeing
> anything that suggest that this is JRun specific... especially seeing as
> none of the other instances are having this issue.
>
> I'll dig though the logs more, but I didn't see many long running requests
> last time I looked.  It warrants more research though.
>
> Anyone have any ideas how to programmatically dump (or directly access) the
> heap at runtime?
>
> Doug
>
>
>
>
>
>
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf
> Of João Fernandes
> Sent: Wednesday, May 24, 2006 9:49 AM
> To: [email protected]
> Subject: RE: [Reactor For CF] Been out of touch, but trying to get back
>
> Doug,
>
> I'm using reactor (the version just before you introduced the iterator I
> think) in production mode since 10/03/2006 where the small job is a simple
> injection from xml files into our database, up to 1000 files each time and
> up to 100 queries per file.
> I created a independent instance so only reactor and the appropriate EG are
> running there.
>
> Since we need that information as soon as possible to be able to import it
> to one of our magazines, I raised simultaneous EG requests to 150 (you can
> shoot me). During that time, It goes up to 600 MB Ram (maybe more) and the
> day after the memory is back to 200 (more or less).
>
> The instance has only been restarted yesterday because of a forced reboot
> (damn windows updates).
>
>
> João Fernandes
> Dep. Informática - Área de Desenvolvimento Cofina media
>
> Avenida João Crisóstomo, Nº 72 . 1069-043 Lisboa PORTUGAL Tel (+351) 213 185
> 200 . Fax (+351) 213 540 370 [EMAIL PROTECTED]
>
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] ] On Behalf
> Of Doug Hughes
> Sent: quarta-feira, 24 de Maio de 2006 13:46
> To: [email protected]
> Subject: RE: [Reactor For CF] Been out of touch, but trying to get back
>
> To quote Einstein, finding this leak - which is very obvious on my blog (but
> not necessarily in reactor) - is like shooting a bird in the dark in a
> country without many birds.
>
> I'm not sure that even looping over a test case is fair.  This will change
> how the JVM handles the request.
>
> All in all, here's what I'm seeing: I load my application for the first time
> and it starts up.  (This seems to take a really long time for some reason.)
> I then let it sit and accept requests.  Over 6 hours or so (this is a guess
> - it used to be shorter) the task manager shows the JRun process using more
> and more memory.  It starts out around 100 megs or so and creeps up to
> roughly 600.  After it hits 600, if I don't do anything about it, eventually
> I see this error:
>
> Server Error
>
> The server encountered an internal error and was unable to complete your
> request.
>
> JRun closed connection.
>
> If I look in the logs I see a java.lang.outofmemory error.
>
> I've tried a LOT of stuff.  In particular...
>
> I've tried running the app under load locally using jmeter hitting the same
> page over and over.  This doesn't seem to produce the errors.
>
> I've also tried writing a .net app that parsed yesterdays logs and ran them
> locally to see if I could reproduce the error and start honing down from
> there.  Unfortunately, IIS 5.1 craps out.  I'll need to, perhaps, manage
> sessions the same as all the clients did.  I'm not sure if this will get me
> anywhere.
>
> I'm stumped.  I'm glad that most people haven't seen this...  I just don't
> know if it IS reactor or not.
>
> Doug
>
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto: [EMAIL PROTECTED]] On Behalf
> Of Tom Chiverton
> Sent: Wednesday, May 24, 2006 4:13 AM
> To: [email protected]
> Subject: Re: [Reactor For CF] Been out of touch, but trying to get back
>
> On Tuesday 23 May 2006 16:49, Doug Hughes wrote:
> > I'll be using the site to create an official roadmap so you're not
> > blindsided by where the project's headed.  This will tell us how we're
>
> Ohh, rar.
>
> > About bug fixes.. Right now I'm working on two big ones.  The biggest
> > is a
> .
> > Does anyone have a feel as for where this might be?  Have you NOT
> > experienced this problem?
>
> We're not in production or heavy dev. usage with Reactor yet, but I can't
> say I've seen it.
>
> Don't suppose you have a simple test case/app for us all to try ? Otherwise
> the idea of writing test cases and a 'loop over each test case a thousend
> times' thing doesn't appeal :-)
>
> --
> Tom Chiverton
>
> ****************************************************
>
> This email is sent for and on behalf of Halliwells LLP.
>
> Halliwells LLP is a limited liability partnership registered in England and
> Wales under registered number OC307980 whose registered office address is at
> St James's Court Brown Street Manchester M2 2JF.  A list of members is
> available for inspection at the registered office. Any reference to a
> partner in relation to Halliwells LLP means a member of Halliwells LLP.
> Regulated by the Law Society.
>
> CONFIDENTIALITY
>
> This email is intended only for the use of the addressee named above and may
> be confidential or legally privileged.  If you are not the addressee you
> must not read it and must not use any information contained in nor copy it
> nor inform any person other than Halliwells LLP or the addressee of its
> existence or contents.  If you have received this email in error please
> delete it and notify Halliwells LLP IT Department on 0870 365 8008.
>
> For more information about Halliwells LLP visit www.halliwells.com.
>
> We are pleased to announce that Halliwells LLP has been voted AIM Lawyer of
> the Year at the 2005 Growth Company Awards
>
>
>
>
> -- Reactor for ColdFusion Mailing List -- [email protected]
> -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/
>
>
>
>
>
> -- Reactor for ColdFusion Mailing List -- [email protected]
> -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/
>
>
>
>
>
> -- Reactor for ColdFusion Mailing List -- [email protected]
> -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/
>
>
>
>
>
> -- Reactor for ColdFusion Mailing List -- [email protected]
> -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/
>
>
>
>
>
>
>
> -- Reactor for ColdFusion Mailing List -- [email protected]
> -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/
>
>
>
>
>
> -- Reactor for ColdFusion Mailing List -- [email protected]
> -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/
>
>
>
>
>
> -- Reactor for ColdFusion Mailing List -- [email protected]
> -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/
>
>
>
>
>
> -- Reactor for ColdFusion Mailing List -- [email protected]
> -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/
>
>
>
>
>
>
> -- Reactor for ColdFusion Mailing List -- [email protected]
> -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/
>
>
>
>
>
> -- Reactor for ColdFusion Mailing List -- [email protected]
> -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/
>
>
>
>
>
>
>
> -- Reactor for ColdFusion Mailing List -- [email protected]
> -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/
>
>
>
>
>
> -- Reactor for ColdFusion Mailing List -- [email protected]
> -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/
>
>
>
-- Reactor for ColdFusion Mailing List -- [email protected] -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/

Reply via email to