Please forgive my lack of "Thank You"s. Your expertise is exactly what I need. Thank you!!
Doug -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Mike Brunt - Webapper Services, LLC Sent: Wednesday, May 24, 2006 11:14 AM To: [email protected] Subject: RE: [Reactor For CF] Been out of touch, but trying to get back Ok Doug, we specialize in tracking down memory leaks along with other things and as I have been totally useless one way or another to your Reactor project being only a spectator; I think it is time I tried to be contributory, if I can. So I asked for you jvm.config to see what your settings are and if you would accept my help I need you first to do the following: Currently your Java Arguments are as follows: java.args=-server -Xmx512m -Dsun.io.useCanonCaches=false -XX:MaxPermSize=128m -XX:+UseParallelGC -DJINTEGRA_NATIVE_MODE -DJINTEGRA_PREFETCH_ENUMS -Dcoldfusion.rootDir={application.home}/ My first suggestion is to change slightly to these: java.args=-server Xms-512m -Xmx512m -Dsun.io.useCanonCaches=false -XX:PermSize=128m -XX:MaxPermSize=128m -XX:+UseParallelGC -DJINTEGRA_NATIVE_MODE -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -verbose:gc -Xloggc:reactorgc.log -DJINTEGRA_PREFETCH_ENUMS -Dcoldfusion.rootDir={application.home}/ Adding Xms-512m and -XX:PermSize=128m will start the heap up with more breathing space. Incidentally, how much RAM do you have on this box? Adding -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -verbose:gc -Xloggc:reactorgc.log will print Garbage Collection details to the -out.log and to a separate reactorgc.log which I will look at to see what is going on with memory in the heap. I also recommend turning on metrics logging in Jrun.xml: Uncomment: <service class="coldfusion.server.jrun4.metrics.MetricsServiceAdapter" name="MetricsService"> <attribute name="bindToJNDI">true</attribute> </service> Add or change these settings: <attribute name="metricsEnabled">true</attribute> <attribute name="metricsLogFrequency">60</attribute> <attribute name="metricsFormat">Web threads (busy/total/listen/idle/delay/handled): {busyTh}/{totalTh}/{listenTh}/{idleTh}/{delayRq}/{handledRq} Sessions: {sessions} Total Memory={totalMemory} Free={freeMemory}</attribute> These logging changes are temporary till we track down the issue from looking at those logs. These changes will need a CF-JRun restart by the way. Kind Regards - Mike Brunt CTO and Lead Architect - Webapper Services LLC Tel 562.243.6255 Web Site: http://www.webapper.com Blog: http://www.webapper.net Tools: http://www.seefusion.com -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Doug Hughes Sent: Wednesday, May 24, 2006 7:47 AM To: [email protected] Subject: RE: [Reactor For CF] Been out of touch, but trying to get back Mike, Java -version returns... java version "1.4.2_05" Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_05-b04) Java HotSpot(TM) Client VM (build 1.4.2_05-b04, mixed mode) The jvm.config is attached. I assume you didn't want to see the wsconfig_jvm.config. What are you looking for? Doug -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Mike Brunt - Webapper Services, LLC Sent: Wednesday, May 24, 2006 10:32 AM To: [email protected] Subject: RE: [Reactor For CF] Been out of touch, but trying to get back Can you let me know what JVM you are using - precise version and what memory settings are in your jvm.config file, in fact can you send me your jvm.config file so I can see what your current settings are. Kind Regards - Mike Brunt CTO and Lead Architect - Webapper Services LLC Tel 562.243.6255 Web Site: http://www.webapper.com Blog: http://www.webapper.net Tools: http://www.seefusion.com -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Doug Hughes Sent: Wednesday, May 24, 2006 7:26 AM To: [email protected] Subject: RE: [Reactor For CF] Been out of touch, but trying to get back I'm a little perplexed as to why it would spike like that. Aside from instance variables (and there are plenty of those) there's not much else that would be put in ram. I wonder if the situation is really that the objects are not getting garbage collected due to the app being under too much load all the time? I suppose that's quite possible. RAM usage DOES dip from time to time, but never as far down as you might expect. Maybe I should try forcing the garbage collector to run more often? As a note, I did forcibly run the garbage collector like this: <cfset CreateObject("Java", "java.lang.System").gc() /> This didn't make any difference. However, the docs say this: "Calling the gc method suggests that the Java Virtual Machine expend effort toward recycling unused objects in order to make the memory they currently occupy available for quick reuse. When control returns from the method call, the Java Virtual Machine has made a best effort to reclaim space from all discarded objects." This implies that there's no guarantee that much of any garbage will be collected. I'm not fond of the idea of adding tomcat to the server where the problem is obviously occurring. I might try it locally. So far I'm not seeing anything that suggest that this is JRun specific... especially seeing as none of the other instances are having this issue. I'll dig though the logs more, but I didn't see many long running requests last time I looked. It warrants more research though. Anyone have any ideas how to programmatically dump (or directly access) the heap at runtime? Doug -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of João Fernandes Sent: Wednesday, May 24, 2006 9:49 AM To: [email protected] Subject: RE: [Reactor For CF] Been out of touch, but trying to get back Doug, I'm using reactor (the version just before you introduced the iterator I think) in production mode since 10/03/2006 where the small job is a simple injection from xml files into our database, up to 1000 files each time and up to 100 queries per file. I created a independent instance so only reactor and the appropriate EG are running there. Since we need that information as soon as possible to be able to import it to one of our magazines, I raised simultaneous EG requests to 150 (you can shoot me). During that time, It goes up to 600 MB Ram (maybe more) and the day after the memory is back to 200 (more or less). The instance has only been restarted yesterday because of a forced reboot (damn windows updates). João Fernandes Dep. Informática - Área de Desenvolvimento Cofina media Avenida João Crisóstomo, Nº 72 . 1069-043 Lisboa PORTUGAL Tel (+351) 213 185 200 . Fax (+351) 213 540 370 [EMAIL PROTECTED] -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Doug Hughes Sent: quarta-feira, 24 de Maio de 2006 13:46 To: [email protected] Subject: RE: [Reactor For CF] Been out of touch, but trying to get back To quote Einstein, finding this leak - which is very obvious on my blog (but not necessarily in reactor) - is like shooting a bird in the dark in a country without many birds. I'm not sure that even looping over a test case is fair. This will change how the JVM handles the request. All in all, here's what I'm seeing: I load my application for the first time and it starts up. (This seems to take a really long time for some reason.) I then let it sit and accept requests. Over 6 hours or so (this is a guess - it used to be shorter) the task manager shows the JRun process using more and more memory. It starts out around 100 megs or so and creeps up to roughly 600. After it hits 600, if I don't do anything about it, eventually I see this error: Server Error The server encountered an internal error and was unable to complete your request. JRun closed connection. If I look in the logs I see a java.lang.outofmemory error. I've tried a LOT of stuff. In particular... I've tried running the app under load locally using jmeter hitting the same page over and over. This doesn't seem to produce the errors. I've also tried writing a .net app that parsed yesterdays logs and ran them locally to see if I could reproduce the error and start honing down from there. Unfortunately, IIS 5.1 craps out. I'll need to, perhaps, manage sessions the same as all the clients did. I'm not sure if this will get me anywhere. I'm stumped. I'm glad that most people haven't seen this... I just don't know if it IS reactor or not. Doug -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Tom Chiverton Sent: Wednesday, May 24, 2006 4:13 AM To: [email protected] Subject: Re: [Reactor For CF] Been out of touch, but trying to get back On Tuesday 23 May 2006 16:49, Doug Hughes wrote: > I'll be using the site to create an official roadmap so you're not > blindsided by where the project's headed. This will tell us how we're Ohh, rar. > About bug fixes.. Right now I'm working on two big ones. The biggest > is a . > Does anyone have a feel as for where this might be? Have you NOT > experienced this problem? We're not in production or heavy dev. usage with Reactor yet, but I can't say I've seen it. Don't suppose you have a simple test case/app for us all to try ? Otherwise the idea of writing test cases and a 'loop over each test case a thousend times' thing doesn't appeal :-) -- Tom Chiverton **************************************************** This email is sent for and on behalf of Halliwells LLP. Halliwells LLP is a limited liability partnership registered in England and Wales under registered number OC307980 whose registered office address is at St James's Court Brown Street Manchester M2 2JF. A list of members is available for inspection at the registered office. Any reference to a partner in relation to Halliwells LLP means a member of Halliwells LLP. Regulated by the Law Society. CONFIDENTIALITY This email is intended only for the use of the addressee named above and may be confidential or legally privileged. If you are not the addressee you must not read it and must not use any information contained in nor copy it nor inform any person other than Halliwells LLP or the addressee of its existence or contents. If you have received this email in error please delete it and notify Halliwells LLP IT Department on 0870 365 8008. For more information about Halliwells LLP visit www.halliwells.com. We are pleased to announce that Halliwells LLP has been voted AIM Lawyer of the Year at the 2005 Growth Company Awards -- Reactor for ColdFusion Mailing List -- [email protected] -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/ -- Reactor for ColdFusion Mailing List -- [email protected] -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/ -- Reactor for ColdFusion Mailing List -- [email protected] -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/ -- Reactor for ColdFusion Mailing List -- [email protected] -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/ -- Reactor for ColdFusion Mailing List -- [email protected] -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/ -- Reactor for ColdFusion Mailing List -- [email protected] -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/ -- Reactor for ColdFusion Mailing List -- [email protected] -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/ -- Reactor for ColdFusion Mailing List -- [email protected] -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/
