NP Doug and if you feel it useful for me to get to your server remotely I have RDS and Ultra VNC on mi notebook.
Kind Regards - Mike Brunt CTO and Lead Architect - Webapper Services LLC Tel 562.243.6255 Web Site: http://www.webapper.com Blog: http://www.webapper.net Tools: http://www.seefusion.com -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Doug Hughes Sent: Wednesday, May 24, 2006 5:09 PM To: [email protected] Subject: RE: [Reactor For CF] Been out of touch, but trying to get back FYI, I tried to install SeeFusion but didnt get anywhere with it. When I selected the specific instance the properties dir, lib dir, and web xml paths had red Xs by them. I'm not sure where to go here... my wife is instructing me to take kid 1 to bed, so I cant dig in further at the moment. That said, I will be zipping up the logs for you shortly and sending the url to you. Doug -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Mike Brunt - Webapper Services, LLC Sent: Wednesday, May 24, 2006 7:24 PM To: [email protected] Subject: RE: [Reactor For CF] Been out of touch, but trying to get back Doug, the GetMetricData() function is actually reporting Windows Performance data which is different to JRun Metrics. I think it would be useful to get SeeFusion up and running as we can then see what is actually happening in real-time. I would also like to get a look at that specific Garbage Collection log we are creating which will be located in either JRun4\bin or CFMX\runtime\bin depending on the installation of CF you have and the metrics data which at present will be going into the cfusion-out.log (assuming your server instance is named cfusion and that log will be in the JRun4\logs or CFMX\runtime\logs depending on the installation of CF. Looking at this stuff now will help us map the next best steps. Sean's description by the way is spot on, if Garbage Collection is working well memory will be consumed and then be reclaimed. Kind Regards - Mike Brunt CTO and Lead Architect - Webapper Services LLC Tel 562.243.6255 Web Site: http://www.webapper.com Blog: http://www.webapper.net Tools: http://www.seefusion.com -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Doug Hughes Sent: Wednesday, May 24, 2006 12:55 PM To: [email protected] Subject: RE: [Reactor For CF] Been out of touch, but trying to get back Mike, FYI - I thought I turned on metrics but calling <cfdump var="#GetMetricData("perf_monitor")#" /> results in an error saying that the service isn't running: The metrics service is not available. This exception is usually caused by service startup failure. Please check your server configuration. The error occurred in C:\Inetpub\DougHughes\www.doughughes.net\live\www\test2.cfm: line 1 1 : <cfdump var="#GetMetricData("perf_monitor")#" /> Would you be interested in me trying to figure out what's wrong with this and restarting the server? This would restart our memory leak though. FYI, I downloaded the log-so-far and ran it through hpjtune. All of the graphs had an obvious upward angle to them. Attached is a screenshot of one that I think relates to this problem. I'm also trying to get jprobe working with JRun to see if maybe I can visualize the heap a bit. (Not optimistic, but it's worth a shot). Thanks, Doug -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Mike Brunt - Webapper Services, LLC Sent: Wednesday, May 24, 2006 2:17 PM To: [email protected] Subject: RE: [Reactor For CF] Been out of touch, but trying to get back Apologies on that Doug, typo. Kind Regards - Mike Brunt CTO and Lead Architect - Webapper Services LLC Tel 562.243.6255 Web Site: http://www.webapper.com Blog: http://www.webapper.net Tools: http://www.seefusion.com -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Doug Hughes Sent: Wednesday, May 24, 2006 11:05 AM To: [email protected] Subject: RE: [Reactor For CF] Been out of touch, but trying to get back I forgot to mention, I changed Xms-512m to -Xms512m. JRun wouldn't start otherwise. Doug -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Mike Brunt - Webapper Services, LLC Sent: Wednesday, May 24, 2006 12:42 PM To: [email protected] Subject: RE: [Reactor For CF] Been out of touch, but trying to get back When you get the logs either shoot me a copy or if they are too big I'll give you a private ftp server we have. I'll tell you what we find. Kind Regards - Mike Brunt CTO and Lead Architect - Webapper Services LLC Tel 562.243.6255 Web Site: http://www.webapper.com Blog: http://www.webapper.net Tools: http://www.seefusion.com -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Doug Hughes Sent: Wednesday, May 24, 2006 9:30 AM To: [email protected] Subject: RE: [Reactor For CF] Been out of touch, but trying to get back Mike - The server has 3.5 gigs of ram. I made the changes to the jvm.config as requested. Note: I only have one jvm.config. All of the other instances share the same jvm.config too. I've only restarted my instances so hopefully that doesnt impact your research. I've never quite figured out how to give a particular instance its own jvm.config. I made the change to log metrics in the Jrun.xml file for the troubled CF instance. I presume I'll watch these logs? Doug -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Mike Brunt - Webapper Services, LLC Sent: Wednesday, May 24, 2006 11:14 AM To: [email protected] Subject: RE: [Reactor For CF] Been out of touch, but trying to get back Ok Doug, we specialize in tracking down memory leaks along with other things and as I have been totally useless one way or another to your Reactor project being only a spectator; I think it is time I tried to be contributory, if I can. So I asked for you jvm.config to see what your settings are and if you would accept my help I need you first to do the following: Currently your Java Arguments are as follows: java.args=-server -Xmx512m -Dsun.io.useCanonCaches=false -XX:MaxPermSize=128m -XX:+UseParallelGC -DJINTEGRA_NATIVE_MODE -DJINTEGRA_PREFETCH_ENUMS -Dcoldfusion.rootDir={application.home}/ My first suggestion is to change slightly to these: java.args=-server Xms-512m -Xmx512m -Dsun.io.useCanonCaches=false -XX:PermSize=128m -XX:MaxPermSize=128m -XX:+UseParallelGC -DJINTEGRA_NATIVE_MODE -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -verbose:gc -Xloggc:reactorgc.log -DJINTEGRA_PREFETCH_ENUMS -Dcoldfusion.rootDir={application.home}/ Adding Xms-512m and -XX:PermSize=128m will start the heap up with more breathing space. Incidentally, how much RAM do you have on this box? Adding -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -verbose:gc -Xloggc:reactorgc.log will print Garbage Collection details to the -out.log and to a separate reactorgc.log which I will look at to see what is going on with memory in the heap. I also recommend turning on metrics logging in Jrun.xml: Uncomment: <service class="coldfusion.server.jrun4.metrics.MetricsServiceAdapter" name="MetricsService"> <attribute name="bindToJNDI">true</attribute> </service> Add or change these settings: <attribute name="metricsEnabled">true</attribute> <attribute name="metricsLogFrequency">60</attribute> <attribute name="metricsFormat">Web threads (busy/total/listen/idle/delay/handled): {busyTh}/{totalTh}/{listenTh}/{idleTh}/{delayRq}/{handledRq} Sessions: {sessions} Total Memory={totalMemory} Free={freeMemory}</attribute> These logging changes are temporary till we track down the issue from looking at those logs. These changes will need a CF-JRun restart by the way. Kind Regards - Mike Brunt CTO and Lead Architect - Webapper Services LLC Tel 562.243.6255 Web Site: http://www.webapper.com Blog: http://www.webapper.net Tools: http://www.seefusion.com -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Doug Hughes Sent: Wednesday, May 24, 2006 7:47 AM To: [email protected] Subject: RE: [Reactor For CF] Been out of touch, but trying to get back Mike, Java -version returns... java version "1.4.2_05" Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_05-b04) Java HotSpot(TM) Client VM (build 1.4.2_05-b04, mixed mode) The jvm.config is attached. I assume you didn't want to see the wsconfig_jvm.config. What are you looking for? Doug -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Mike Brunt - Webapper Services, LLC Sent: Wednesday, May 24, 2006 10:32 AM To: [email protected] Subject: RE: [Reactor For CF] Been out of touch, but trying to get back Can you let me know what JVM you are using - precise version and what memory settings are in your jvm.config file, in fact can you send me your jvm.config file so I can see what your current settings are. Kind Regards - Mike Brunt CTO and Lead Architect - Webapper Services LLC Tel 562.243.6255 Web Site: http://www.webapper.com Blog: http://www.webapper.net Tools: http://www.seefusion.com -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Doug Hughes Sent: Wednesday, May 24, 2006 7:26 AM To: [email protected] Subject: RE: [Reactor For CF] Been out of touch, but trying to get back I'm a little perplexed as to why it would spike like that. Aside from instance variables (and there are plenty of those) there's not much else that would be put in ram. I wonder if the situation is really that the objects are not getting garbage collected due to the app being under too much load all the time? I suppose that's quite possible. RAM usage DOES dip from time to time, but never as far down as you might expect. Maybe I should try forcing the garbage collector to run more often? As a note, I did forcibly run the garbage collector like this: <cfset CreateObject("Java", "java.lang.System").gc() /> This didn't make any difference. However, the docs say this: "Calling the gc method suggests that the Java Virtual Machine expend effort toward recycling unused objects in order to make the memory they currently occupy available for quick reuse. When control returns from the method call, the Java Virtual Machine has made a best effort to reclaim space from all discarded objects." This implies that there's no guarantee that much of any garbage will be collected. I'm not fond of the idea of adding tomcat to the server where the problem is obviously occurring. I might try it locally. So far I'm not seeing anything that suggest that this is JRun specific... especially seeing as none of the other instances are having this issue. I'll dig though the logs more, but I didn't see many long running requests last time I looked. It warrants more research though. Anyone have any ideas how to programmatically dump (or directly access) the heap at runtime? Doug -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of João Fernandes Sent: Wednesday, May 24, 2006 9:49 AM To: [email protected] Subject: RE: [Reactor For CF] Been out of touch, but trying to get back Doug, I'm using reactor (the version just before you introduced the iterator I think) in production mode since 10/03/2006 where the small job is a simple injection from xml files into our database, up to 1000 files each time and up to 100 queries per file. I created a independent instance so only reactor and the appropriate EG are running there. Since we need that information as soon as possible to be able to import it to one of our magazines, I raised simultaneous EG requests to 150 (you can shoot me). During that time, It goes up to 600 MB Ram (maybe more) and the day after the memory is back to 200 (more or less). The instance has only been restarted yesterday because of a forced reboot (damn windows updates). João Fernandes Dep. Informática - Área de Desenvolvimento Cofina media Avenida João Crisóstomo, Nº 72 . 1069-043 Lisboa PORTUGAL Tel (+351) 213 185 200 . Fax (+351) 213 540 370 [EMAIL PROTECTED] -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Doug Hughes Sent: quarta-feira, 24 de Maio de 2006 13:46 To: [email protected] Subject: RE: [Reactor For CF] Been out of touch, but trying to get back To quote Einstein, finding this leak - which is very obvious on my blog (but not necessarily in reactor) - is like shooting a bird in the dark in a country without many birds. I'm not sure that even looping over a test case is fair. This will change how the JVM handles the request. All in all, here's what I'm seeing: I load my application for the first time and it starts up. (This seems to take a really long time for some reason.) I then let it sit and accept requests. Over 6 hours or so (this is a guess - it used to be shorter) the task manager shows the JRun process using more and more memory. It starts out around 100 megs or so and creeps up to roughly 600. After it hits 600, if I don't do anything about it, eventually I see this error: Server Error The server encountered an internal error and was unable to complete your request. JRun closed connection. If I look in the logs I see a java.lang.outofmemory error. I've tried a LOT of stuff. In particular... I've tried running the app under load locally using jmeter hitting the same page over and over. This doesn't seem to produce the errors. I've also tried writing a .net app that parsed yesterdays logs and ran them locally to see if I could reproduce the error and start honing down from there. Unfortunately, IIS 5.1 craps out. I'll need to, perhaps, manage sessions the same as all the clients did. I'm not sure if this will get me anywhere. I'm stumped. I'm glad that most people haven't seen this... I just don't know if it IS reactor or not. Doug -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Tom Chiverton Sent: Wednesday, May 24, 2006 4:13 AM To: [email protected] Subject: Re: [Reactor For CF] Been out of touch, but trying to get back On Tuesday 23 May 2006 16:49, Doug Hughes wrote: > I'll be using the site to create an official roadmap so you're not > blindsided by where the project's headed. This will tell us how we're Ohh, rar. > About bug fixes.. Right now I'm working on two big ones. The biggest > is a . > Does anyone have a feel as for where this might be? Have you NOT > experienced this problem? We're not in production or heavy dev. usage with Reactor yet, but I can't say I've seen it. Don't suppose you have a simple test case/app for us all to try ? Otherwise the idea of writing test cases and a 'loop over each test case a thousend times' thing doesn't appeal :-) -- Tom Chiverton **************************************************** This email is sent for and on behalf of Halliwells LLP. Halliwells LLP is a limited liability partnership registered in England and Wales under registered number OC307980 whose registered office address is at St James's Court Brown Street Manchester M2 2JF. A list of members is available for inspection at the registered office. Any reference to a partner in relation to Halliwells LLP means a member of Halliwells LLP. Regulated by the Law Society. CONFIDENTIALITY This email is intended only for the use of the addressee named above and may be confidential or legally privileged. If you are not the addressee you must not read it and must not use any information contained in nor copy it nor inform any person other than Halliwells LLP or the addressee of its existence or contents. If you have received this email in error please delete it and notify Halliwells LLP IT Department on 0870 365 8008. For more information about Halliwells LLP visit www.halliwells.com. We are pleased to announce that Halliwells LLP has been voted AIM Lawyer of the Year at the 2005 Growth Company Awards -- Reactor for ColdFusion Mailing List -- [email protected] -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/ -- Reactor for ColdFusion Mailing List -- [email protected] -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/ -- Reactor for ColdFusion Mailing List -- [email protected] -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/ -- Reactor for ColdFusion Mailing List -- [email protected] -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/ -- Reactor for ColdFusion Mailing List -- [email protected] -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/ -- Reactor for ColdFusion Mailing List -- [email protected] -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/ -- Reactor for ColdFusion Mailing List -- [email protected] -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/ -- Reactor for ColdFusion Mailing List -- [email protected] -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/ -- Reactor for ColdFusion Mailing List -- [email protected] -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/ -- Reactor for ColdFusion Mailing List -- [email protected] -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/ -- Reactor for ColdFusion Mailing List -- [email protected] -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/ -- Reactor for ColdFusion Mailing List -- [email protected] -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/ -- Reactor for ColdFusion Mailing List -- [email protected] -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/ -- Reactor for ColdFusion Mailing List -- [email protected] -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/ -- Reactor for ColdFusion Mailing List -- [email protected] -- Archives at http://www.mail-archive.com/reactor%40doughughes.net/
