4585 tomcat sessions? thats quite large if may say that.. and even more 10 wicket sessions that tomcat sessions Do you have multiply apps deployed on that server?
if a search engine doesnt send a cookie back then the urls should be encoded with jsessionid and we get the session from that.. johan On Thu, Apr 10, 2008 at 12:22 AM, Jeremy Thomerson < [EMAIL PROTECTED]> wrote: > I finally am able to get a good analysis of it. It dumped two memory > dumps > when it died in the past couple days (it's still dying about once or twice > a > day). Using this GREAT tool: > > https://www.sdn.sap.com/irj/sdn/wiki?path=/display/Java/Java+Memory+AnalysisI > am able to see deep memory views that tell me what is taking up the > memory. > > Each time it dies, it is usually during heavy web crawler traffic from > Google / Ask.com / etc. The memory tool shows me having: > 4585 instances of org.apache.catalina.session.StandardSession, totaling > 984,160,016 bytes of memory (I give Tomcat 1GB). > 4595 instances of com.texashuntfish.web.wicket.WebSession, totaling > 530,524,680 bytes of memory. > > I have had the session expiration turned down to 90 minutes in Tomcat for > a > very long time. So, this means that 4,585 sessions are created in 90 > minutes, right? It seems like I shouldn't have this many sessions, right? > Are the search engines crawlers creating a new session for every page they > hit? I'm going to put some logging output in my subclass of WebSession > that > tells me when it's being created, and by what IP so that I can take a look > at this. > > Does anyone have any ideas? I don't know what my session counts were in > 1.2.6, but I never ran into this problem or ran out of memory, excecpt for > about a year and a half ago when I had session lengths turned up to a > couple > days long. > > Thank you, > Jeremy Thomerson > > > On Fri, Apr 4, 2008 at 12:03 AM, Jeremy Thomerson < > [EMAIL PROTECTED]> wrote: > > > Nope - one page never holds on to another. I never even pass pages into > > another page or link or something as a reference. > > > > Interestingly, I DECREASED the memory the JVM could have from 1.5 GB to > > 1.0 GB today, and it has been stable all day (after also releasing a > version > > using Wicket 1.3.3). That's not a definite sign - it was stable for > several > > days after upgrading to 1.3.2 from 1.2.6 before freaking out. But I'll > > watch it closely. The memory creeped slowly up to the max, and has > stayed > > there, but without the site crashing, and without any degradation of > > performance. Does that give anyone any ideas? I'm so exhausted, I > think > > that I'm starting to lose my ability to think freshly about it. > > > > Thank you, > > Jeremy > > > > > > On Thu, Apr 3, 2008 at 5:44 PM, Matej Knopp <[EMAIL PROTECTED]> > wrote: > > > > > This is really weird. Do you have any inter-page references in your > > > application? > > > > > > -Matej > > > > > > On Thu, Apr 3, 2008 at 9:35 PM, Jeremy Thomerson > > > <[EMAIL PROTECTED]> wrote: > > > > The oddness is what baffles me: Tomcat has no output anywhere. I > have > > > > grepped and tailed the entire Tomcat logs directory, stdout*, > > > stderr*, > > > > localhost*, etc. Nothing in eventvwr. > > > > > > > > It must be memory related, though. There is a steadily increasing > > > memory > > > > footprint - it was increasing so fast yesterday because we were > > > getting > > > > pounded by tons of traffic and Google's crawler and Ask's crawler > all > > > > simultaneously. Of course, the traffic was still no higher than it > > > has been > > > > in the past - this is definitely a new problem. > > > > > > > > I redeployed today with the pending 1.3.3 release built by Frank to > > > see if > > > > my leak could be the same as Martijn's below, but the memory > > > continues to > > > > increase. It will die soon. I have added the parameter to tell it > > > to dump > > > > on OOM - hopefully I got the right parameter and it will work. > > > > > > > > Anyone here know how to (or if you can) use jstat / jmap with > > > tomcat5.exe, > > > > running as Windows service? All my development is on Linux > machines, > > > and I > > > > can easily use those tools, but on the Windows prod environment > > > (ughh), jps > > > > doesn't give me a VMID for Tomcat. > > > > > > > > Thank you for your help! > > > > Jeremy > > > > > > > > > > > > > > > > On Thu, Apr 3, 2008 at 2:27 PM, Al Maw <[EMAIL PROTECTED]> wrote: > > > > > > > > > You can use as many anonymous inner classes as you like. I have > > > them > > > > > coming > > > > > out of my ears, personally. > > > > > > > > > > It's very odd for tomcat to die with no output. There will be > > > output > > > > > somewhere. Check logs/catalina.out and also logs/localhost*. If > the > > > JVM > > > > > dies, it will hotspot or even segfault and log that, at least. If > > > you have > > > > > gradually increasing memory footprint then this should be pretty > > > easy to > > > > > track down with a profiler. > > > > > > > > > > Make sure you run Tomcat with a sensible amount of permanent > > > generation > > > > > space (128M+). > > > > > > > > > > Regards, > > > > > > > > > > Alastair > > > > > > > > > > > > > > > > > > > > On Thu, Apr 3, 2008 at 6:43 AM, Martijn Dashorst < > > > > > [EMAIL PROTECTED]> > > > > > wrote: > > > > > > > > > > > There are commandline options for the jvm to dump on OOM. > > > > > > > > > > > > Anyway, doesn't the log file give any insight into what is > > > happening > > > > > > in your application? Did you (or your sysadmin) disable logging > > > for > > > > > > Wicket? > > > > > > > > > > > > You can also run external tools to see what is happening inside > > > your > > > > > > JVM without blocking the app. e.g. use jmap -histo to see how > > > many > > > > > > objects are alive at a particular moment. The top 10 is always > > > > > > interesting. In my case I found a memory leak in the > > > diskpagestore > > > > > > when exceptions occurred during writing to disk. This is solved > > > in > > > > > > 1.3.3 (which is just days away from an official release, try > it!) > > > > > > > > > > > > jstat -gc -h50 <pid> 1000 will log the garbage collector > > > statistics > > > > > > every second. > > > > > > > > > > > > Martijn > > > > > > > > > > > > On 4/3/08, Jeremy Thomerson <[EMAIL PROTECTED]> wrote: > > > > > > > I upgraded my biggest production app from 1.2.6 to 1.3 last > > > week. I > > > > > > have > > > > > > > had several apps running on 1.3 since it was in beta with no > > > problems > > > > > - > > > > > > > running for months without restarting. > > > > > > > > > > > > > > This app receives more traffic than any of the rest. We > have > > > a > > > > > decent > > > > > > > server, and I had always allowed Tomcat 1.5GB of RAM to > > > operate with. > > > > > > It > > > > > > > never had a problem doing so, and I didn't have OutOfMemory > > > errors. > > > > > > Now, > > > > > > > after the upgrade to 1.3.2, I am having all sorts of > trouble. > > > It ran > > > > > > for > > > > > > > several days without a problem, but then started dying a > > > couple times > > > > > a > > > > > > > day. Today it has died four times. Here are a couple odd > > > things > > > > > about > > > > > > > this: > > > > > > > > > > > > > > - On 1.2.6, I never had a problem with stability - the app > > > would > > > > > run > > > > > > > weeks between restarts (I restart once per deployment, > > > anywhere > > > > > from > > > > > > once a > > > > > > > week to at the longest about two months between deploy / > > > restart). > > > > > > > - Tomcat DIES instead of hanging when there is a problem. > > > Always > > > > > > > before, if I had an issue, Tomcat would hang, and there > > > would be > > > > > OOM > > > > > > in the > > > > > > > logs. Now, when it crashes, and I sign in to the server, > > > Tomcat is > > > > > > not > > > > > > > running at all. There is nothing in the Tomcat logs that > > > says > > > > > > anything, or > > > > > > > in eventvwr. > > > > > > > - I do not get OutOfMemory error in any logs, whereas I > have > > > always > > > > > > > seen it in the logs before when I had an issue with other > > > apps. I > > > > > am > > > > > > > running Tomcat as a service on Windows, but it writes > stdout > > > / > > > > > stderr > > > > > > to > > > > > > > logs, and I write my logging out to logs, and none of > these > > > logs > > > > > > include ANY > > > > > > > errors - they all just suddenly stop at the time of the > > > crash. > > > > > > > > > > > > > > My money is that it is an OOM error caused by somewhere that > I > > > am > > > > > doing > > > > > > > something I shouldn't be with Wicket. There's no logs that > > > even say > > > > > it > > > > > > is > > > > > > > an OOM, but the memory continues to increase linearly over > > > time as > > > > > the > > > > > > app > > > > > > > runs now (it didn't do that before). My first guess is my > > > previous > > > > > > > proliferate use of anonymous inner classes. I have seen in > > > the email > > > > > > > threads that this shouldn't be done in 1.3. > > > > > > > > > > > > > > Of course, the real answer is that I'm going to be digging > > > through > > > > > > profilers > > > > > > > and lines of code until I get this fixed. > > > > > > > > > > > > > > My question, though, is from the Wicket devs / experienced > > > users - > > > > > > where > > > > > > > should I look first? Is there something that changed > between > > > 1.2.6 > > > > > and > > > > > > 1.3 > > > > > > > that might have caused me problems where 1.2.6 was more > > > forgiving? > > > > > > > > > > > > > > I'm running the app with JProbe right now so that I can get > a > > > > > snapshot > > > > > > of > > > > > > > memory when it gets really high. > > > > > > > > > > > > > > Thank you, > > > > > > > > > > > > > > Jeremy Thomerson > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Buy Wicket in Action: http://manning.com/dashorst > > > > > > Apache Wicket 1.3.2 is released > > > > > > Get it now: http://www.apache.org/dyn/closer.cgi/wicket/1.3.2 > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > > > > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Resizable and reorderable grid components. > > > http://www.inmethod.com > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > >
