RE: tomcat memory leak problem
As stated below profiling (Yourkit is a very good recommendation, you can try their early access version http://www.yourkit.com/eap/index.jsp) will give you more insight in your tomcat application. Sometimes changing your memory / garbage collect parameters can change the world for you. For more insight read: http://www.informit.com/guides/content.aspx?g=javaseqNum=253 http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html Hubert -Original Message- From: Piller Sébastien [mailto:pi...@hmcrecord.ch] Sent: 05 February 2009 17:25 To: Tomcat Users List Subject: Re: tomcat memory leak problem Are you sure you don't have any thread or task (I mean, quartz or so) that is running and consume memory? I agree, this sounds quite strange (our app is running for several weeks and don't have any major issue, we sometimes need to restart tomcat because it hangs, but our memory is under control, and we use lotz of external api like spring, hibernate, wicket, etc.) Could you maybe profile it using Yourkit or any other tool? fcxjp a écrit : My system environment is: Windows 2000 Server. JDK 1.5, tomcat 5.5, Oracle 9 The problems are: 1. After tomcat was started, the memory of the tomcat was normal, about 200M-300M. But after a certain time(this time was not set), the memory began to grow, and the growing speed was so fast that in about 5 minites the maximum would be reached. The difference of this problem with other problems I've checked is, in other problems, the memory growing speed is continual and not so fast, while in our problem the memory didn't grow at the beginning, but after a certain time instead. 2. There is another strange phenomenon here. We restarted tomcat at 23:30 last night. We checked the log at 8:00 this morning and found out the system was not used by anyone during this time. But the memory of tomcat had reached to 1.5G, which is the maximun of tomcat's memory. Later, after some users began to use this system, tomcat' memory dropped to 500M+. How and why was the memory collected? By the way, we have cheched the code for any optimization, including StringBuffer, Vector, datasource connections, etc., which only resulted with a faster response speed for users, not any influence for the memory problem. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: mod_jk
Well... as far as I know there is an option to check if a backend is available (up to a certain level). Look in the documentation for ping_mode (mod_jk 1.2.27) http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html Hubert -Original Message- From: gerhardus.geldenh...@gta-travel.com [mailto:gerhardus.geldenh...@gta-travel.com] Sent: 06 February 2009 14:19 To: users@tomcat.apache.org Subject: RE: mod_jk 1) As far as I know, no, mod_jk does not read workers.properties dynamically. 2) Yes and no, it will not send a request unless communication has been established with the worker, it may happen that the worker fails, or someone shut it down. Depending on how you configure the workers and the number of workers, it can retry the request and/or try a different worker. Mod_jk will mark the worker on error when it does not respond, and it will try again after a configurable time -but it tries again with an actual request-. It would be really nice if you could test availability of a node with a configurable request instead of a live production request... (hint, hint) Regards __ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email __ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
mod_jk 1.2.27 strange default reply_timeout
Hi, We have an environment running Apache 2.2.9, mod_jk 1.2.27 and Tomcat 5.5.27. The following settings are in place: Apache (prefork mode) = Timeout 60 KeepAliveTimeout 5 JkWatchdogInterval 90 JkOptions +DisableReuse worker.properties = worker.host_1.type=ajp13 worker.host_1.host=127.0.0.1 worker.host_1.port=8009 worker.host_1.socket_keepalive=1 worker.host_1.socket_timeout=15 worker.host_1.connect_timeout=1 worker.host_1.prepost_timeout=8000 worker.host_1.lbfactor=1 worker.host_1.ping_mode=A worker.host_1.connection_pool_timeout=600 server.xml == Connector port=8009 maxThreads=750 minSpareThreads=25 maxSpareThreads=75 connectionTimeout=60 enableLookups=false redirectPort=8443 protocol=AJP/1.3 / What we notice is that Apache will respond with an error 502 Bad Gateway when a response takes more than 30sec. This seems quite strange as without a reply_timeout set I would expect the reply_timeout to default to 0 (timeout disabled). The same setup with mod_jk 1.2.26 does not give this 502 Bad Gateway error, can't find any information on a behavioural change regarding reply timeouts. The error from mod_jk logging: [Mon Feb 02 13:45:18 2009] [8645:3086362304] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (): (host_1) can't receive the response message from tomcat, network problems or tomcat (127.0.0.1:8009) is down (errno=11) [Mon Feb 02 13:45:18 2009] [8645:3086362304] [error] ajp_get_reply::jk_ajp_common.c (1920): (host_1) Tomcat is down or refused connection. No response has been sent to the client (yet) [Mon Feb 02 13:45:18 2009] [8645:3086362304] [info] ajp_service::jk_ajp_common.c (2407): (host_1) sending request to tomcat failed (recoverable), (attempt=2) [Mon Feb 02 13:45:18 2009] [8645:3086362304] [error] ajp_service::jk_ajp_common.c (2426): (host_1) connecting to tomcat failed. [Mon Feb 02 13:45:18 2009] host_1 127.0.0.1 POST /hv/main/nav/processflightqry HTTP/1.1 34.101650 Regards, Hubert de Heer
RE: mod_jk 1.2.27 strange default reply_timeout
Hi, Thanks for the quick answer. So the main difference between socket_timeout and socket_connect_timeout is that the later only affects the connect phase. I rolled back to mod_jk 1.2.26 but will try 1.2.27 with socket_connect_timeout tomorrow. Any advice on whether I should forget about the socket_timeout or not? What would be recommendable for a production environment? PS we're running RHEL ES 4 lastest update. Regards, Hubert -Original Message- From: Mladen Turk [mailto:mt...@apache.org] Sent: Mon 02/02/2009 6:16 PM To: Tomcat Users List Subject: Re: mod_jk 1.2.27 strange default reply_timeout Hubert de Heer wrote: Hi, worker.host_1.socket_timeout=15 This is the root of the problem cause it imposes the timeout on any socket operation between mod_jk and tomcat. What we notice is that Apache will respond with an error 502 Bad Gateway when a response takes more than 30sec. socket_timeout x 2 (one op and one retry) This seems quite strange as without a reply_timeout set I would expect the reply_timeout to default to 0 (timeout disabled). This is to have infinite socket_timeout and still have reply timeout. Note that socket_timeout is fixed with 1.2.27, so it actually works now ;) With 1.2.27 you have a new socket_connect_timeout property that you can set to a lower value (see the changelog and docs) Regards -- ^(TM)
RE: SECURITY breach in Tomcat
Hi,
If you really, really need the manager webapp, you can restrict access
to that one not only by password but also by source-ip, e.g. access is
only allowed from your office IP.
In server.xml:
Context path=/manager
docBase=${catalina.home}/server/webapps/manager debug=0
privileged=true
Valve className=org.apache.catalina.valves.RemoteAddrValve
allow=office_ip/
/Context
Hubert
-Original Message-
From: Toby Kurien [mailto:tobyis7...@gmail.com]
Sent: 22 January 2009 16:17
To: users@tomcat.apache.org
Subject: SECURITY breach in Tomcat
Hi,
I have a webapp for my company that has been running for several
years. Recently, we got infected by a trojan or virus and this has
been causing a lot of abnormal behavior. The trojan creates user
accounts in Windows and also creates web applications like safee.war
and zhu.war into the webapps folder of Tomcat and also shuts down
Tomcat. The trojan webapps have jsp and exe files which try to modify,
copy and delete files in the system and also try to access the
database. Symantec and Norton have not been able to rectify or detect
much.
I am totally at loss on what's going on and how to tighten or rectify
this. Anyone with any ideas is highly appreciated.
Thanks,
-Toby
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat running out of threads.
Hello, We run a setup with 3 Apache servers with mod_jk (2.2.6 / 1.2.26) and 10 tomcat instances running on 5 servers (tomcat 5.5.26 / RHEL4). We used the combination of connectionTimeout connection_pool_timeout, both set to 5min (connectionTimeout=30, connection_pool_timeout=300). But we still saw a lot of KeepAlive connections on the tomcat side staying open (average thread count 400 per tomcat instance). worker.properties: worker.server1.type=ajp13 worker.server1.host=xxx.xxx.xxx.xxx worker.server1.port=8009 worker.server1.socket_keepalive=1 worker.server1.socket_timeout=15 worker.server1.connect_timeout=1 worker.server1.prepost_timeout=8000 worker.server1.lbfactor=1 worker.server1.connection_pool_timeout=300 server.xml: Connector port=8009 maxThreads=750 minSpareThreads=25 maxSpareThreads=75 connectionTimeout=30 This week we started to use JkOptions +DisableReuse and our current thread busy count dropped to an average of 8. So we will stick with the DisableReuse and find another use for 2 out of 5 tomcat servers. Regards, Hubert de Heer -Original Message- From: Bernardo Cabezas [mailto:ber...@bergantells.net] Sent: 12 January 2009 13:01 To: users@tomcat.apache.org Subject: RE: Tomcat running out of threads. Hello, According to tomcat documentation, JkOptions +DisableReuse has some performance penalty, because opens new connection for each request. In my case, issue was solved by setting the param: connectionTimeout on the ajp13 connector / entry on tomcat's server.xml Also look at docs for firstReadTimeout parameter. Now mod_jk is using a pool of connectors, but tomcat expires old threads on the pool so issue was solved. Regards, Bernardo. Mark Thomas-18 wrote: From: Hari Prasad [mailto:ursha...@yahoo.co.in] hi.. i too have the same issue.. did anyone find solution? 1. JkOptions +DisableReuse 2. Use correct timeouts and CPing/CPong I'd go with option 1. YMMV. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- View this message in context: http://www.nabble.com/Tomcat-running-out-of-threads.-tp16812001p21413171 .html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Memory pool Survivor space
Andre, Have a look at http://www.informit.com/guides/content.aspx?g=javaseqNum=253 for more details about JVM. A bit more difficult to read but with tons of info about JVM behaviour and default settings: http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp So for the SurvivorRatio the default setting is: -XX:SurvivorRatio=8 Ratio of eden/survivor space size [Solaris amd64: 6; Sparc in 1.3.1: 25; other Solaris platforms in 5.0 and earlier: 32] Hubert -Original Message- From: André Warnier [mailto:[EMAIL PROTECTED] Sent: 30 November 2008 15:54 To: Tomcat Users List Subject: Memory pool Survivor space Hi. I am monitoring a Tomcat during startup, using jconsole. This Tomcat's JVM is started with the switches -Xms200M -Xmx200M, and it starts a rather heavy webapp that just about occupies Tomcat 100% during 5 minutes whenever I start it. (*) In the memory tab of jconsole, I observe that one of the display sections (Survivor space, the middle bar in the Heap section), has the following behaviour : - before the application is started, it remains constant at about 1 Mb - as soon as I start the application, it grows to about 2 MB, then drops down to 0, then grows up again to 2Mb, then drops to about 1 Mb, then back up again, etc.. The 2 Mb seems to be some kind of hard limit, because it nevers goes over it. The bottom is more variable, sometimes between 1 Mb and 0, but more often 0. - when the application has finished loading (and Tomcat becomes responsive again), the amount of memory used by the Survivor space seems to stabilise again at 1.3 Mb, which is 0.3 Mb more than it was before the application started. Then it slowly over time drops down to 1 Mb. If the attachment survives this post, then you can see this graphically in it. If not, you can get a snapshot here : http://dev.dev.wissensbank.com/public/jconsole_survivor.png What I would like to know is : - what does this mean ? - where does the JVM get this apparent hard limit of 2 Mb for this pool size ? - does it matter ? I mean, assuming I could change it, would that have an impact in how the JVM works while starting this application, and how ? Below are three cut and paste of the summary display at the bottom left of jconsole when displaying this pool. 1) at some point during app startup Time: 2008-11-30 15:27:24 Used: 0 kbytes Committed: 2.240 kbytes Max: 2.240 kbytes 2) also during app startup Time: 2008-11-30 15:28:33 Used: 2.240 kbytes Committed: 2.240 kbytes Max: 2.240 kbytes 3) after app is started Time: 2008-11-30 15:34:32 Used: 1.306 kbytes Committed: 2.240 kbytes Max: 2.240 kbytes (*) to be completely precise : - if I use load-on-startup in the web.xml of this application (no matter with which value), then it seems to get started when Tomcat is started. Whenever that happens, Tomcat becomes unresponsive during approximately 5 minutes, using 95% of the machine's cpu time and not answering HTTP requests. At the end of these 5 minutes, Tomcat writes server startup in ms in its catalina.out file, and it becomes responsive again to HTTP requests. - if I do not have a load-on-startup tag in web.xml, then Tomcat prints server startup in ms in catalina.out, and becomes responsive in about 15 seconds (instead of 5 minutes). But then, it is when I first request the application in the browser that Tomcat becomes irresponsive during 5 minutes and uses 95% of cpu time during these 5 minutes. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
