AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hi all, I just want to give you an update about my problem THANKS TO ALL Felix gave me the final hint... I really had some connection not given back to the pool, so the connections kept open.. - :-( I thought I checked that. Since I fixed that and use removeAbandoned there was no more crash!!! Finally.. :-) I REALY appreciate your help. You guys are doing a really GREAT job! Thanks again.. David Kumar Hi Felix, 1. You are abandoning db connections and thus are draining the pool (most likely in my eyes) 2. The pool is too small (unlikely, since no work seems to be done other than waiting for new connections) 3. Lock contention while going through borrowObject (unlikely, since no thread is doing a validation) So I would configure 'logAbandoned=true' and even 'removeAbandoned=true' and 'removeAbandonedTimeout=60' as stated on http://tomcat.apache.org/tomcat-7.0-doc/jndi-datasource-examples-howto.html. When your pool is getting drained, you will see a message in your log files, where you have taken connections out of the pool, but haven used them for a while. Than you can start to look for places, where those connections might have leaked. thanks for that hint, I will have a look at all of my databasepools.. Hopefully you're right and that is our problem.. thanks avid - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hi Felix, 1. You are abandoning db connections and thus are draining the pool (most likely in my eyes) 2. The pool is too small (unlikely, since no work seems to be done other than waiting for new connections) 3. Lock contention while going through borrowObject (unlikely, since no thread is doing a validation) So I would configure 'logAbandoned=true' and even 'removeAbandoned=true' and 'removeAbandonedTimeout=60' as stated on http://tomcat.apache.org/tomcat-7.0-doc/jndi-datasource-examples-howto.html. When your pool is getting drained, you will see a message in your log files, where you have taken connections out of the pool, but haven used them for a while. Than you can start to look for places, where those connections might have leaked. thanks for that hint, I will have a look at all of my databasepools.. Hopefully you're right and that is our problem.. thanks avid - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hey Rainer, It got stripped by the list. Can you post it somewhere and make the URL available? here you go: - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hey Rainer, It got stripped by the list. Can you post it somewhere and make the URL available? here you go: http://www.telestar.de/tmp10356/threaddump.txt thanks David - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Interesting. If the problem was too many sockets in CLOSE_WAIT, consider looking at the connectionLinger setting on your AJP connector's in Tomcat. Mark Mhh.. I haven't heard about the options before.. But it sound like a great idea.. Default is -1. So what would be a good time, one minute? Thanks Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
sry I was wrong default is -1 what menas it is disabled, so I'm fine..?!? Interesting. If the problem was too many sockets in CLOSE_WAIT, consider looking at the connectionLinger setting on your AJP connector's in Tomcat. Mark Mhh.. I haven't heard about the options before.. But it sound like a great idea.. Default is -1. So what would be a good time, one minute? Thanks Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW:AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
hey, Yes. Whether this is the java that runs tomcat is not immediately evident. But it also seems that they concern port numbers like 7009, 9009 etc. which must be your AJP Connectors, so let's assume the CLOSE-WAITs all involve Tomcat. The first observation hat I would make is that this looks like a pretty busy system. Then, there are 2100+ lines in that list. Let's assume first that they are all internal connections (which probable they are not, but for simplicity), so that there are actually 2 lines for the same connection, like these : tcp0 0 127.0.0.1:39872 127.0.0.1:7009 FIN_WAIT2 20766/apache2 . .. tcp6 1 0 127.0.0.1:7009 127.0.0.1:39872 CLOSE_WAIT 20564/java .. So let's say that we have in total about 1000 connections active in some state. Compared to that, there are 54 connections in CLOSE_WAIT, which is like 5%. That does not really sound pathological to me, considering that CLOSE_WAIT is a normal state through which any TCP connection goes at some point. If you look again at the above connection, and if I am not confusing my clients and servers again, it looks here like : - Apache http (PID 20766), as a client, has established a connection to port 7009 of the server Tomcat (PID 20564). That must be an AJP connection, from the mod_jk module of Apache, to the AJP Connector in Tomcat. - A while later, Apache closes the connection. It sends a FIN packet to Tomcat, and waits for Tomcat to acknowledge this FIN packet. - when it receives the FIN packet from Apache, Tomcat responds with an ACK, and then goes to the CLOSE_WAIT state. - when Apache receives the ACK from Tomcat, it goes to state FIN_WAIT2. (and that is the current state of the above connection) - now Tomcat is supposed to send a FIN to Apache, and receive an ACK in return. - if Apache receives the FIN from Tomcat, it sends back an ACK, and goes into state TIME_WAIT (where it doesn't expect anything anymore), and then after a short moment, it discards that connection. If there are connections that remain in the CLOSE_WAIT state for very long, it means that something in the last 2 steps above is not working. Now in your listing, we see about 1900 lines in the TIME_WAIT state, all of them *to* an AJP port of Tomcat. This seems to suggest that the Apache side is working as it should. But if the Tomcat side stays in CLOSE_WAIT for a long time, then it would suggest that on the Tomcat side, that connection is never properly close()'d. But this is the request/response connection, so I doubt that there would be a bug there, otherwise we'd have problems reports filling this list every day. Might there be that there is somewhere a discrepancy between the keep-alive settings, between Apache and Tomcat ? or maybe your server is just not strong enough for the load you are putting on it ? According to vmtools the server is boring most times.. I almost totally agree, but I'm not sure if it is a AJP fault. Because most time everything is alright, so all of theses steps are done. Just sometimes after a arbitrarily amount of time ajp / tomcat runs crazy. And in my sample there is the sourceport 9009 / java. It looks for me that apache didn't received the ACK from tomcat after CLOSE_WAIT or it just don't care because apache thought this connections is already closed? Let's assume that is the case.. Isn't there a option to tell tomcat to wait max. time for a response and after that time kill the connection? As far as I understand connectionTimeout is just for the initial request, so after Apache did answer in first correctly connectionTimeout is valueless? Attached you can find our configuration files, maybe you can find a / some misconfigurations? I would agree that we have maybe a kind of misconfigured timeout problem... thanks Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ # # Based upon the NCSA server configuration files originally by Rob McCool. # # This is the main Apache server configuration file. It contains the # configuration directives that give the server its instructions. # See http://httpd.apache.org/docs/2.2/ for detailed information about # the directives. # # Do NOT simply read the instructions in here without understanding # what they do. They're here only as hints or reminders. If you are unsure # consult the online docs. You have been warned. # # The configuration directives are grouped into three basic sections: # 1. Directives that control the operation of the Apache server process as a # whole (the 'global environment'). # 2. Directives that define the parameters of the 'main' or 'default' server, # which responds to requests that aren't handled by a virtual host. # These directives also provide default values for the settings # of all virtual hosts
AW: AW:AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hey André, According to vmtools the server is boring most times.. What do you mean, boring ? (like German langweilig ?). -- Jep prety idle.. :-) If you have 2000+ connections there in one state or another, that should mean that the system is usually pretty busy, I would think. On my production machines (vSphere VM's, Linux, Apache+Tomcat, +database), some of them pretty busy, I never see more than 200-300 connections with netstat. That is all states together, with netstat -t -pan | wc -l). Granted, I am not running Yahoo or Facebook, but still, these are public Internet servers, fairly sollicited. I almost totally agree, but I'm not sure if it is a AJP fault. Because most time everything is alright, so all of theses steps are done. Just sometimes after a arbitrarily amount of time ajp / tomcat runs crazy. And in my sample there is the sourceport 9009 / java. It looks for me that apache didn't received the ACK from tomcat after CLOSE_WAIT or it just don't care because apache thought this connections is already closed? Let's assume that is the case.. Isn't there a option to tell tomcat to wait max. time for a response and after that time kill the connection? As far as I understand connectionTimeout is just for the initial request, so after Apache did answer in first correctly connectionTimeout is valueless? Let's look at the various options in the Connector : First, the -1 in some options means unlimited or forever, it doesn't mean disabled. Second, some parameters say that if they are not specified, they default to the value of another parameter, so you should check really what this other parameter is set to. According to Tomcat Configuration Reference 5.5 : connectionLinger The number of milliseconds during which the sockets used by this Connector will linger when they are closed. The default value is -1 (socket linger is disabled). According to Tomcat Configuration Reference 7.0.37 : connectionLinger The number of seconds during which the sockets used by this Connector will linger when they are closed. If not specified, the JVM default will be used. As I'm at 7.0.37 I have to figure out where an how to set linger options at the JVM. I'll have a look. thx - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW:AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen Tel.: + 49 (0) 6592 / 712 -2826 Tel.: + 49 (0) 2676 / 9520 -183 Fax: + 49 (0) 6592 / 712 -2829 http://www.telestar.de/ -Ursprüngliche Nachricht- Von: André Warnier [mailto:a...@ice-sa.com] Gesendet: Freitag, 15. März 2013 10:45 An: Tomcat Users List Betreff: Re: AW:AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open Some notes : - according to the comments in your Apache config file, you are using the worker MPM. But are you sure ? what does /usr/sbin/apache2ctl -l say ? Compiled in modules: core.c mod_log_config.c mod_logio.c worker.c http_core.c mod_so.c - we are missing the contents of your /etc/apache2/workers.properties file Sry didn't rename properties, see attached.. - we are missing the JkMounts or equivalent that you are doing from Apache to Tomcat. Same to our default website related questions : you have 2 tomcats. Are you doing load-balancing ? or are you just sending some URLs to tomcat1 and other to tomcat2 ? We doing loadbalacning Are you proxying everything to the tomcats, or is the Apache httpd front-end serving some URLs on its own ? if so, what proportion ? Everything goes to the tomcats - how many hits per second/minute/hour (any of them, approx.) is your server handling ? according to our logs yesterday between 8pm and 9pm we had 73667 hits together on both tomcats. - you do not have any specific timeout parameters set in your Tomcat AJP Connectors, which is basically a good thing : better to leave the defaults in place, than to start playing with settings that you do not really understand, and make things worse. But you do set one : connectionTimeout=20. Which sounds extremely high to me. Re: http://tomcat.apache.org/tomcat-7.0-doc/config/ajp.html It means that when a client connects to that Connector, a Tomcat thread will be started to handle this connection; then the thread will wait on the connection, to read the request. If the request does not appear, this thread will still wait, up to 200 seconds (more than 3 minutes !) for this request to appear, before it gives up, closes that connection, and goes back to the pool of available threads. A normal client would not do that, but a badly-intentioned client that tries to create a DOS attack on your server, will do that, just to block threads on your server until there are no more threads available. In this case, it is even worse, because other parameters are using this same value as a default, like : keepAliveTimeout. This one means that once one request has been processed by this Tomcat thread on that connection, the thread will not close this connection, but wait to see if any other request appears on that connection from the same client, within the timeout given. In this case, because it defaults to connectionTimeout, the thread will wait more than 3 minutes to see if there is another request. If the client (browser) does not send any additional request on that connection, you have a thread that is blocked doing nothing, for more than 3 minutes. In this case, the client is in reality the mod_jk module under Apache. So the settings of Apache and mod_jk will matter, and I cannot tell exactly at this point what will happen. But if this was a HTTP Connector open to the external WWW world, what would certainly heppen is that you would run out of threads in Tomcat within a couple of minutes, and your server would stop accepting new requests. Or to put this another way : if this was a HTTP connector open to the outside world, and I knew the address of your server, I could bring it down in 3 seconds, using the standard ab program that comes with Apache. that is how our servers are connected together: We have another Webserver (different server) connecting to our Apache - Proxy which is loadbalacning our two tomcats. (Apache and tomcat are together at on server) I will go end reduce the timeout on both worker.properties and server.xml. What will be a value? 30 Seconds? Thanks a lot.. workers.tomcat_home=/opt/tomcat workers.java_home=/usr/lib/jvm/java-6-sun ps=/ worker.list=loadbalancer,status #//siehe https://community.jboss.org/wiki/OptimalModjk12Configuration worker.tomcatX.host=localhost worker.tomcatX.type=ajp13 worker.tomcatX.fail_on_status=404 worker.tomcatX.lbfactor=1 worker.tomcatX.ping_timeout=1000 worker.tomcatX.ping_mode=A worker.tomcatX.socket_timeout=10 worker.tomcatX.connection_pool_timeout=200 worker.tomcat1.reference=worker.tomcatX worker.tomcat1.port=7009 worker.tomcat2.reference=worker.tomcatX worker.tomcat2.port=9009 worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=tomcat1,tomcat2 worker.status.type=status VirtualHost *:8080 ServerAdmin webmaster@localhost SetEnvIf User-Agent (internal dummy connection
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
close_wait for an AJP connection seen from Tomcat means the other side - mod_jk - has closed the connection, but not Tomcat. This is often due to a shorter Timeout on the mod_jk side than on the Tomcat side. It is not a problem per se, but it is if it happens for too many connections for a too long time. I vaguely remember that you have a 10 second socket_timeout in your workers.properties. That's typically bad. Look at the example config in the source mod_jk download. I will do that. It could be, that your requests in Tomcat got stuck and Tomcat still is in the state of working on the requests, therefore keeping the connection open to send back stuff finally, whereas mod_jk has already timed out. To check for that, take a couöple of threa dumps (not: heap dumps) of the running Tomcat process while the close_wait problem is visible. Check what your Tomcat threads are currently doing, e.g. are they mostly sitting idle in the thread pool or executor, or are many of them deep in your application stacks and waiting for database, locks or other stuff. I try to, currently we have a watchdog running, checking every 2 minutes if the tomcats are available at 7009/9009 and http. If one of the tomcats are not responding it will be restarted Thanks David - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hey, Please please before restarting take two thread dumps. On Linux/Unix a kill -QUIT will do that (and write the result to STDOUT, which depending on your setup likely goes to catalina.out). The QUIT signal does *not* terminate the JVM, it just quickly writes out the stacks of all threads. The interruption for the running JVM is minimal, so you can already try how this works before the process gets into the critical state. great I just tried a kill -QUIT in our watchdog -- pretty fast. Next time we got our problem I will send the threaddump to you guys. thanks a lot David - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hey Rainer, attached you can find a Threaddump. Just rename it to .zip. I'm not sure waht all the stuff at the dump means. but I'm sure you know.. :-) thanks Please please before restarting take two thread dumps. On Linux/Unix a kill -QUIT will do that (and write the result to STDOUT, which depending on your setup likely goes to catalina.out). The QUIT signal does *not* terminate the JVM, it just quickly writes out the stacks of all threads. The interruption for the running JVM is minimal, so you can already try how this works before the process gets into the critical state. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AW: AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hey André, André Warnier wrote: The jsvc process here is the wrapper which wraps the JVM and Tomcat to allow them to use orts below 1024 and still run as non-root. For practical purposes, consider it as tomcat. As you can see, there are some AJP connections (local port 8009) in the CLOSE_WAIT state also. It is a normal state of a TCP connection. What is less normal, is that they would remain in that state for a long time. That means that one of the sides of the TCP connection is not doing what it should do. In my experience, under Linux, this can become a problem if you have hundreds of these. At some point, the TCP stack becomes unresponsive and does not allow any new connection. Sometimes, it can happen because the client side in the connection (the one who initially creates the connection to a server's listening socket), discards a Java object which contains an open low-level OS socket object with a connection still open, without explicitly closing this connection first. The discarded (and unreachable) Java object is then still sitting in the heap for a while, and only a GC will actually destroy it and as a side-effect close the underlying connection. That is why I was asking you to force a GC, to see if this made your CLOSE_WAIT connections disappear. Apparently in your case it doesn't, which mean that yours is another case. Now, instead of restarting Tomcat or doing a GC, have you tried to restart Apache ? or maybe just do a /etc/init.d/apache2 reload. That may kill off and restart the Apache children processes, and clean up their mod_jk connections to Tomcat. I tried to just restart the Apache, didn't work. As suggested I tried to use the option -DisableReuse. That did make our situation a lot more horrible instead of days / week we it took just minutes to shut all AJP ports down. Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hey,
we just had that problem again. I did that GC trick, it didn't work out well.
I used:
set hosts [list]
lappend hosts {localhost:7008}
lappend hosts {localhost:9008}
# ...add as many as you want...
foreach {host} $hosts {
set parts [split $host :]
set hostname [lindex $parts 0]
set port [lindex $parts 1]
# for each host...
# Connect to it.
jmx_connect -h $hostname -p $port
# Invoke the garbage collector.
jmx_invoke -n -m java.lang:type=Memory gc
# Close this connection
jmx_close
}
I found that while googleing.
But it doesn't have any effect. Any other ideas?
Thanks
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-Ursprüngliche Nachricht-
Von: André Warnier [mailto:a...@ice-sa.com]
Gesendet: Dienstag, 12. März 2013 09:54
An: Tomcat Users List
Betreff: Re: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
David Kumar wrote:
Hey,
we are still having that issue.
But we could manage to figure out some more stuff.
We made a Tomcat and Java update since that time we had our problem a few
times again, also we did some reconfiguration with connectors etc.
The last 2 times we where able to see, that both tomcat by them self where
alive. Just ajp on both where dead. We couldn't make a connection either
trough 7009 nor 9009. An with our openFiles trick we found a lot of
close_wait again, e.g. 200 for 9009. I left the second tomcat on this state
for a few ours just to see, what happens. The count of 200 connection with
close_wait was kept until a reboot of the tomcat.
Instead of rebooting Tomcat, try to force the Tomcat JVM to do a Major Garbage
Collection.
There are a number of tools that allow to do that.
One command-line one which I found practical is jmxsh, here :
http://code.google.com/p/jmxsh/
If when you do a Major GC, these CLOSE_WAIT connections disappear, you will
have learned
something about their origin.
And if then - without restarting Tomcat - you can connect again via the AJP
ports, you'll
have learned something else.
Go do it and report.
I would say with some of our reconfiguration we managed to stop increasing
connections. But still we are not sure why our ajp connections dying..
Here is our connector out of Server.xml:
Connector port=9009 protocol=AJP/1.3 redirectPort=9443
maxThreads=200 connectionTimeout=60 /
worker.properties:
worker.tomcatX.host=localhost
worker.tomcatX.type=ajp13
worker.tomcatX.fail_on_status=404
worker.tomcatX.lbfactor=1
worker.tomcatX.ping_timeout=1000
worker.tomcatX.ping_mode=A
worker.tomcatX.socket_timeout=10
worker.tomcatX.connection_pool_timeout=600
worker.tomcat1.reference=worker.tomcatX
worker.tomcat1.port=7009
worker.tomcat2.reference=worker.tomcatX
worker.tomcat2.port=9009
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=tomcat1,tomcat2
worker.status.type=status
Hopefully one of you guys can give us a hint to fix that problem.
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-Ursprüngliche Nachricht-
Von: David Kumar [mailto:dku...@telestar.de]
Gesendet: Dienstag, 22. Januar 2013 07:36
An: Tomcat Users List
Betreff: AW: AW: AW: ajp on 7009 and 9009 : connections keept open
Hey,
last friday I changed our configuration to use a executor.
Here is what I did:
Connector port=7009 protocol=AJP/1.3 redirectPort=8443
maxThreads=200 executor=active-executor /
Executor name=active-executor namePrefix=activeThread- maxThreads=200
minSpareThreads=30 maxIdleTime=6 /
Connector port=7080 protocol=HTTP/1.1
connectionTimeout=2
redirectPort=8443 executor=active-executor /
The second tomcat has same configuration besides ports..
Until yesterday it worked like a charm. But at late afternoon one of the
tomcats failed again..
I couldn't start the garbagecollection so far..
Any other ideas?
Thanks
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-Ursprüngliche Nachricht-
Von: David Kumar [mailto:dku...@telestar.de]
Gesendet: Freitag, 18. Januar 2013 11:19
An: Tomcat Users List; Tomcat Users List
Betreff: AW: AW: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections
keept open
Hey,
I do that at next deployment. -- I Thursday...
So far I'm trying executor for tomcat. As far I read, when I'm using
connectors idle process are forced to be close..
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D
AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hey,
I'm really not sure if that tool does anything, because there is no feedback..
Funny: Just a few minutes ago a colleague showed ma a tool called something
like VM monitor on his mac. It look a kind of similar to jconsole.exe. Next
time I will try to make GC with this tool.
Meanwhile I reduced the connectionTimeout, reconfigured maxClients and added
-DisableReuse to jkpropertys.
Lets see what happens...
Thanks a lot..
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-Ursprüngliche Nachricht-
Von: André Warnier [mailto:a...@ice-sa.com]
Gesendet: Mittwoch, 13. März 2013 12:20
An: Tomcat Users List
Betreff: Re: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
David Kumar wrote:
Hey,
we just had that problem again. I did that GC trick, it didn't work out well.
I used:
set hosts [list]
lappend hosts {localhost:7008}
lappend hosts {localhost:9008}
# ...add as many as you want...
foreach {host} $hosts {
set parts [split $host :]
set hostname [lindex $parts 0]
set port [lindex $parts 1]
# for each host...
# Connect to it.
jmx_connect -h $hostname -p $port
# Invoke the garbage collector.
jmx_invoke -n -m java.lang:type=Memory gc
# Close this connection
jmx_close
}
I found that while googleing.
But it doesn't have any effect. Any other ideas?
Did it really connect with the Tomcat JVM and really do anything ?
Is your JVM started with options that allow JMX interaction ?
Also, there are ways to start the JVM with parameters telling it to log the
Garbage
Collections to a logfile. Make sure that you do that, and check the logfile to
see if it
really does a GC when you ask it to.
Note that there are also other ways to trogger a GC than using jmxsh. I was
only pointing
you to that tool as an easy-to-script, non-GUI tool.
But in your java directory (at least the JDK/bin), there should also be a
jconsole
program which allows you to do pretty much the same interactively with a GUI.
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hey,
right no we're having our Problem.
I tried gc through jconsole -- no changes and still a lot CLOSE_WAIT.
So it is not a GC Problem and disablereuse doesn't work either..
Any other ideas?
What do you guy think about proxy_ajp instead of jk? What are the advantages of
proxy_ajp?
Thanks a lot..
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-Ursprüngliche Nachricht-
Von: David Kumar [mailto:dku...@telestar.de]
Gesendet: Mittwoch, 13. März 2013 14:32
An: Tomcat Users List
Betreff: AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Hey,
I'm really not sure if that tool does anything, because there is no feedback..
Funny: Just a few minutes ago a colleague showed ma a tool called something
like VM monitor on his mac. It look a kind of similar to jconsole.exe. Next
time I will try to make GC with this tool.
Meanwhile I reduced the connectionTimeout, reconfigured maxClients and added
-DisableReuse to jkpropertys.
Lets see what happens...
Thanks a lot..
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-Ursprüngliche Nachricht-
Von: André Warnier [mailto:a...@ice-sa.com]
Gesendet: Mittwoch, 13. März 2013 12:20
An: Tomcat Users List
Betreff: Re: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
David Kumar wrote:
Hey,
we just had that problem again. I did that GC trick, it didn't work out well.
I used:
set hosts [list]
lappend hosts {localhost:7008}
lappend hosts {localhost:9008}
# ...add as many as you want...
foreach {host} $hosts {
set parts [split $host :]
set hostname [lindex $parts 0]
set port [lindex $parts 1]
# for each host...
# Connect to it.
jmx_connect -h $hostname -p $port
# Invoke the garbage collector.
jmx_invoke -n -m java.lang:type=Memory gc
# Close this connection
jmx_close
}
I found that while googleing.
But it doesn't have any effect. Any other ideas?
Did it really connect with the Tomcat JVM and really do anything ?
Is your JVM started with options that allow JMX interaction ?
Also, there are ways to start the JVM with parameters telling it to log the
Garbage
Collections to a logfile. Make sure that you do that, and check the logfile to
see if it
really does a GC when you ask it to.
Note that there are also other ways to trogger a GC than using jmxsh. I was
only pointing
you to that tool as an easy-to-script, non-GUI tool.
But in your java directory (at least the JDK/bin), there should also be a
jconsole
program which allows you to do pretty much the same interactively with a GUI.
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
BTW... after a while CLOSE_WAIT changes to can't identify protocol
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
Tel.: + 49 (0) 6592 / 712 -2826
Tel.: + 49 (0) 2676 / 9520 -183
Fax: + 49 (0) 6592 / 712 -2829
http://www.telestar.de/
-Ursprüngliche Nachricht-
Von: David Kumar
Gesendet: Mittwoch, 13. März 2013 20:01
An: 'Tomcat Users List'
Betreff: AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Hey,
right no we're having our Problem.
I tried gc through jconsole -- no changes and still a lot CLOSE_WAIT.
So it is not a GC Problem and disablereuse doesn't work either..
Any other ideas?
What do you guy think about proxy_ajp instead of jk? What are the advantages of
proxy_ajp?
Thanks a lot..
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-Ursprüngliche Nachricht-
Von: David Kumar [mailto:dku...@telestar.de]
Gesendet: Mittwoch, 13. März 2013 14:32
An: Tomcat Users List
Betreff: AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Hey,
I'm really not sure if that tool does anything, because there is no feedback..
Funny: Just a few minutes ago a colleague showed ma a tool called something
like VM monitor on his mac. It look a kind of similar to jconsole.exe. Next
time I will try to make GC with this tool.
Meanwhile I reduced the connectionTimeout, reconfigured maxClients and added
-DisableReuse to jkpropertys.
Lets see what happens...
Thanks a lot..
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-Ursprüngliche Nachricht-
Von: André Warnier [mailto:a...@ice-sa.com]
Gesendet: Mittwoch, 13. März 2013 12:20
An: Tomcat Users List
Betreff: Re: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
David Kumar wrote:
Hey,
we just had that problem again. I did that GC trick, it didn't work out well.
I used:
set hosts [list]
lappend hosts {localhost:7008}
lappend hosts {localhost:9008}
# ...add as many as you want...
foreach {host} $hosts {
set parts [split $host :]
set hostname [lindex $parts 0]
set port [lindex $parts 1]
# for each host...
# Connect to it.
jmx_connect -h $hostname -p $port
# Invoke the garbage collector.
jmx_invoke -n -m java.lang:type=Memory gc
# Close this connection
jmx_close
}
I found that while googleing.
But it doesn't have any effect. Any other ideas?
Did it really connect with the Tomcat JVM and really do anything ?
Is your JVM started with options that allow JMX interaction ?
Also, there are ways to start the JVM with parameters telling it to log the
Garbage
Collections to a logfile. Make sure that you do that, and check the logfile to
see if it
really does a GC when you ask it to.
Note that there are also other ways to trogger a GC than using jmxsh. I was
only pointing
you to that tool as an easy-to-script, non-GUI tool.
But in your java directory (at least the JDK/bin), there should also be a
jconsole
program which allows you to do pretty much the same interactively with a GUI.
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hey, we are still having that issue. But we could manage to figure out some more stuff. We made a Tomcat and Java update since that time we had our problem a few times again, also we did some reconfiguration with connectors etc. The last 2 times we where able to see, that both tomcat by them self where alive. Just ajp on both where dead. We couldn't make a connection either trough 7009 nor 9009. An with our openFiles trick we found a lot of close_wait again, e.g. 200 for 9009. I left the second tomcat on this state for a few ours just to see, what happens. The count of 200 connection with close_wait was kept until a reboot of the tomcat. I would say with some of our reconfiguration we managed to stop increasing connections. But still we are not sure why our ajp connections dying.. Here is our connector out of Server.xml: Connector port=9009 protocol=AJP/1.3 redirectPort=9443 maxThreads=200 connectionTimeout=60 / worker.properties: worker.tomcatX.host=localhost worker.tomcatX.type=ajp13 worker.tomcatX.fail_on_status=404 worker.tomcatX.lbfactor=1 worker.tomcatX.ping_timeout=1000 worker.tomcatX.ping_mode=A worker.tomcatX.socket_timeout=10 worker.tomcatX.connection_pool_timeout=600 worker.tomcat1.reference=worker.tomcatX worker.tomcat1.port=7009 worker.tomcat2.reference=worker.tomcatX worker.tomcat2.port=9009 worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=tomcat1,tomcat2 worker.status.type=status Hopefully one of you guys can give us a hint to fix that problem. Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: David Kumar [mailto:dku...@telestar.de] Gesendet: Dienstag, 22. Januar 2013 07:36 An: Tomcat Users List Betreff: AW: AW: AW: ajp on 7009 and 9009 : connections keept open Hey, last friday I changed our configuration to use a executor. Here is what I did: Connector port=7009 protocol=AJP/1.3 redirectPort=8443 maxThreads=200 executor=active-executor / Executor name=active-executor namePrefix=activeThread- maxThreads=200 minSpareThreads=30 maxIdleTime=6 / Connector port=7080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 executor=active-executor / The second tomcat has same configuration besides ports.. Until yesterday it worked like a charm. But at late afternoon one of the tomcats failed again.. I couldn't start the garbagecollection so far.. Any other ideas? Thanks Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: David Kumar [mailto:dku...@telestar.de] Gesendet: Freitag, 18. Januar 2013 11:19 An: Tomcat Users List; Tomcat Users List Betreff: AW: AW: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open Hey, I do that at next deployment. -- I Thursday... So far I'm trying executor for tomcat. As far I read, when I'm using connectors idle process are forced to be close.. Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: André Warnier [mailto:a...@ice-sa.com] Gesendet: Freitag, 18. Januar 2013 11:10 An: Tomcat Users List Betreff: Re: AW: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open David Kumar wrote: Hey André, are you talking about running System.gc()? Yes. That should be possible.. Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: André Warnier [mailto:a...@ice-sa.com] Gesendet: Freitag, 18. Januar 2013 10:07 An: Tomcat Users List Betreff: Re: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open David, (and sorry for top-posting here) just to verify something. Can you trigger a Major Garbage Collection at the Tomcat JVM level, at a moment when you have all these connections in CLOSE_WAIT, and see if they disappear after the GC ? If yes, it may give a good clue about where all these CLOSE_WAITs are coming from. David Kumar wrote: Just read this email.. :-) I figured out we are not using executor connector... Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: David Kumar [mailto:dku...@telestar.de] Gesendet: Freitag, 18. Januar 2013 09:11 An: Tomcat Users List Betreff: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hey Mark, thanks for reply. I do have some more questions as the main configuration i not done by myself. We are using Apache 2.2.16 on Debian. Therefore MaxRequestWorkers is called MaxClients, isn't it? Currently it is set to 312. So if we have two tomcats with 200 thread each MaxClients is to low? I either should reduce the Thread at the connector or increase MaxClients? We got the connection_pool_timeout from here: https://community.jboss.org/wiki/OptimalModjk12Configuration I will have a look on the other recommended options Thanks.. Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: Mark Thomas [mailto:ma...@apache.org] Gesendet: Dienstag, 12. März 2013 10:25 An: Tomcat Users List Betreff: Re: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open On 12/03/2013 06:53, David Kumar wrote: Hey, we are still having that issue. But we could manage to figure out some more stuff. We made a Tomcat and Java update since that time we had our problem a few times again, also we did some reconfiguration with connectors etc. The last 2 times we where able to see, that both tomcat by them self where alive. Just ajp on both where dead. We couldn't make a connection either trough 7009 nor 9009. An with our openFiles trick we found a lot of close_wait again, e.g. 200 for 9009. I left the second tomcat on this state for a few ours just to see, what happens. The count of 200 connection with close_wait was kept until a reboot of the tomcat. I would say with some of our reconfiguration we managed to stop increasing connections. But still we are not sure why our ajp connections dying.. Here is our connector out of Server.xml: Connector port=9009 protocol=AJP/1.3 redirectPort=9443 maxThreads=200 connectionTimeout=60 / Only 200 threads on the Tomcat side. If httpd's MaxClients/MaxRequestWorkers is greater than 200 you may get thread starvation in Tomcat. worker.properties: worker.tomcatX.host=localhost worker.tomcatX.type=ajp13 worker.tomcatX.fail_on_status=404 That is a really bad idea. A single 404 and the entire Tomcat instance gets taken out of the loadbalancer for 60 seconds. Hello DOS attack. worker.tomcatX.lbfactor=1 worker.tomcatX.ping_timeout=1000 worker.tomcatX.ping_mode=A worker.tomcatX.socket_timeout=10 worker.tomcatX.connection_pool_timeout=600 10 minutes is a long time to keep a persistent connection around. With even a moderate load you'll easily get to 200 connections in a 10 minute period. worker.tomcat1.reference=worker.tomcatX worker.tomcat1.port=7009 worker.tomcat2.reference=worker.tomcatX worker.tomcat2.port=9009 worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=tomcat1,tomcat2 worker.status.type=status Hopefully one of you guys can give us a hint to fix that problem. Do one of the following: 1. Increase maxThreads in Tomcat's connector to MaxRequestWorkers 2. Use JkOptions +DisableReuse (ignore the performance warnings) 3. Reduce the connection_pool_timeout Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hey, thanks.. I downloaded jmxsh... I keep you guy up to date.. thanks... Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: André Warnier [mailto:a...@ice-sa.com] Gesendet: Dienstag, 12. März 2013 09:54 An: Tomcat Users List Betreff: Re: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open David Kumar wrote: Hey, we are still having that issue. But we could manage to figure out some more stuff. We made a Tomcat and Java update since that time we had our problem a few times again, also we did some reconfiguration with connectors etc. The last 2 times we where able to see, that both tomcat by them self where alive. Just ajp on both where dead. We couldn't make a connection either trough 7009 nor 9009. An with our openFiles trick we found a lot of close_wait again, e.g. 200 for 9009. I left the second tomcat on this state for a few ours just to see, what happens. The count of 200 connection with close_wait was kept until a reboot of the tomcat. Instead of rebooting Tomcat, try to force the Tomcat JVM to do a Major Garbage Collection. There are a number of tools that allow to do that. One command-line one which I found practical is jmxsh, here : http://code.google.com/p/jmxsh/ If when you do a Major GC, these CLOSE_WAIT connections disappear, you will have learned something about their origin. And if then - without restarting Tomcat - you can connect again via the AJP ports, you'll have learned something else. Go do it and report. I would say with some of our reconfiguration we managed to stop increasing connections. But still we are not sure why our ajp connections dying.. Here is our connector out of Server.xml: Connector port=9009 protocol=AJP/1.3 redirectPort=9443 maxThreads=200 connectionTimeout=60 / worker.properties: worker.tomcatX.host=localhost worker.tomcatX.type=ajp13 worker.tomcatX.fail_on_status=404 worker.tomcatX.lbfactor=1 worker.tomcatX.ping_timeout=1000 worker.tomcatX.ping_mode=A worker.tomcatX.socket_timeout=10 worker.tomcatX.connection_pool_timeout=600 worker.tomcat1.reference=worker.tomcatX worker.tomcat1.port=7009 worker.tomcat2.reference=worker.tomcatX worker.tomcat2.port=9009 worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=tomcat1,tomcat2 worker.status.type=status Hopefully one of you guys can give us a hint to fix that problem. Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: David Kumar [mailto:dku...@telestar.de] Gesendet: Dienstag, 22. Januar 2013 07:36 An: Tomcat Users List Betreff: AW: AW: AW: ajp on 7009 and 9009 : connections keept open Hey, last friday I changed our configuration to use a executor. Here is what I did: Connector port=7009 protocol=AJP/1.3 redirectPort=8443 maxThreads=200 executor=active-executor / Executor name=active-executor namePrefix=activeThread- maxThreads=200 minSpareThreads=30 maxIdleTime=6 / Connector port=7080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 executor=active-executor / The second tomcat has same configuration besides ports.. Until yesterday it worked like a charm. But at late afternoon one of the tomcats failed again.. I couldn't start the garbagecollection so far.. Any other ideas? Thanks Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: David Kumar [mailto:dku...@telestar.de] Gesendet: Freitag, 18. Januar 2013 11:19 An: Tomcat Users List; Tomcat Users List Betreff: AW: AW: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open Hey, I do that at next deployment. -- I Thursday... So far I'm trying executor for tomcat. As far I read, when I'm using connectors idle process are forced to be close.. Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: André Warnier [mailto:a...@ice-sa.com] Gesendet: Freitag, 18. Januar 2013 11:10 An: Tomcat Users List Betreff: Re: AW: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open David Kumar wrote: Hey André, are you talking about running System.gc()? Yes. That should be possible.. Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14
AW: AW: AW: ajp on 7009 and 9009 : connections keept open
Hey, last friday I changed our configuration to use a executor. Here is what I did: Connector port=7009 protocol=AJP/1.3 redirectPort=8443 maxThreads=200 executor=active-executor / Executor name=active-executor namePrefix=activeThread- maxThreads=200 minSpareThreads=30 maxIdleTime=6 / Connector port=7080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 executor=active-executor / The second tomcat has same configuration besides ports.. Until yesterday it worked like a charm. But at late afternoon one of the tomcats failed again.. I couldn't start the garbagecollection so far.. Any other ideas? Thanks Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: David Kumar [mailto:dku...@telestar.de] Gesendet: Freitag, 18. Januar 2013 11:19 An: Tomcat Users List; Tomcat Users List Betreff: AW: AW: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open Hey, I do that at next deployment. -- I Thursday... So far I'm trying executor for tomcat. As far I read, when I'm using connectors idle process are forced to be close.. Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen Tel.: + 49 (0) 6592 / 712 -2826 Tel.: + 49 (0) 2676 / 9520 -183 Fax: + 49 (0) 6592 / 712 -2829 http://www.telestar.de/ -Ursprüngliche Nachricht- Von: André Warnier [mailto:a...@ice-sa.com] Gesendet: Freitag, 18. Januar 2013 11:10 An: Tomcat Users List Betreff: Re: AW: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open David Kumar wrote: Hey André, are you talking about running System.gc()? Yes. That should be possible.. Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: André Warnier [mailto:a...@ice-sa.com] Gesendet: Freitag, 18. Januar 2013 10:07 An: Tomcat Users List Betreff: Re: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open David, (and sorry for top-posting here) just to verify something. Can you trigger a Major Garbage Collection at the Tomcat JVM level, at a moment when you have all these connections in CLOSE_WAIT, and see if they disappear after the GC ? If yes, it may give a good clue about where all these CLOSE_WAITs are coming from. David Kumar wrote: Just read this email.. :-) I figured out we are not using executor connector... Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: David Kumar [mailto:dku...@telestar.de] Gesendet: Freitag, 18. Januar 2013 09:11 An: Tomcat Users List Betreff: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open here you are with attachment :-) btw: in mod_jk.log I found some [Thu Jan 17 23:00:08 2013] [11196:140336689317632] [error] ajp_get_reply::jk_ajp_common.c (2055): (tomcat2) Tomcat is down or refused connection. No response has been sent to the client (yet) [Thu Jan 17 23:00:08 2013] [11196:140336689317632] [error] ajp_service::jk_ajp_common.c (2559): (tomcat2) connecting to tomcat failed. but realy just a few one... Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: David Kumar Gesendet: Freitag, 18. Januar 2013 09:08 An: 'Tomcat Users List' Betreff: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open Hey, thanks for reply. I got that about the Apache configuration. Since we had our problem yesterday, again and there was no error at the apache logs I'm willing to say that is not the main problem and I have to check that, when my main problem is solved.. :-) I agree with you about wrong reporting of service. Its just shown up as afs3 because these service uses 7009 per default. But I'm using 7009 and 9009 for ajp. So doesn't this mean there is a connection problems between my Apache and the tomcats? You're right, both Webapps doing the same and are configured identically besides the ports. I'm using more than one database, but all of them are used through a database pool. If there is a bug, I think I should have found some error at my logs like no free connection or something like that. As there is no such log entry I'm willing to say that my database connections processing like they should. Basically on each tomcat there are running two services. One
( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open
Hey, thanks for reply. I got that about the Apache configuration. Since we had our problem yesterday, again and there was no error at the apache logs I'm willing to say that is not the main problem and I have to check that, when my main problem is solved.. :-) I agree with you about wrong reporting of service. Its just shown up as afs3 because these service uses 7009 per default. But I'm using 7009 and 9009 for ajp. So doesn't this mean there is a connection problems between my Apache and the tomcats? You're right, both Webapps doing the same and are configured identically besides the ports. I'm using more than one database, but all of them are used through a database pool. If there is a bug, I think I should have found some error at my logs like no free connection or something like that. As there is no such log entry I'm willing to say that my database connections processing like they should. Basically on each tomcat there are running two services. One is a axis2 project. Our CRM is posting customer data to this webapp. This data will be persisted into a database. Depending on the information given by our CRM axis sends a email. The second one is basically a cache for our websites. We have a PIM with all our product data. These app is gathering all the data from PIM and a CMS and is merging these information together so that the data can be displayed. All the mentioned data is hold in different cache objects. Also some communication with our ERP and some databases are made trough this app. The second app is a REST service. Information will be posted as POST or GET request to it. Most likely the responses are JSON Object. When ever one webApp is reloading (automatically or manually) itself, the result will be posted to the other tomcat/webapp as a serialized object, so the other on do not need to reload it self. I can't say how many SMB files there are, it is depending on some other stuff so it is dynamic. Attached you can find a printed list by lsof. There you can see a really strange thing. Yesterday just tomcat2 had the problem with to many open files. A few days before it was just tomcat1 having this problem. Now let my answer your question: 1. That is hard to say, I guess I have to do some more investigation on our logfiles. 2. / 3. Here is my httpd.conf: IfModule mpm_worker_module ThreadLimit 25 StartServers 2 MaxClients 150 MinSpareThreads 25 MaxSpareThreads 75 ThreadsPerChild 25 MaxRequestsPerChild 4000 /IfModule we are using worker And here are our tomcat connectors again: tomcat1: Connector port=7080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 / Connector executor=tomcatThreadPool port=7080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 / Connector port=8443 protocol=HTTP/1.1 SSLEnabled=true maxThreads=150 scheme=https secure=true clientAuth=false sslProtocol=TLS / Connector port=7009 protocol=AJP/1.3 redirectPort=8443 / tomcat2: Connector port=9080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=9443 / Connector executor=tomcatThreadPool port=8080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 / Connector port=8443 protocol=HTTP/1.1 SSLEnabled=true maxThreads=150 scheme=https secure=true clientAuth=false sslProtocol=TLS / Connector port=9009 protocol=AJP/1.3 redirectPort=9443 / as you can see we are using a executor... You probably read my copy-paste error. I did copy some comments out of out server config -- Sry again. 4. we are using.. 5. via a multipart message sending to the other tomcat. 6. I don't think so also because of that the connections are kept open on our ajp ports. I know that CLOSE_WAIT means, waiting for connections to be closed, but wondering that it is not closing.. Thanks again Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: Christopher Schultz [mailto:ch...@christopherschultz.net] Gesendet: Donnerstag, 17. Januar 2013 18:38 An: Tomcat Users List Betreff: Re: AW: AW: afs3-rmtsys: connections keept open -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 David, On 1/17/13 1:49 AM, David Kumar wrote: I just checked /var/logs/apache2/error.logs. And found following errors: [Wed Jan 16 15:14:46 2013] [error] server is within MinSpareThreads of MaxClients, consider raising the MaxClients setting [Wed Jan 16 15:14:56 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting So you are maxing-out your connections: you
AW: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open
Hey André, are you talking about running System.gc()? That should be possible.. Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: André Warnier [mailto:a...@ice-sa.com] Gesendet: Freitag, 18. Januar 2013 10:07 An: Tomcat Users List Betreff: Re: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open David, (and sorry for top-posting here) just to verify something. Can you trigger a Major Garbage Collection at the Tomcat JVM level, at a moment when you have all these connections in CLOSE_WAIT, and see if they disappear after the GC ? If yes, it may give a good clue about where all these CLOSE_WAITs are coming from. David Kumar wrote: Just read this email.. :-) I figured out we are not using executor connector... Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: David Kumar [mailto:dku...@telestar.de] Gesendet: Freitag, 18. Januar 2013 09:11 An: Tomcat Users List Betreff: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open here you are with attachment :-) btw: in mod_jk.log I found some [Thu Jan 17 23:00:08 2013] [11196:140336689317632] [error] ajp_get_reply::jk_ajp_common.c (2055): (tomcat2) Tomcat is down or refused connection. No response has been sent to the client (yet) [Thu Jan 17 23:00:08 2013] [11196:140336689317632] [error] ajp_service::jk_ajp_common.c (2559): (tomcat2) connecting to tomcat failed. but realy just a few one... Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: David Kumar Gesendet: Freitag, 18. Januar 2013 09:08 An: 'Tomcat Users List' Betreff: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open Hey, thanks for reply. I got that about the Apache configuration. Since we had our problem yesterday, again and there was no error at the apache logs I'm willing to say that is not the main problem and I have to check that, when my main problem is solved.. :-) I agree with you about wrong reporting of service. Its just shown up as afs3 because these service uses 7009 per default. But I'm using 7009 and 9009 for ajp. So doesn't this mean there is a connection problems between my Apache and the tomcats? You're right, both Webapps doing the same and are configured identically besides the ports. I'm using more than one database, but all of them are used through a database pool. If there is a bug, I think I should have found some error at my logs like no free connection or something like that. As there is no such log entry I'm willing to say that my database connections processing like they should. Basically on each tomcat there are running two services. One is a axis2 project. Our CRM is posting customer data to this webapp. This data will be persisted into a database. Depending on the information given by our CRM axis sends a email. The second one is basically a cache for our websites. We have a PIM with all our product data. These app is gathering all the data from PIM and a CMS and is merging these information together so that the data can be displayed. All the mentioned data is hold in different cache objects. Also some communication with our ERP and some databases are made trough this app. The second app is a REST service. Information will be posted as POST or GET request to it. Most likely the responses are JSON Object. When ever one webApp is reloading (automatically or manually) itself, the result will be posted to the other tomcat/webapp as a serialized object, so the other on do not need to reload it self. I can't say how many SMB files there are, it is depending on some other stuff so it is dynamic. Attached you can find a printed list by lsof. There you can see a really strange thing. Yesterday just tomcat2 had the problem with to many open files. A few days before it was just tomcat1 having this problem. Now let my answer your question: 1. That is hard to say, I guess I have to do some more investigation on our logfiles. 2. / 3. Here is my httpd.conf: IfModule mpm_worker_module ThreadLimit 25 StartServers 2 MaxClients 150 MinSpareThreads 25 MaxSpareThreads 75 ThreadsPerChild 25 MaxRequestsPerChild 4000 /IfModule we are using worker And here are our tomcat connectors again: tomcat1: Connector port=7080 protocol=HTTP/1.1 connectionTimeout
AW: AW: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open
Hey, I do that at next deployment. -- I Thursday... So far I'm trying executor for tomcat. As far I read, when I'm using connectors idle process are forced to be close.. Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen Tel.: + 49 (0) 6592 / 712 -2826 Tel.: + 49 (0) 2676 / 9520 -183 Fax: + 49 (0) 6592 / 712 -2829 http://www.telestar.de/ -Ursprüngliche Nachricht- Von: André Warnier [mailto:a...@ice-sa.com] Gesendet: Freitag, 18. Januar 2013 11:10 An: Tomcat Users List Betreff: Re: AW: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open David Kumar wrote: Hey André, are you talking about running System.gc()? Yes. That should be possible.. Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: André Warnier [mailto:a...@ice-sa.com] Gesendet: Freitag, 18. Januar 2013 10:07 An: Tomcat Users List Betreff: Re: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open David, (and sorry for top-posting here) just to verify something. Can you trigger a Major Garbage Collection at the Tomcat JVM level, at a moment when you have all these connections in CLOSE_WAIT, and see if they disappear after the GC ? If yes, it may give a good clue about where all these CLOSE_WAITs are coming from. David Kumar wrote: Just read this email.. :-) I figured out we are not using executor connector... Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: David Kumar [mailto:dku...@telestar.de] Gesendet: Freitag, 18. Januar 2013 09:11 An: Tomcat Users List Betreff: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open here you are with attachment :-) btw: in mod_jk.log I found some [Thu Jan 17 23:00:08 2013] [11196:140336689317632] [error] ajp_get_reply::jk_ajp_common.c (2055): (tomcat2) Tomcat is down or refused connection. No response has been sent to the client (yet) [Thu Jan 17 23:00:08 2013] [11196:140336689317632] [error] ajp_service::jk_ajp_common.c (2559): (tomcat2) connecting to tomcat failed. but realy just a few one... Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: David Kumar Gesendet: Freitag, 18. Januar 2013 09:08 An: 'Tomcat Users List' Betreff: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open Hey, thanks for reply. I got that about the Apache configuration. Since we had our problem yesterday, again and there was no error at the apache logs I'm willing to say that is not the main problem and I have to check that, when my main problem is solved.. :-) I agree with you about wrong reporting of service. Its just shown up as afs3 because these service uses 7009 per default. But I'm using 7009 and 9009 for ajp. So doesn't this mean there is a connection problems between my Apache and the tomcats? You're right, both Webapps doing the same and are configured identically besides the ports. I'm using more than one database, but all of them are used through a database pool. If there is a bug, I think I should have found some error at my logs like no free connection or something like that. As there is no such log entry I'm willing to say that my database connections processing like they should. Basically on each tomcat there are running two services. One is a axis2 project. Our CRM is posting customer data to this webapp. This data will be persisted into a database. Depending on the information given by our CRM axis sends a email. The second one is basically a cache for our websites. We have a PIM with all our product data. These app is gathering all the data from PIM and a CMS and is merging these information together so that the data can be displayed. All the mentioned data is hold in different cache objects. Also some communication with our ERP and some databases are made trough this app. The second app is a REST service. Information will be posted as POST or GET request to it. Most likely the responses are JSON Object. When ever one webApp is reloading (automatically or manually) itself, the result will be posted to the other tomcat/webapp as a serialized object, so the other on do not need to reload it self. I can't say how many SMB files there are, it is depending on some other stuff so it is dynamic. Attached you can find a printed list by lsof. There you can see a really strange thing
afs3-rmtsys: connections keept open
Hey, my name is David an I'm working for a electronic consumer company in Germany we are selling and producing TV's, receiver, satellite dishes etc.. We got some serious trouble with our tomcats. Basically we using two 7.0.12 tomcats on a single Debian 6.0. One is running on port 7090 and the other on 9080. Time after time both tomcats are crashing. I found too many open files in tomcatlogfiles. Both Servers are running with one user (TOMCAT). After changing max open files for that particular user I was able to get more than 1024 open files. Currently I'm checking log files every few minutes. With lsof|grep tomcat|wc -l I'm counting the open files by user. Usually there are around 600-700 open files. But sometimes the count grows and we have more than 1200 open files. When ever the server does have so many open files I checked: lsof -u tomcat an figured out there are many connection waiting to be closed. The connections are looking like this: java22312 tomcat 153u IPv61508517 0t0 TCP localhost:afs3-rmtsys-localhost:50127 (CLOSE_WAIT) I googled a bit for afs3-rmtsys and figured out that the protocol seems to be a remote cache manager. But I couldn't find any more information. Does anyone of you know what my problem might be? Thanks! Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen www.telestar.dehttp://www.telestar.de/ www.facebook.com/TelestarDigitalhttp://www.facebook.com/TelestarDigital
AW: afs3-rmtsys: connections keept open
Hey, thanks for replying.. I forgot to mention that we have a Apache being a loadbalancer in front of these two tomcats. The Apache uses port 8080. Let me start with André Warnier: Both of the tomcats are synchronising them self. The send some serialized objects via http to each other. And both of them getting some file from SMB shares. But I can't imagine that might be the problem? I'm wondering why the tcp connections with state CLOSE_WAIT doesn't get closed. Here are the requested connectors tomcat1: Connector port=7080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 / Connector executor=tomcatThreadPool port=7080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 / Connector port=8443 protocol=HTTP/1.1 SSLEnabled=true maxThreads=150 scheme=https secure=true clientAuth=false sslProtocol=TLS / Connector port=7009 protocol=AJP/1.3 redirectPort=8443 / tomcat2: Connector port=9080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=9443 / Connector executor=tomcatThreadPool port=8080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 / Connector port=8443 protocol=HTTP/1.1 SSLEnabled=true maxThreads=150 scheme=https secure=true clientAuth=false sslProtocol=TLS / Connector port=9009 protocol=AJP/1.3 redirectPort=9443 / Yeah I'm sure its port 7090, as you can see above.. :-) 7009 is used by AJP!? As I had to restart the server, I don't know who was the owner of that PID. Right now we do have 2 or 3 of: java9495 tomcat 201u IPv61866645 0t0 TCP localhost:afs3-rmtsys-localhost:55116 (ESTABLISHED) If I run ulimit -n with the tomcat user I get: 1024. But I did set the max open files at the startscript for each tomcat (init.d) with: ulimit -Hn 4096 ulimit -Sn 4096 And if I check the files for the current tomcat process (cat proc/9180/limits while 9180 is one of my tomcat PIDs) I get: Max open files4096 4096 files Thanks again.. Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: Christopher Schultz [mailto:ch...@christopherschultz.net] Gesendet: Mittwoch, 16. Januar 2013 19:14 An: Tomcat Users List Betreff: Re: afs3-rmtsys: connections keept open -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 David, On 1/16/13 10:52 AM, David Kumar wrote: We got some serious trouble with our tomcats. Basically we using two 7.0.12 tomcats on a single Debian 6.0. One is running on port 7090 and the other on 9080. Time after time both tomcats are crashing. I found too many open files in tomcatlogfiles. Bummer. Both Servers are running with one user (TOMCAT). Can you post the Connector parts of your server.xml files (from both Tomcat configurations)? Also, what does `ulimit -n` return from an otherwise unmodified shell running as the user who owns Tomcat processes? After changing max open files for that particular user I was able to get more than 1024 open files. Currently I'm checking log files every few minutes. With lsof|grep tomcat|wc -l I'm counting the open files by user. Usually there are around 600-700 open files. But sometimes the count grows and we have more than 1200 open files. You'll get file handled opened for both real files (like /tmp/foo, etc.) and for socket connections. If you aren't careful, you can have a configuration that will work for a while bit tip-over under load. When ever the server does have so many open files I checked: lsof -u tomcat an figured out there are many connection waiting to be closed. The connections are looking like this: java22312 tomcat 153u IPv61508517 0t0 TCP localhost:afs3-rmtsys-localhost:50127 (CLOSE_WAIT) Are you sure your port is 7090 and not 7009? Is 22312 the right process id for Tomcat? The well-known port number for the afs3-rmtsys service is 7009. It would be odd, though, for your Java process to be calling *out* from a port like that ... usually you get a high-numbered port when making an outgoing connection. Who owns localhost:50127 in that case? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEAREIAAYFAlD27f4ACgkQ9CaO5/Lv0PD7RQCfd7PaA5eY4pANZY6NYhGB3OmN 5+4AoJ9GihEA9NZr2BoCIARfBTaTqs8T =x9/z -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AW: afs3-rmtsys: connections keept open
Hey, I changed the global filehandle as well. And executing ulimit with the tomcat starting Thread is working well. I yesterday I hab around 1600 connection before I rebooted both of them. About Server config. I did a mistake. some of the connector are just comments. I didn't see that yesterday when I was editing the files within a shell. So here are the working connectors: tomcat1: Connector port=7080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 / Connector port=7009 protocol=AJP/1.3 redirectPort=8443 / tomcat2: Connector port=9080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=9443 / Connector port=9009 protocol=AJP/1.3 redirectPort=9443 / thanks.. Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: Christopher Schultz [mailto:ch...@christopherschultz.net] Gesendet: Mittwoch, 16. Januar 2013 22:51 An: Tomcat Users List Betreff: Re: AW: afs3-rmtsys: connections keept open -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 David, On 1/16/13 3:41 PM, David Kumar wrote: I forgot to mention that we have a Apache being a loadbalancer in front of these two tomcats. The Apache uses port 8080. So, Apache httpd listens on port 8080? Why not port 80? Here are the requested connectors tomcat1: Connector port=7080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 / Connector executor=tomcatThreadPool port=7080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 / Is that a double-paste error? You have two connectors both with port=7080. Connector port=8443 protocol=HTTP/1.1 SSLEnabled=true maxThreads=150 scheme=https secure=true clientAuth=false sslProtocol=TLS / Connector port=7009 protocol=AJP/1.3 redirectPort=8443 / tomcat2: Connector port=9080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=9443 / Connector executor=tomcatThreadPool port=8080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 / Connector port=8443 protocol=HTTP/1.1 SSLEnabled=true maxThreads=150 scheme=https secure=true clientAuth=false sslProtocol=TLS / Connector port=9009 protocol=AJP/1.3 redirectPort=9443 / Weird: you have httpd-lb out front plus 4 connectors in each Tomcat? Yeah I'm sure its port 7090, as you can see above.. :-) 7009 is used by AJP!? Yes, you have port=7009 in one of your Connectors: As I had to restart the server, I don't know who was the owner of that PID. Right now we do have 2 or 3 of: java9495 tomcat 201u IPv61866645 0t0 TCP localhost:afs3-rmtsys-localhost:55116 (ESTABLISHED) I'm willing to bet that the owner of port 55116 is httpd. If I run ulimit -n with the tomcat user I get: 1024. But I did set the max open files at the startscript for each tomcat (init.d) with: ulimit -Hn 4096 ulimit -Sn 4096 That might not work unless the script is being run by root. There might also be a global file handle limit as well. The real question is why there are so many file handles being used. Do you really need 4 connectors in each Tomcat? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlD3IK0ACgkQ9CaO5/Lv0PCUbQCfeVRFhe9h0XB69wJoPnXdTpSt sAcAoISfEaQdArlFhkhxJHGJArOEZlmL =xwRe -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AW: afs3-rmtsys: connections keept open
Hey, some additional Information: I just checked /var/logs/apache2/error.logs. And found following errors: [Wed Jan 16 15:14:46 2013] [error] server is within MinSpareThreads of MaxClients, consider raising the MaxClients setting [Wed Jan 16 15:14:56 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting Yesterday my problem occurred about the same time. I'm checking every five minutes how many open files there are: count open files started: 01-16-2013_15:10: Count: 775 count open files started: 01-16-2013_15:15: Count: 1092 But maybe the afs3 connection causing the Apache error? thanks Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: David Kumar [mailto:dku...@telestar.de] Gesendet: Donnerstag, 17. Januar 2013 07:33 An: Tomcat Users List Betreff: AW: AW: afs3-rmtsys: connections keept open Hey, I changed the global filehandle as well. And executing ulimit with the tomcat starting Thread is working well. I yesterday I hab around 1600 connection before I rebooted both of them. About Server config. I did a mistake. some of the connector are just comments. I didn't see that yesterday when I was editing the files within a shell. So here are the working connectors: tomcat1: Connector port=7080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 / Connector port=7009 protocol=AJP/1.3 redirectPort=8443 / tomcat2: Connector port=9080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=9443 / Connector port=9009 protocol=AJP/1.3 redirectPort=9443 / thanks.. Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: Christopher Schultz [mailto:ch...@christopherschultz.net] Gesendet: Mittwoch, 16. Januar 2013 22:51 An: Tomcat Users List Betreff: Re: AW: afs3-rmtsys: connections keept open -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 David, On 1/16/13 3:41 PM, David Kumar wrote: I forgot to mention that we have a Apache being a loadbalancer in front of these two tomcats. The Apache uses port 8080. So, Apache httpd listens on port 8080? Why not port 80? Here are the requested connectors tomcat1: Connector port=7080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 / Connector executor=tomcatThreadPool port=7080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 / Is that a double-paste error? You have two connectors both with port=7080. Connector port=8443 protocol=HTTP/1.1 SSLEnabled=true maxThreads=150 scheme=https secure=true clientAuth=false sslProtocol=TLS / Connector port=7009 protocol=AJP/1.3 redirectPort=8443 / tomcat2: Connector port=9080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=9443 / Connector executor=tomcatThreadPool port=8080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 / Connector port=8443 protocol=HTTP/1.1 SSLEnabled=true maxThreads=150 scheme=https secure=true clientAuth=false sslProtocol=TLS / Connector port=9009 protocol=AJP/1.3 redirectPort=9443 / Weird: you have httpd-lb out front plus 4 connectors in each Tomcat? Yeah I'm sure its port 7090, as you can see above.. :-) 7009 is used by AJP!? Yes, you have port=7009 in one of your Connectors: As I had to restart the server, I don't know who was the owner of that PID. Right now we do have 2 or 3 of: java9495 tomcat 201u IPv61866645 0t0 TCP localhost:afs3-rmtsys-localhost:55116 (ESTABLISHED) I'm willing to bet that the owner of port 55116 is httpd. If I run ulimit -n with the tomcat user I get: 1024. But I did set the max open files at the startscript for each tomcat (init.d) with: ulimit -Hn 4096 ulimit -Sn 4096 That might not work unless the script is being run by root. There might also be a global file handle limit as well. The real question is why there are so many file handles being used. Do you really need 4 connectors in each Tomcat? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlD3IK0ACgkQ9CaO5/Lv0PCUbQCfeVRFhe9h0XB69wJoPnXdTpSt sAcAoISfEaQdArlFhkhxJHGJArOEZlmL =xwRe -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
