apache/tomcat communication issues (502 response)
We are running apache and tomcat on the same machine (using the http connector) and logging requests in both. Occasionally (maybe 1% of requests) we see 502 response in the apache log spread fairly evenly throughout the day. these requests don't appear in the tomcat log produced by the access valve. in the apache error log I see: proxy: error reading status line from remote server 127.0.0.1 proxy: Error reading from remote server returned by [url] The requests are a mix of methods (GET/POST/PUT) of pretty small bodies and responses, generally under 1k. We added request time (%T) in the apache logs and many of the requests have 0 response time so I don't think it's a timeout issue. Reading the tomcat connector documentation, I tried increasing the maxThreads to 100 and backlog to 1000 in the http connector since it seemed like with the default settings a burst of traffic may cause this issue. This didn't solve the problem. I have done periodic thread dumps and it looks like the number of http threads doesn't get past 80. concerning machine resources, cpu is not loaded at all (~10% utilization). i checked file descriptor usage, it's about 400 and there is a per process limit of 1024. I am not sure if this should be posted to the apache list or tomcat list, but thought i would start here since to me it seems more likely that I am not configuring tomcat correctly as opposed to some problem in apache. I have searched the web quite a bit, but nothing seems to fit. The next thing I will try is to use the AJP connector. But even if it gets rid of the problem, I am worried it may reappear once traffic increases. Has anyone experienced a similar problem? other stuff: OS: CentOS apache: 2.2.3, using mod_proxy tomcat: 6.0.16 using netstat, i see a moderate number (~80) of tomcat's sockets in the CLOSE_WAIT state, not sure if this is relevant. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: tomcat not working
Caldarale, Charles R wrote: From: jigneshjsoni [mailto:jigneshjs...@gmail.com] Subject: tomcat not working what is wrong in here ? Lots of things, such as you not telling us the Tomcat version you're using, the JDK/JRE it's running on, the platform this is all installed on, the complete contents of server.xml, how you're starting Tomcat, and any messages in the Tomcat logs - for starters. http://www.catb.org/~esr/faqs/smart-questions.html Yeah, and you forgot to mention the subject of the post. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: apache/tomcat communication issues (502 response)
feedly team wrote: [...] using netstat, i see a moderate number (~80) of tomcat's sockets in the CLOSE_WAIT state, not sure if this is relevant. Approximately, because I am not sure I have this really understood yet : a TCP CLOSE_WAIT state happens when the writing side of a TCP connection has finished writing and (nicely) closes its side of the socket to indicate the fact, but the reading side of the connection does not read what is left in the buffers, so there is still some data unread in the pipeline, and the reading side never closes the socket. And now I'm stuck in my explanation, because I am not sure which side is seeing the CLOSE_WAIT... ;-) But anyway, it indicates a problem somewhere in one of the two applications, my guess being the reading one. It should do more reads to exhaust the remaining data, get an end-of-stream, then close its side of the connection, but it never does. There is apparently no timeout for that, so the OS can never get rid of that socket, which probably leaves a bunch of things hanging around and consuming resources that they shouldn't. On one of our systems, I have occasionally seen this issue grow until the point where the system seemed unable to accept new connections. Now whether that has any bearing on your particular issue, I don't know. But it sure indicates a logical problem somewhere. There is quite a bit on the subject on Google, of unequal quality. If someone knows a more rigorous explanation, please go ahead. I will still add a purely sibjective and personal note : from personal experience, this phenomenon seems to happens more frequently whith Java applications than with others, so I would guess that there might be something in the Java handling of sockets that makes it a bit harder to write correct socket-handling code. A Java expert may want to comment on that too. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
[Mildly OT] RE: apache/tomcat communication issues (502 response)
From: André Warnier [mailto:a...@ice-sa.com] a TCP CLOSE_WAIT state happens when the writing side of a TCP connection has finished writing and (nicely) closes its side of the socket to indicate the fact, Yes. but the reading side of the connection does not read what is left in the buffers, so there is still some data unread in the pipeline, No. It merely means that the reader has not yet closed the socket. You're putting more context behind that than there really is - the reader may, for example, have read all the data, but never have issued the close() call. and the reading side never closes the socket. Has not yet closed :-). http://www.cs.northwestern.edu/~agupta/cs340/project2/TCPIP_State_Transition_Diagram.pdf (itself reproduced from TCP/IP Illustrated, Volume 2) is a useful resource if you want to know what the states mean. If you're seeing CLOSE_WAITs on the Tomcat server, it means the client has sent a FIN and Tomcat's ACKed that. If you can, look on the client to see what's happening. If you see connections in FIN_WAIT_1, the client never got the ACK from Tomcat, if you see connections in FIN_WAIT_2, the client got the ACK and is sitting around waiting for Tomcat to close the connection and send a FIN. - Peter P.S. Yes, I used to teach this stuff ;-). - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Redirect from https to http
lol, any easy way of doing it? e.g. by the reverse of http-https Thanks -- View this message in context: http://www.nabble.com/Redirect-from-https-to-http-tp22938216p22946827.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
CLOSE_WAIT and what to do about it
Hi. As a follow-upon another thread originally entitled apache/tomcat communication issues (502 response), I'd like to pursue the CLOSE-WAIT subject. Sorry if this post is a bit long, I want to make sure that I do provide all the necessary information. Like the original poster, I am seeing on my systems a fair number of sockets apparently stuck for a long time in the CLOSE_WAIT state. (Sometimes several hundreds of them). They seem to predominantly concern Tomcat and other java processes, but as Alan pointed out previously and I confirm, my perspective is slanted, because we use a lot of common java programs and webapps on our servers, and the ones mostly affected talk to eachother and come from the same vendor. Unfortunately also, I do not have the sources of these programs/webapps available, and will not get them, and I can't do without these programs. It has been previously established that a socket in a long-time-lingering CLOSE-WAIT status, is due to one or the other side of a TCP connection not properly closing its side of the connection when it is done with it. I also surmise (without having a definite proof of this), that this is essentially bad, as it ties up some resources that could be otherwise freed. I have also been told or discovered that, our servers being Linux Debian servers, programs such as ps, netstat and lsof can help in determining precisely how many such lingering sockets there are, and who the culprit processes are (to some extent). In our case, we know which are the programs involved, because we know which ones open a listening socket and on what fixed port, and we also know which are the other processes talking to them. But, as mentioned previously, we do not have the source of these programs and will not get them, but cannot practically do without them for now. But we do have full root control of the Linux servers where these programs are running. So my question is : considering the situation above, is there something I can do locally to free these lingering CLOSE_WAIT sockets, and under which conditions ? (I must admit that I am a bit lost among the myriad options of lsof) For example, suppose I start with a netstat -pan command and I see the display below (sorry for the line-wrapping). I see a number of sockets in the CLOSE_WAIT state, and for those I have a process-id, which I can associate to a particular process. For example, I see this line : tcp6 12 0 :::127.0.0.1:41764 :::127.0.0.1:11002 CLOSE_WAIT 29649/java which tells me that there is a local process 29649/java, whith a local socket port 41674 in the CLOSE_WAIT state, related to another socket #11002 on the same host. On the other hand, I see this line : tcp0 0 127.0.0.1:11002 127.0.0.1:41764 FIN_WAIT2 - which shows a local socket on port 11002, related to this other local socket port #41764, with no process-id/program displayed. What does that tell me ? I also know that the process-id 29649 corresponds to a local java process, of the daemon variety, multi-threaded. That program talks to another known server program, written in C, of which instances are started on an ad-hoc base by inetd, and which listens on port 11002 (in fact it is inetd who does, and it passes this socket on to the process it forks, I understand that). (The link with Tomcat is that I also see frequently the same situation, where the process owning the CLOSE_WAIT socket is Tomcat, more specifically one webapp running inside it. It's just that in this particular snapshot it isn't.) What it looks like to me in this case, is that at some point one of the threads of process # 29649 opened a client socket #41674 to the local inetd port #11002; that inetd then started the underlying server process (the C program); that the underlying C program then at some point exited; but that process #41674 never closes one of the sides of its connection with port #11002. Can I somehow detect this condition, and force the offending thread of process #29649 to close that socket (or just force this thread to exit) ? I realise this may be a complex question, and that the answers may be different if it is a Tomcat webapp than a stand-alone process. I would be content to just have answers for the webapp case. Full display of netstat -pan | grep WAIT : Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp0 0 127.0.0.1:11002 127.0.0.1:41763 TIME_WAIT - tcp0 0 127.0.0.1:11002 127.0.0.1:41764 FIN_WAIT2 - tcp0 0 127.0.0.1:11002 127.0.0.1:41738 TIME_WAIT - tcp0 0 127.0.0.1:11002 127.0.0.1:41739 FIN_WAIT2 - tcp0 0 127.0.0.1:11002 127.0.0.1:41741 TIME_WAIT - tcp0 0 127.0.0.1:11002 127.0.0.1:41735 TIME_WAIT - tcp0 0 127.0.0.1:11002 127.0.0.1:41755 TIME_WAIT - tcp
Best Practise Apache + Tomcat + Multiple Hosts
Hello, Is there are a small how-to to configure a hosting server with multiple Tomcat Contexts? e.g.: http://www.test1.de - Tomcat Context /test1 http://www.test2.de - Tomcat Context /test2 . Greetings Alexander smime.p7s Description: S/MIME cryptographic signature
Re: apache/tomcat communication issues (502 response)
Hi Andre André Warnier wrote: feedly team wrote: [...] using netstat, i see a moderate number (~80) of tomcat's sockets in the CLOSE_WAIT state, not sure if this is relevant. Approximately, because I am not sure I have this really understood yet : a TCP CLOSE_WAIT state happens when the writing side of a TCP connection has finished writing and (nicely) closes its side of the socket to indicate the fact, but the reading side of the connection does not read what is left in the buffers, so there is still some data unread in the pipeline, and the reading side never closes the socket. And now I'm stuck in my explanation, because I am not sure which side is seeing the CLOSE_WAIT... ;-) I think that you are indicating one condition in which you can see a CLOSE_WAIT but there are many others. I also think that the condition you indicate is appropriate when the CLOSE_WAIT is observed at the receiving end of a communication, but is possible for a socket to be in this state when it has sent data as well, but of course there will be no outstanding data to send. More generally CLOSE_WAIT is the state in which a socket is left AFTER the other end says its finished and BEFORE the application which is using the socket actually closes the socket. The WAIT refers to the operating system waiting for the application to finish using the socket. I think a socket can be in a CLOSE_WAIT state without there being any further data to read or write - literally just waiting for the calling application to close it. Having written socket handling code for both Java and C++ on a variety of platforms I don't think there is any particular reason why Java should be better or worse (in fact, code which uses sockets in Java is generally pretty easy.) I suspect that your observations may be affected by local conditions eg one application is badly written but represents a lot of your network activity, so its behavior is predominant in conditioning your thinking. Or not! :) regards Alan Chaney But anyway, it indicates a problem somewhere in one of the two applications, my guess being the reading one. It should do more reads to exhaust the remaining data, get an end-of-stream, then close its side of the connection, but it never does. There is apparently no timeout for that, so the OS can never get rid of that socket, which probably leaves a bunch of things hanging around and consuming resources that they shouldn't. On one of our systems, I have occasionally seen this issue grow until the point where the system seemed unable to accept new connections. Now whether that has any bearing on your particular issue, I don't know. But it sure indicates a logical problem somewhere. There is quite a bit on the subject on Google, of unequal quality. If someone knows a more rigorous explanation, please go ahead. I will still add a purely sibjective and personal note : from personal experience, this phenomenon seems to happens more frequently whith Java applications than with others, so I would guess that there might be something in the Java handling of sockets that makes it a bit harder to write correct socket-handling code. A Java expert may want to comment on that too. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org !DSPAM:49dc6634305142136417547! - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Best Practise Apache + Tomcat + Multiple Hosts
Alexander Diedler wrote: Hello, Is there are a small how-to to configure a hosting server with multiple Tomcat Contexts? e.g.: http://www.test1.de - Tomcat Context /test1 http://www.test2.de - Tomcat Context /test2 You could do worse than searching the users-list archives, here : http://marc.info/?l=tomcat-user Other than that, I think that just creating adding different Host .. tags is probably what you are looking for, and in each making the corresponding webapp into the ROOT webapp. Look this up here : http://tomcat.apache.org/tomcat-6.0-doc/config/host.html and here : http://wiki.apache.org/tomcat/HowTo look for How do I make my web application be the Tomcat default application ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Best Practise Apache + Tomcat + Multiple Hosts
André Warnier wrote: Alexander Diedler wrote: Hello, Is there are a small how-to to configure a hosting server with multiple Tomcat Contexts? e.g.: http://www.test1.de - Tomcat Context /test1 http://www.test2.de - Tomcat Context /test2 You could do worse than searching the users-list archives, here : http://marc.info/?l=tomcat-user Other than that, I think that just creating adding different Host .. tags is probably what you are looking for, and in each making the corresponding webapp into the ROOT webapp. Look this up here : http://tomcat.apache.org/tomcat-6.0-doc/config/host.html and here : http://wiki.apache.org/tomcat/HowTo look for How do I make my web application be the Tomcat default application ? Don't forget http://tomcat.apache.org/tomcat-6.0-doc/virtual-hosting-howto.html Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: CLOSE_WAIT and what to do about it
From: André Warnier [mailto:a...@ice-sa.com] It has been previously established that a socket in a long-time-lingering CLOSE-WAIT status, is due to one or the other side of a TCP connection not properly closing its side of the connection when it is done with it. I also surmise (without having a definite proof of this), that this is essentially bad, as it ties up some resources that could be otherwise freed. At the very least it'll tie up a kernel data structure for the socket itself. I don't know modern Linux kernels well enough to know how buffers are allocated, but I suspect you won't be wasting much memory on buffers as they'll be allocated on-demand. You're probably talking tens to low hundreds of bytes for each one of these. You will also be consuming resources in whichever program is not closing the sockets correctly. So my question is : considering the situation above, is there something I can do locally to free these lingering CLOSE_WAIT sockets, and under which conditions ? For example, I see this line : tcp6 12 0 :::127.0.0.1:41764 :::127.0.0.1:11002 CLOSE_WAIT 29649/java which tells me that there is a local process 29649/java, whith a local socket port 41674 in the CLOSE_WAIT state, related to another socket #11002 on the same host. On the other hand, I see this line : tcp0 0 127.0.0.1:11002 127.0.0.1:41764 FIN_WAIT2 - which shows a local socket on port 11002, related to this other local socket port #41764, with no process-id/program displayed. What does that tell me ? The process that was on port 11002 closed its end of the socket and sent a FIN. Process 29649 hasn't closed its end of the socket yet. I also know that the process-id 29649 corresponds to a local java process, of the daemon variety, multi-threaded. That program talks to another known server program, written in C, of which instances are started on an ad-hoc base by inetd, and which listens on port 11002 (in fact it is inetd who does, and it passes this socket on to the process it forks, I understand that). The local Java process may have a resource leak. It appears not to have closed the socket it was using to communicate with the server. A possible reason for the lack of a PID on port 11002 is that the socket was handed across from inetd to the C daemon - not sure about this. What it looks like to me in this case, is that at some point one of the threads of process # 29649 opened a client socket #41674 to the local inetd port #11002; that inetd then started the underlying server process (the C program); that the underlying C program then at some point exited; but that process #41674 never closes one of the sides of its connection with port #11002. Agree. Can I somehow detect this condition, and force the offending thread of process #29649 to close that socket (or just force this thread to exit) ? Threads are flows of control. Threads do not reference objects other than from their stack and any thread-local storage - and there are plenty of other places that can hold onto objects! The socket may well be referenced from an object on the heap (not the stack) that's ultimately referenced by a static variable in a class, for example, in which case zapping a thread may well do nothing. You need to find out what, if anything, is holding onto the socket. If you have some way of forcing that Java process to collect garbage, you should do so. It's possible for sockets that haven't been close()d to hang around, unreferenced but not yet garbage collected. A full GC would collect any of these, finalizing them as it does and hence closing the socket. If a full GC doesn't close the socket, some other object is still referencing it. If a full GC doesn't clear the problem, you may need to go in with some memory-tracing tool and find out what's holding onto the socket. It's a long, long time since I had to do this in Java, so I have no idea of the appropriate tools - my brain's telling me Son of Strike, which is for the .Net CLR and *definitely* wrong! Does that help? Or is it clear as mud? - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Redirect from https to http
as2 wrote: lol, any easy way of doing it? e.g. by the reverse of http-https response.sendRedirect() Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
[Mildly OT] Re: apache/tomcat communication issues (502 response)
Alan Chaney wrote: Hi Andre Hi. And thanks, to Peter also. As this might start to look like a thread hijack, I'll repost with a new CLOSE_WAIT subject to continue this discussion. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: CLOSE_WAIT and what to do about it
Peter Crowther wrote: [...] Does that help? Or is it clear as mud? For no-java-expert-me, it is indeed of the hazy category. But it helps a lot, in the sense of adding a +3 in the column get back to the vendor and ask them to fix their code. ;-) Thanks. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: CLOSE_WAIT and what to do about it
Peter Crowther wrote: [...] If you have some way of forcing that Java process to collect garbage, you should do so. It's possible for sockets that haven't been close()d to hang around, unreferenced but not yet garbage collected. A full GC would collect any of these, finalizing them as it does and hence closing the socket. If a full GC doesn't close the socket, some other object is still referencing it. Hopping on that idea, and still considering the try something from the outside, without modifying the code kind of view : This process is started as a daemon, with a java command-line. Is it possible to add some arguments to that command-line to induce the JVM to do a GC more often ? (I don't think that in this case it would have a very negative impact on performance.) It currently starts without any -D switches at all to the command-line, basically : path/to/java/java -jar theapp.jar The same question for the related Tomcat webapp (which I suspect of having the same issue). But in that case I do have to be a bit more careful regarding the performance impact, although this webapp is pretty much all that is running in this Tomcat. And that Tomcat (on some of our systems) starts under jsvc, and I don't really know where to set the parameters for that one under Linux. Relatedly, does there exist any way to force a given JVM process to do a full GC interactively, but from a Linux command-line ? I have full access to these systems, but usually only in SSH console mode, and I don't know if there is any kind of graphical GUI installed or accessible on them. Basically, I'd like to see if triggering a GC reduces this number of lingering sockets. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Apache / Tomcat - Mod_jk
Hi, I have am having some trouble getting my Apache/Tomcat/Mod_jk setup to work. In short, the folder I specify in my Apache Vhost is not accessible. I have added the following to my vhost: JkMount /vhost1/* vhost1Worker I can access the domain itself, but when I try accessing the domain with /vhost1 I get a 404 error. I am running Apache/2.2.3, Tomcat 6.0.18 and using the Tomcat Connectors-1.2.28-src. I can access all my Apache Vhosts perfectly via http://domainname/ and the documentroot for my vhosts is: /web/username/www I can access my Tomcat webapp fine via http://serverip:8080/vhost1 or using the domain name, my webapp is stored in /usr/apache-tomcat-6.0.18/webapps/vhost1 Any help with this will be much appreciated as I have searched and searched but dont seem to find a working solution. Please advise if more info is required. Thank you, Leo.
Re: Apache / Tomcat - Mod_jk
Leonard wrote: Hi, I have am having some trouble getting my Apache/Tomcat/Mod_jk setup to work. In short, the folder I specify in my Apache Vhost is not accessible. I have added the following to my vhost: JkMount /vhost1/* vhost1Worker I have not read the rest, but just add another line JkMount /vhost1 vhost1Worker and see if that solves it. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: CLOSE_WAIT and what to do about it
From: André Warnier [mailto:a...@ice-sa.com] This process is started as a daemon, with a java command-line. Is it possible to add some arguments to that command-line to induce the JVM to do a GC more often ? http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html - I don't think so, although the RMI option under Explicit Garbage Collection might work. The same question for the related Tomcat webapp (which I suspect of having the same issue). But in that case I do have to be a bit more careful regarding the performance impact, although this webapp is pretty much all that is running in this Tomcat. That one's easy. Add another webapp with one page. When the page is requested, call System.GC(). Job done! Relatedly, does there exist any way to force a given JVM process to do a full GC interactively, but from a Linux command-line ? I'm not aware of one, but I'm not an expert. I await the experts' comments with interest! - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache / Tomcat - Mod_jk
In JkMount /vhost1/* vhost1Worker vhost1 is the application context . gianc64 Leonard wrote: Hi, I have am having some trouble getting my Apache/Tomcat/Mod_jk setup to work. In short, the folder I specify in my Apache Vhost is not accessible. I have added the following to my vhost: JkMount /vhost1/* vhost1Worker I can access the domain itself, but when I try accessing the domain with /vhost1 I get a 404 error. I am running Apache/2.2.3, Tomcat 6.0.18 and using the Tomcat Connectors-1.2.28-src. I can access all my Apache Vhosts perfectly via http://domainname/ and the documentroot for my vhosts is: /web/username/www I can access my Tomcat webapp fine via http://serverip:8080/vhost1 or using the domain name, my webapp is stored in /usr/apache-tomcat-6.0.18/webapps/vhost1 Any help with this will be much appreciated as I have searched and searched but dont seem to find a working solution. Please advise if more info is required. Thank you, Leo. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache / Tomcat - Mod_jk
Hi André That's got to be the easiest fix ever. Thanks, that worked! André Warnier wrote: Leonard wrote: Hi, I have am having some trouble getting my Apache/Tomcat/Mod_jk setup to work. In short, the folder I specify in my Apache Vhost is not accessible. I have added the following to my vhost: JkMount /vhost1/* vhost1Worker I have not read the rest, but just add another line JkMount /vhost1 vhost1Worker and see if that solves it. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
tomcat process memory
Hi guys! I'm getting a strange behaviour of Tomcat process. I'm using a memory profiler and it shows the following info: heap allocated memory: 278MB non-heap allocated memory : 151MB However, TOP shows that memory used by tomcat process is more or less 1GB. WhY this happens? Is it possible to understant what is using the remaining MB? Thanks a lot
Re: tomcat process memory
Andrew Hole wrote: Hi guys! I'm getting a strange behaviour of Tomcat process. I'm using a memory profiler and it shows the following info: heap allocated memory: 278MB non-heap allocated memory : 151MB However, TOP shows that memory used by tomcat process is more or less 1GB. What column of top is telling you that, on which platform and with which Tomcat ? WhY this happens? Is it possible to understant what is using the remaining MB? Probably difficult at the best of time without a lot more information, but start with the above. By the way, a good document of which I just heard of today is here : http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: tomcat process memory
From: Andrew Hole [mailto:andremailingl...@gmail.com] Subject: tomcat process memory WhY this happens? Is it possible to understant what is using the remaining MB? That 1 GB is the virtual space allocated to the process, so it includes not only the Java heap, but also code, the C heap, OS structures, library structures, file buffers, and myriad other bits and pieces needed to actually run a program. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: tomcat process memory
Linux. RES column. What is the difference between heap and non-heap and where I can configure the limit to non-heap? Thanks On Wed, Apr 8, 2009 at 2:58 PM, André Warnier a...@ice-sa.com wrote: Andrew Hole wrote: Hi guys! I'm getting a strange behaviour of Tomcat process. I'm using a memory profiler and it shows the following info: heap allocated memory: 278MB non-heap allocated memory : 151MB However, TOP shows that memory used by tomcat process is more or less 1GB. What column of top is telling you that, on which platform and with which Tomcat ? WhY this happens? Is it possible to understant what is using the remaining MB? Probably difficult at the best of time without a lot more information, but start with the above. By the way, a good document of which I just heard of today is here : http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: CLOSE_WAIT and what to do about it
From: André Warnier [mailto:a...@ice-sa.com] Subject: Re: CLOSE_WAIT and what to do about it Relatedly, does there exist any way to force a given JVM process to do a full GC interactively, but from a Linux command-line ? I haven't found one yet, but there are numerous command-line monitoring utilities included with the JDK that display all sorts of GC information, using the same connection mechanism as JConsole. Since JConsole can force a GC in a JVM its monitoring, doing it from the command line is feasible. Might have to do a little coding... - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: tomcat process memory
From: Andrew Hole [mailto:andremailingl...@gmail.com] Subject: Re: tomcat process memory What is the difference between heap and non-heap and where I can configure the limit to non-heap? The terminology depends on the profiling tool you're using - one of the many pertinent things you didn't bother to tell us. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Apache in front of Tomcat or APR?
Hi, What do I use best to set up a Shibboleth Identity provider: Apache in front of Tomcat, Tomcat with APR or just Tomcat? And why? I know Apache and APR use the OpenSSL libraries, which some say are safer. Are there any more advantages I should know of? Thanks in Advance! Bart Ophelders
Re: tomcat process memory
I'm using Your Kit Java Profiler: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 32121 tomcat25 0 1649m 962m 15m S 76 24.3 23:00.14 java At this moment I've 501 heap allocated memory and 146 non-heap. The total memory used by the process is increasing and has the value of 962MB. On Wed, Apr 8, 2009 at 3:13 PM, Caldarale, Charles R chuck.caldar...@unisys.com wrote: From: Andrew Hole [mailto:andremailingl...@gmail.com] Subject: Re: tomcat process memory What is the difference between heap and non-heap and where I can configure the limit to non-heap? The terminology depends on the profiling tool you're using - one of the many pertinent things you didn't bother to tell us. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Apache in front of Tomcat or APR?
From: Bart Ophelders [mailto:bartopheld...@hotmail.com] Subject: Apache in front of Tomcat or APR? What do I use best to set up a Shibboleth Identity provider: Apache in front of Tomcat, Tomcat with APR or just Tomcat? (I'll assume by Apache you mean httpd, since both Tomcat and httpd are Apache products.) I'm always of the opinion that one should use the minimum necessary components to get the job done. So, unless you have some compelling need for httpd (e.g., running PHP), I would not include it in the configuration. If you expect to have a lot of SSL traffic, then using the APR option of Tomcat would help there. If the amount of SSL traffic is limited, I wouldn't bother with it; if the traffic increases, you can always add it later. I know Apache and APR use the OpenSSL libraries, which some say are safer. Some say the world is flat (sorry, Mr Pratchett). Got any data? - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Migration from Tomcat 4.1 to 6.0 Urgent
Chris/Chuck, You are probably right about httpd. Please see the attached images for details. The reason why I thought I had a 2.2 application is because whenever I shutdown the 2.2 service, the main web site running on port 80 would not launch. Please excuse my naiveness about our configuration. I just recently inherited this system and was not provide any documentation.. so I am trying to figure it all out. Anyway, I only need to keep the tomcat 6.0 version of the tomcat. Can you please explain how I should configure httpd to work with the tomcat 6.0 version of the application, not 4.1. Chris provided me with the following instructions. It seems like I have to use mod_proxy_http or mod_proxy_ajp. Can you please provide me step by step instruction on how I should change my existing configuration to work with the newer application? Thank you so so much. God Bless... Rashid, On 4/7/2009 5:08 PM, Rashid Malik wrote: Hi, I changed port 8080 to 80 and I get page not found error. Please note that I have another website that is running on an older version of tomcat. Aah, you didn't mention that. Only one process may bind to a particular interface/port combination. In your case, port 80 is already taken (by Tomcat 4.1). If you want both Tomcat 4.1 ans 6.0 to appear to be using port 80, you will need to put another piece of software out in front in order to play traffic cop and properly route the requests to the appropriate server (app on TC4.1 versus app on TC6.0). Apache httpd is a popular choice for accomplishing this. You will have to: 1. Change Tomcat 4.1's configuration so that the HTTP port is something /other/ than port 80 (8080 is a popular choice) 2. Change Tomcat 6.0's configuration so that the HTTP port is something /other/ than port 80 and 8080 (8081?) 3. Install Apache httpd and have it listen on port 80 4. Configure httpd to send the appropriate requests to the appropriate back-end server To accomplish #4, you will need to use either mod_proxy_http, mod_proxy_ajp (both built-into Apache httpd 2.2.x) or mod_jk (which is a separate download). There is a lot of documentation for using mod_jk here: http://tomcat.apache.org/connectors-doc/ If you use mod_proxy_http, you'll need to do: ProxyPass /appA http://localhost:8080/appA ProxyPass /appB http://localhost:8081/appB That should be enough to get you started. - -chris Christopher Schultz ch...@christopherschultz.net 4/7/2009 6:03:42 pm -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Rashid, On 4/7/2009 5:45 PM, Rashid Malik wrote: What I mean here is that I have three 3 installations of tomcat not 2. How many total web applications do you have? Do any of them have any requirements for which Tomcat version on which they are running? It sounds like the answer is I have 3 webapps and at least one of them requires Tomcat 6.0. If there are no other constraints, then just move all your webapps to Tomcat 6.0, uninstall Tomcat 4.1, and change Tomcat 6.0's Connector to port=80 as I said previously. There is tomcat 2.2 application (using port 80) As Chuck suggests, that is probably Apache httpd, which might actually make things easier. If this is the case, you have not correctly configured httpd to support your new application. Go into httpd's configuration and find out where your other application is configured. Read and /understand/ the configuration before you start changing things. Reading that configuration plus the config reference I already posted should clear a lot of things up. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAknbzb4ACgkQ9CaO5/Lv0PCgngCeJLFU7QsoPwisTu7EVwvLHJ+g 3aEAoKgI0amw4JfW1BU0+5fbmG1rRO4G =1bQY -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: tomcat process memory
From: Andrew Hole [mailto:andremailingl...@gmail.com] Subject: Re: tomcat process memory At this moment I've 501 heap allocated memory and 146 non-heap. If you're using a HotSpot JVM (again, you didn't tell us) with YourKit, heap memory is the space consumed by allocations in the young and old generations, and non-heap is JIT-generated code and permanent generation. The exact components of the non-heap space also vary depending on whether class sharing is on or not (on by default in client mode, not available in server mode). - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Installing Tomcat
I'm trying to get Tomcat 6 running on a RedHat box. I don't want to build deamon with jsvc as the docs say I should do, at least not yet. Is this the extent of the official documentation for setting up Tomcat on Linux? I installed JDK 6 and Tomcat 6, defined JAVA_HOME and TOMCAT_HOME and setup have a Host and Context in server.xml for www.mysite.com. I invoke catalina.sh start and get a process running, but I can't reach mysite. I should say that apache httpd 2.0 is already running on this machine and my www.mysite.com is reachable there, so I figured www.mysite.com:8080 would take me to tomcat. Is that a correct assumption? IE. Should I be able to by httpd by tacking :8080 onto the url? If I kill apache httpd (I have no intention of using it going forward), will updating server.xml with port 80 rather than port 8080 automagically get everything to work, or is there other stuff I need to do before Tomcat will run as a standalone webserver? thanks,
RE: Installing Tomcat
From: Jonathan Mast [mailto:jhmast.develo...@gmail.com] I'm trying to get Tomcat 6 running on a RedHat box. I don't want to build deamon with jsvc as the docs say I should do, at least not yet. Is this the extent of the official documentation for setting up Tomcat on Linux? You don't say what this is, so we can't help you. I installed JDK 6 and Tomcat 6, defined JAVA_HOME and TOMCAT_HOME and setup have a Host and Context in server.xml for www.mysite.com. I invoke catalina.sh start and get a process running, but I can't reach mysite. *Exactly* how are you trying to reach it? From what computer, and what are you trying as the address? If it's www.mysite.com, have you changed your hosts file or DNS so that it points to your computer rather than off into the Internet? What happens if you browse to http://localhost:8080? I should say that apache httpd 2.0 is already running on this machine and my www.mysite.com is reachable there, so I figured www.mysite.com:8080 would take me to tomcat. Is that a correct assumption? IE. Should I be able to by httpd by tacking :8080 onto the url? If you still have a default server.xml, this should work. However, you've changed your server.xml. You haven't bothered to post your new one, so we can't tell whether this would work or not. If I kill apache httpd (I have no intention of using it going forward), will updating server.xml with port 80 rather than port 8080 automagically get everything to work, or is there other stuff I need to do before Tomcat will run as a standalone webserver? With the default server.xml, simply changing 8080 to 80 (and ensuring the process at least starts as root, which is why jsvc is so important) will be all you need to do. With your server.xml? Nobody on the list will have any idea. Give us some more information, please - we're not psychic! - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Installing Tomcat
Is this: http://tomcat.apache.org/tomcat-6.0-doc/setup.html the extent of the setup docs? I'm reaching it by editing the hosts file on my windoze machine to resolve all www.mysite.com requests back to an internal ip which is the machine in question. Tomcat is running as root or at least thats what ps auxf indicates. I rolled back to the original server.xml, restarted Tomcat and I'm still getting a Failed to Establish Connection Error in my browser for www.mysite.com:8080. But www.mysite.com is working. Furthermore, I can tell you that netstat -a indicated no listener for port 8080 and that that Catalina logs look ok. thanks, On Wed, Apr 8, 2009 at 12:24 PM, Peter Crowther peter.crowt...@melandra.com wrote: From: Jonathan Mast [mailto:jhmast.develo...@gmail.com] I'm trying to get Tomcat 6 running on a RedHat box. I don't want to build deamon with jsvc as the docs say I should do, at least not yet. Is this the extent of the official documentation for setting up Tomcat on Linux? You don't say what this is, so we can't help you. I installed JDK 6 and Tomcat 6, defined JAVA_HOME and TOMCAT_HOME and setup have a Host and Context in server.xml for www.mysite.com. I invoke catalina.sh start and get a process running, but I can't reach mysite. *Exactly* how are you trying to reach it? From what computer, and what are you trying as the address? If it's www.mysite.com, have you changed your hosts file or DNS so that it points to your computer rather than off into the Internet? What happens if you browse to http://localhost:8080? I should say that apache httpd 2.0 is already running on this machine and my www.mysite.com is reachable there, so I figured www.mysite.com:8080 would take me to tomcat. Is that a correct assumption? IE. Should I be able to by httpd by tacking :8080 onto the url? If you still have a default server.xml, this should work. However, you've changed your server.xml. You haven't bothered to post your new one, so we can't tell whether this would work or not. If I kill apache httpd (I have no intention of using it going forward), will updating server.xml with port 80 rather than port 8080 automagically get everything to work, or is there other stuff I need to do before Tomcat will run as a standalone webserver? With the default server.xml, simply changing 8080 to 80 (and ensuring the process at least starts as root, which is why jsvc is so important) will be all you need to do. With your server.xml? Nobody on the list will have any idea. Give us some more information, please - we're not psychic! - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Migration from Tomcat 4.1 to 6.0
http://httpd.apache.org/docs/2.2/ Read Compiling and Installing section You will need to undertand the modules and directives of several modules. You will find in the Apache site instructions for mod_proxy, but if you decide mod_jk then here are the instructions http://tomcat.apache.org/connectors-doc/webserver_howto/apache.html http://www.linuxjournal.com/article/8561 Can you please provide me step by step instruction... Don't expect the people in the list to do your work. Research, read, experiment! Then come back with specific questions. -Jorge -Original Message- From: Rashid Malik [mailto:rma...@dhmh.state.md.us] Sent: Wednesday, April 08, 2009 10:48 AM To: Tomcat Users List Subject: Re: Migration from Tomcat 4.1 to 6.0 Urgent Chris/Chuck, You are probably right about httpd. Please see the attached images for details. The reason why I thought I had a 2.2 application is because whenever I shutdown the 2.2 service, the main web site running on port 80 would not launch. Please excuse my naiveness about our configuration. I just recently inherited this system and was not provide any documentation.. so I am trying to figure it all out. Anyway, I only need to keep the tomcat 6.0 version of the tomcat. Can you please explain how I should configure httpd to work with the tomcat 6.0 version of the application, not 4.1. Chris provided me with the following instructions. It seems like I have to use mod_proxy_http or mod_proxy_ajp. Can you please provide me step by step instruction on how I should change my existing configuration to work with the newer application? Thank you so so much. God Bless... Rashid, On 4/7/2009 5:08 PM, Rashid Malik wrote: Hi, I changed port 8080 to 80 and I get page not found error. Please note that I have another website that is running on an older version of tomcat. Aah, you didn't mention that. Only one process may bind to a particular interface/port combination. In your case, port 80 is already taken (by Tomcat 4.1). If you want both Tomcat 4.1 ans 6.0 to appear to be using port 80, you will need to put another piece of software out in front in order to play traffic cop and properly route the requests to the appropriate server (app on TC4.1 versus app on TC6.0). Apache httpd is a popular choice for accomplishing this. You will have to: 1. Change Tomcat 4.1's configuration so that the HTTP port is something /other/ than port 80 (8080 is a popular choice) 2. Change Tomcat 6.0's configuration so that the HTTP port is something /other/ than port 80 and 8080 (8081?) 3. Install Apache httpd and have it listen on port 80 4. Configure httpd to send the appropriate requests to the appropriate back-end server To accomplish #4, you will need to use either mod_proxy_http, mod_proxy_ajp (both built-into Apache httpd 2.2.x) or mod_jk (which is a separate download). There is a lot of documentation for using mod_jk here: http://tomcat.apache.org/connectors-doc/ If you use mod_proxy_http, you'll need to do: ProxyPass /appA http://localhost:8080/appA ProxyPass /appB http://localhost:8081/appB That should be enough to get you started. - -chris Christopher Schultz ch...@christopherschultz.net 4/7/2009 6:03:42 pm -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Rashid, On 4/7/2009 5:45 PM, Rashid Malik wrote: What I mean here is that I have three 3 installations of tomcat not 2. How many total web applications do you have? Do any of them have any requirements for which Tomcat version on which they are running? It sounds like the answer is I have 3 webapps and at least one of them requires Tomcat 6.0. If there are no other constraints, then just move all your webapps to Tomcat 6.0, uninstall Tomcat 4.1, and change Tomcat 6.0's Connector to port=80 as I said previously. There is tomcat 2.2 application (using port 80) As Chuck suggests, that is probably Apache httpd, which might actually make things easier. If this is the case, you have not correctly configured httpd to support your new application. Go into httpd's configuration and find out where your other application is configured. Read and /understand/ the configuration before you start changing things. Reading that configuration plus the config reference I already posted should clear a lot of things up. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAknbzb4ACgkQ9CaO5/Lv0PCgngCeJLFU7QsoPwisTu7EVwvLHJ+g 3aEAoKgI0amw4JfW1BU0+5fbmG1rRO4G =1bQY -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: tomcat, mod_jk, and apache
Another possibility is having a single source for your host setup, with some kind of script/program to generate both the httpd and Tomcat config files from that. (That could turn out to be a marketable product.) This is our backup plan, and probably what we'll end up doing. Thanks folks! Warm regards, Jordan Michaels Vivio Technologies http://www.viviotech.net/ Open BlueDragon Steering Committee Adobe Solution Provider Caldarale, Charles R wrote: From: Jordan Michaels [mailto:jor...@viviotech.net] Subject: tomcat, mod_jk, and apache I'm using mod_jk with apache and I was wondering if it would be possible to configure mod_jk in a way that I wouldn't have to specify the Host entries in Tomcat's server.xml file in addition to apache. (I'll assume by apache you really mean httpd, since both Tomcat and httpd are Apache products.) Let's turn the question around: do you really need httpd in front of Tomcat? There are certainly valid reasons for doing so (e.g., running PHP, poor man's load balancing, etc.), but is your environment one of those? If you do need to use httpd, can you get away with a single Host in Tomcat, and simply differentiate the varied requests via webapp name? (Using mod_rewrite might help here.) Another possibility is having a single source for your host setup, with some kind of script/program to generate both the httpd and Tomcat config files from that. (That could turn out to be a marketable product.) - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: tomcat, mod_jk, and apache
Jordan Michaels wrote: Another possibility is having a single source for your host setup, with some kind of script/program to generate both the httpd and Tomcat config files from that. (That could turn out to be a marketable product.) The organisation of the Tomcat documentation is sometimes a bit confusing. Here is an example : I do not know if this relates to what you are looking for, but it may be worth looking at anyway : http://tomcat.apache.org/connectors-doc/webserver_howto/apache.html and have a look at the section Using Tomcat auto-configure Don't ask me how I got to the above Tomcat documentation page, nor if this represents the state-of-the-art. I think it was from the top documentation page, then the Reference section on the left, then the JK 1.2 documentation, then some Apache howto. One would want to hide that page, he could not do much better. Maybe there is a reason... - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat 6 Cluster with Apache 2.2.9 issues
Hi All, I have been working with this problem for 3 weeks now and can't seem to figure it out. I have a cluster of 3 Tomcat servers using mod_jk/1.2.27 and Apache 2.2.9 and Tomcat 6.0.18 I am seeing these errors in my mod_jk.log. [Tue Apr 07 17:57:11.829 2009] [29020:3075438336] [warn] jk_map_validate_property::jk_map.c (410): The attribute 'worker.loadbalancer.balanced_workers' is deprecated - please check the documentation for the correct replacement. [Tue Apr 07 17:57:11.830 2009] [29020:3075438336] [info] init_jk::mod_jk.c (3018): mod_jk/1.2.27 initialized [Tue Apr 07 17:57:11.830 2009] [29020:3075438336] [error] uri_worker_map_ext::jk_uri_worker_map.c (505): Could not find worker with name 'ajp13' in uri map post processing. [Tue Apr 07 17:57:11.830 2009] [29020:3075438336] [error] uri_worker_map_ext::jk_uri_worker_map.c (505): Could not find worker with name 'ajp13' in uri map post processing. [Tue Apr 07 17:57:11.912 2009] [29021:3075438336] [warn] jk_map_validate_property::jk_map.c (410): The attribute 'worker.loadbalancer.balanced_workers' is deprecated - please check the documentation for the correct replacement. [Tue Apr 07 17:57:11.912 2009] [29021:3075438336] [info] init_jk::mod_jk.c (3018): mod_jk/1.2.27 initialized [Tue Apr 07 17:57:11.912 2009] [29021:3075438336] [error] uri_worker_map_ext::jk_uri_worker_map.c (505): Could not find worker with name 'ajp13' in uri map post processing. [Tue Apr 07 17:57:11.912 2009] [29021:3075438336] [error] uri_worker_map_ext::jk_uri_worker_map.c (505): Could not find worker with name 'ajp13' in uri map post processing. [Wed Apr 08 09:06:09.510 2009] [29022:3075438336] [warn] map_uri_to_worker_ext::jk_uri_worker_map.c (961): Uri HTTP/1.1 is invalid. Uri must start with / [Wed Apr 08 11:04:03.917 2009] [29056:3075438336] [warn] map_uri_to_worker_ext::jk_uri_worker_map.c (961): Uri * is invalid. Uri must start with / [Wed Apr 08 11:06:36.917 2009] [29025:3075438336] [warn] map_uri_to_worker_ext::jk_uri_worker_map.c (961): Uri * is invalid. Uri must start with / [Wed Apr 08 11:06:37.917 2009] [29058:3075438336] [warn] map_uri_to_worker_ext::jk_uri_worker_map.c (961): Uri * is invalid. Uri must start with / [Wed Apr 08 11:06:40.917 2009] [29595:3075438336] [warn] map_uri_to_worker_ext::jk_uri_worker_map.c (961): Uri * is invalid. Uri must start with / [Wed Apr 08 13:25:48.917 2009] [29045:3075438336] [warn] map_uri_to_worker_ext::jk_uri_worker_map.c (961): Uri * is invalid. Uri must start with / ~ I am having issues with my jsessionId throwing a 404 page not found error. I have Apache returning the user to login again. Then the second time the login works and no jsessionId 404. Do you think this is the problem in my logs? Is the issue something else? Thanks Phil -- View this message in context: http://www.nabble.com/Tomcat-6-Cluster-with-Apache-2.2.9-issues-tp22957994p22957994.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6 Cluster with Apache 2.2.9 issues
This is my workers.properties file too. Thanks Phil # workers.properties ps=/ # list the workers by name worker.list=tomcat1,tomcat2,tomcat3,loadbalancer # # Specifies the load balance factor when used with # a load balancing worker. # Note: # lbfactor must be 0 # Low lbfactor means less work done by the worker. # # First tomcat server # worker.tomcat1.port=18009 worker.tomcat1.host=192.168.70.101 worker.tomcat1.type=ajp13 worker.tomcat1.lbfactor=33 # # Second tomcat server # worker.tomcat2.port=19009 worker.tomcat2.host=192.168.70.102 worker.tomcat2.type=ajp13 worker.tomcat2.lbfactor=33 # # Third tomcat server # worker.tomcat3.port=20009 worker.tomcat3.host=192.168.70.103 worker.tomcat3.type=ajp13 worker.tomcat3.lbfactor=33 # --- # Load Balancer worker # # # The loadbalancer (type lb) worker performs weighted round-robin # load balancing with sticky sessions. # Note: # If a worker dies, the load balancer will check its state #once in a while. Until then all work is redirected to peer #worker. worker.loadbalancer.type=lb worker.loadbalancer.balanced_workers=tomcat1,tomcat2,tomcat3 # # END workers.properties # -- View this message in context: http://www.nabble.com/Tomcat-6-Cluster-with-Apache-2.2.9-issues-tp22957994p22958289.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat 6 Cluster with Apache 2.2.9 issues
Remove your tomcat workers from the worker.list, you only need to reference the balance worker here: worker.list=loadbalancer your load balancer worker then refers to the tomcat workers: worker.loadbalancer.balanced_workers=tomcat1,tomcat2,tomcat3 -Jorge -Original Message- From: nohacks [mailto:philipfwil...@gmail.com] Sent: Wednesday, April 08, 2009 4:11 PM To: users@tomcat.apache.org Subject: Re: Tomcat 6 Cluster with Apache 2.2.9 issues This is my workers.properties file too. Thanks Phil # workers.properties ps=/ # list the workers by name worker.list=tomcat1,tomcat2,tomcat3,loadbalancer # # Specifies the load balance factor when used with # a load balancing worker. # Note: # lbfactor must be 0 # Low lbfactor means less work done by the worker. # # First tomcat server # worker.tomcat1.port=18009 worker.tomcat1.host=192.168.70.101 worker.tomcat1.type=ajp13 worker.tomcat1.lbfactor=33 # # Second tomcat server # worker.tomcat2.port=19009 worker.tomcat2.host=192.168.70.102 worker.tomcat2.type=ajp13 worker.tomcat2.lbfactor=33 # # Third tomcat server # worker.tomcat3.port=20009 worker.tomcat3.host=192.168.70.103 worker.tomcat3.type=ajp13 worker.tomcat3.lbfactor=33 # --- # Load Balancer worker # # # The loadbalancer (type lb) worker performs weighted round-robin # load balancing with sticky sessions. # Note: # If a worker dies, the load balancer will check its state #once in a while. Until then all work is redirected to peer #worker. worker.loadbalancer.type=lb worker.loadbalancer.balanced_workers=tomcat1,tomcat2,tomcat3 # # END workers.properties # -- View this message in context: http://www.nabble.com/Tomcat-6-Cluster-with-Apache-2.2.9-issues-tp229579 94p22958289.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat 6 Cluster with Apache 2.2.9 issues
thanks for the reply. I updated my workers.properties file to worker.list=loadbalancer still get this in my mod_jk.log. [Wed Apr 08 17:21:46.840 2009] [29811:3075569408] [warn] jk_map_validate_property::jk_map.c (410): The attribute 'worker.loadbalancer.balanced_workers' is deprecated - please check the documentation for the correct replacement. [Wed Apr 08 17:21:46.841 2009] [29811:3075569408] [info] init_jk::mod_jk.c (3018): mod_jk/1.2.27 initialized [Wed Apr 08 17:21:46.841 2009] [29811:3075569408] [error] uri_worker_map_ext::jk_uri_worker_map.c (505): Could not find worker with name 'ajp13' in uri map post processing. [Wed Apr 08 17:21:46.841 2009] [29811:3075569408] [error] uri_worker_map_ext::jk_uri_worker_map.c (505): Could not find worker with name 'ajp13' in uri map post processing. [Wed Apr 08 17:21:46.924 2009] [29812:3075569408] [warn] jk_map_validate_property::jk_map.c (410): The attribute 'worker.loadbalancer.balanced_workers' is deprecated - please check the documentation for the correct replacement. [Wed Apr 08 17:21:46.924 2009] [29812:3075569408] [info] init_jk::mod_jk.c (3018): mod_jk/1.2.27 initialized [Wed Apr 08 17:21:46.924 2009] [29812:3075569408] [error] uri_worker_map_ext::jk_uri_worker_map.c (505): Could not find worker with name 'ajp13' in uri map post processing. [Wed Apr 08 17:21:46.924 2009] [29812:3075569408] [error] uri_worker_map_ext::jk_uri_worker_map.c (505): Could not find worker with name 'ajp13' in uri map post processing. ~ -- View this message in context: http://www.nabble.com/Tomcat-6-Cluster-with-Apache-2.2.9-issues-tp22957994p22959874.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6 Cluster with Apache 2.2.9 issues
On 08.04.2009 23:23, nohacks wrote: thanks for the reply. I updated my workers.properties file to worker.list=loadbalancer still get this in my mod_jk.log. [Wed Apr 08 17:21:46.840 2009] [29811:3075569408] [warn] jk_map_validate_property::jk_map.c (410): The attribute 'worker.loadbalancer.balanced_workers' is deprecated - please check the documentation for the correct replacement. Look for the replacement of balanced_workers in http://tomcat.apache.org/connectors-doc/reference/workers.html [Wed Apr 08 17:21:46.841 2009] [29811:3075569408] [error] uri_worker_map_ext::jk_uri_worker_map.c (505): Could not find worker with name 'ajp13' in uri map post processing. [Wed Apr 08 17:21:46.841 2009] [29811:3075569408] [error] uri_worker_map_ext::jk_uri_worker_map.c (505): Could not find worker with name 'ajp13' in uri map post processing. It seems you haven't successfully defined any JkMount(s) or uriworkermap.properties. If there are no such, then a builtin automatic worker named ajp13 is assumed, which is not configured in your workers.properties. Don't configure an additional worker with that name, instead try to explain JkMount or uriworkermap.properties (both are nearly equivalent, you can choose whatever you like more). They are needed to explain, which URLs Apache httpd should forward via which worker (or not forward at all). Also remove the ps line from workers.properties. It has no meaning. Furthermore look at the page http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html to learn more about timeouts you should use in the configuration. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6 Cluster with Apache 2.2.9 issues
Look for the replacement of balanced_workers in http://tomcat.apache.org/connectors-doc/reference/workers.html Also remove the ps line from workers.properties. It has no meaning. Furthermore look at the page http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html to learn more about timeouts you should use in the configuration. === Thanks great links. I will check them out now. Phil -- View this message in context: http://www.nabble.com/Tomcat-6-Cluster-with-Apache-2.2.9-issues-tp22957994p22960190.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Odd Tomcat setup - graphics not displaying
We've got two identical tomcat servers load balanced with apache as frontend. Our site has a private side where users can log in. Once you've logged in, there is a link to a forum. The forum ONLY resides on server1 at /tomcat/webapps/forum (on server2 there is NO /tomcat/webapps/forum and so any links in the code explicitly refer to the IP of server1). Our main site is at /tomcat/webapps/thr4, so obviously the forum folder is outside the parent dir. The problem happens when tomcat2 (server2) serves a session... then the forum is very unpredictable and inconsistent on whether or not it will display static graphic files. When you hit refresh, some graphics display while others go away keep hitting refresh and its a different result every time maybe every blue moon, ALL graphics will display. If a session is threaded on tomcat1 (server1), there's not a problem. I'm an apache tomcat newbie. My IT dept (of 2 people) was asked to make our web server lb'd and replicated between two servers and we are not web devs/admins so I'm unsure what exactly is causing and what to do... I don't like the fact that the webapps dir structure is not the same on both servers, but as its only a forum, it doesn't need to replicate I don't guess? (I'm not in charge of where webapp folders go and the person who is isn't a web admin either). Any suggestions? Can post config files if need be... Thanks so much! BJ
Odd Tomcat setup - graphics not displaying
We've got two identical tomcat servers load balanced with apache as frontend. Our site has a private side where users can log in. Once you've logged in, there is a link to a forum. The forum ONLY resides on server1 at /tomcat/webapps/forum (on server2 there is NO /tomcat/webapps/forum and so any links in the code explicitly refer to the IP of server1). Our main site is at /tomcat/webapps/thr4, so obviously the forum folder is outside the parent dir. The problem happens when tomcat2 (server2) serves a session... then the forum is very unpredictable and inconsistent on whether or not it will display static graphic files. When you hit refresh, some graphics display while others go away keep hitting refresh and its a different result every time maybe every blue moon, ALL graphics will display. If a session is threaded on tomcat1 (server1), there's not a problem. I'm an apache tomcat newbie. My IT dept (of 2 people) was asked to make our web server lb'd and replicated between two servers and we are not web devs/admins so I'm unsure what exactly is causing and what to do... I don't like the fact that the webapps dir structure is not the same on both servers, but as its only a forum, it doesn't need to replicate I don't guess? (I'm not in charge of where webapp folders go and the person who is isn't a web admin either). Any suggestions? Can post config files if need be... Thanks so much! BJ
Re: CLOSE_WAIT and what to do about it
Hi André, I didn't fully read all responses, so I hope i don't repeat to much (or worse contradict statements contained in other replies). On 08.04.2009 12:32, André Warnier wrote: Like the original poster, I am seeing on my systems a fair number of sockets apparently stuck for a long time in the CLOSE_WAIT state. (Sometimes several hundreds of them). They seem to predominantly concern Tomcat and other java processes, but as Alan pointed out previously and I confirm, my perspective is slanted, because we use a lot of common java programs and webapps on our servers, and the ones mostly affected talk to eachother and come from the same vendor. Unfortunately also, I do not have the sources of these programs/webapps available, and will not get them, and I can't do without these programs. It has been previously established that a socket in a long-time-lingering CLOSE-WAIT status, is due to one or the other side of a TCP connection not properly closing its side of the connection when it is done with it. CLOSE_WAIT says the other side shut down the connection. TCP connections are allowed to stay for an arbitrary time in half-closed state. In general TCP connection can be used in a duplex way. But assume one end has finished communication (sending data). Then it can already close its side of the connection. The nice TCP state diagram is contained in the fundamental book of Stevens, and can be seen e.g. at http://www.cse.iitb.ac.in/perfnet/cs456/tcp-state-diag.pdf As you can see, CLOSE_WAIT on one end always implies FIN_WAIT2 on the other end (except, when between the two ends there's yet another component, that interferes with the communication like maybe a firewall). In the special situation where both ends of the communication are on the same system, one finds each connection twice, one from the point of view of each side of the connection. It is always important to think about which end one is looking at, when interpreting the two lines. I also surmise (without having a definite proof of this), that this is essentially bad, as it ties up some resources that could be otherwise freed. I have also been told or discovered that, our servers being Linux Debian servers, programs such as ps, netstat and lsof can help in determining precisely how many such lingering sockets there are, and who the culprit processes are (to some extent). True. In our case, we know which are the programs involved, because we know which ones open a listening socket and on what fixed port, and we also know which are the other processes talking to them. But, as mentioned previously, we do not have the source of these programs and will not get them, but cannot practically do without them for now. But we do have full root control of the Linux servers where these programs are running. The details may depend on the used protocols and sometimes you can get information about timeouts you can set in the application, like idle timeouts for persistent connections. So my question is : considering the situation above, is there something I can do locally to free these lingering CLOSE_WAIT sockets, and under which conditions ? (I must admit that I am a bit lost among the myriad options of lsof) I would say no, if you can't change the application and the developper of it didn't provide any configuration options. CLOSE_WAIT from the point of view of tcp is a legitimate state without any builtin timeout. For example, suppose I start with a netstat -pan command and I see the display below (sorry for the line-wrapping). I see a number of sockets in the CLOSE_WAIT state, and for those I have a process-id, which I can associate to a particular process. For example, I see this line : tcp6 12 0 :::127.0.0.1:41764 :::127.0.0.1:11002 CLOSE_WAIT 29649/java which tells me that there is a local process 29649/java, whith a local socket port 41674 in the CLOSE_WAIT state, related to another socket #11002 on the same host. On the other hand, I see this line : tcp0 0 127.0.0.1:11002 127.0.0.1:41764 FIN_WAIT2 - which shows a local socket on port 11002, related to this other local socket port #41764, with no process-id/program displayed. What does that tell me ? My interpretation (not 100% sure): Not sure, what your OS shows in netstat after closing the local side of a connection, more precisely whether the pid is still shown, or is removed. Depending on this answer, either we have a simple one-sided shutdown, or even a process exit. In both cases the process 41764 didn't have any reason to use the established connection in the meantime, so it didn't realise, that the connection is only half-open. As soon as it tried to use it, it should/would detect that and most likely (if programmed correctly) close it. I also know that the process-id 29649 corresponds to a local java process, of the daemon variety, multi-threaded. That program talks to another known server
Re: apache/tomcat communication issues (502 response)
On 08.04.2009 08:23, feedly team wrote: We are running apache and tomcat on the same machine (using the http connector) and logging requests in both. Occasionally (maybe 1% of requests) we see 502 response in the apache log spread fairly evenly throughout the day. these requests don't appear in the tomcat log produced by the access valve. in the apache error log I see: proxy: error reading status line from remote server 127.0.0.1 proxy: Error reading from remote server returned by [url] The requests are a mix of methods (GET/POST/PUT) of pretty small bodies and responses, generally under 1k. We added request time (%T) in the apache logs and many of the requests have 0 response time so I don't think it's a timeout issue. Reading the tomcat connector documentation, I tried increasing the maxThreads to 100 and backlog to 1000 in the http connector since it seemed like with the default settings a burst of traffic may cause this issue. This didn't solve the problem. I have done periodic thread dumps and it looks like the number of http threads doesn't get past 80. concerning machine resources, cpu is not loaded at all (~10% utilization). i checked file descriptor usage, it's about 400 and there is a per process limit of 1024. I am not sure if this should be posted to the apache list or tomcat list, but thought i would start here since to me it seems more likely that I am not configuring tomcat correctly as opposed to some problem in apache. I have searched the web quite a bit, but nothing seems to fit. The next thing I will try is to use the AJP connector. But even if it gets rid of the problem, I am worried it may reappear once traffic increases. Has anyone experienced a similar problem? I guess it's the usual HTTP Keep-Alive race with mod_proxy_http. The connections between Apache and Tomcat are only reused for a certain amount of time. What happens every now and then is that httpd thinks it has a valid connection and sends the next data, but in parallel Tomcat detects the HTTP KeepAlive has timed out and closes the connection. There have been improvements in newer 2.2.x versions (latest is 2.2.11). You should especially look at the various pool size and timeout configuration attributes on the mod_proxy documentation page and also at the Keepalive configuration values for the Tomcat HTTP connector you are using. other stuff: OS: CentOS apache: 2.2.3, using mod_proxy tomcat: 6.0.16 using netstat, i see a moderate number (~80) of tomcat's sockets in the CLOSE_WAIT state, not sure if this is relevant. I assume Apache and Tomcat are on separate systems, and you notice CLOSE_WAIT when doing netstat on the Tomcat system and the connections are proxy connections? Then this means Apache has closed a connection and Tomcat would still use it. Investigate the related timeout parameters as suggested above. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: tomcat, mod_jk, and apache
On 08.04.2009 20:39, André Warnier wrote: Jordan Michaels wrote: Another possibility is having a single source for your host setup, with some kind of script/program to generate both the httpd and Tomcat config files from that. (That could turn out to be a marketable product.) The organisation of the Tomcat documentation is sometimes a bit confusing. Here is an example : I do not know if this relates to what you are looking for, but it may be worth looking at anyway : http://tomcat.apache.org/connectors-doc/webserver_howto/apache.html and have a look at the section Using Tomcat auto-configure Unfortunately this time you found one of the dark parts of mod_jk. Don't use auto-config. It mostly produces garbage. But: why do you plan to use different Hosts in Tomcat at all? Unless you need to set Host based settings for different webapps to different values or you want to deploy different webapps under the same name, you can put all webapps into the default Host. Or do you use multiple ROOT contexts? Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Installing Tomcat
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jonathan, On 4/8/2009 1:29 PM, Jonathan Mast wrote: Furthermore, I can tell you that netstat -a indicated no listener for port 8080 and that that Catalina logs look ok. Seriously. Post. Your. Server. Dot. XML. File. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkndJQ8ACgkQ9CaO5/Lv0PC0dgCgmuaMEPpIUteUHtg8z0OUXLRV Y/UAoJeoGgajogSBhVoW01Rqwh1dS+up =xuuf -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Odd Tomcat setup - graphics not displaying
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 BJ, On 4/8/2009 5:54 PM, BJ Selman wrote: We've got two identical tomcat servers load balanced with apache as frontend. [snip] When you hit refresh, some graphics display while others go away keep hitting refresh and its a different result every time maybe every blue moon, ALL graphics will display. If a session is threaded on tomcat1 (server1), there's not a problem. Are you using mod_proxy_[something] or mod_jk to connect httpd to Tomcat? If you are using mod_jk, does setting the JkStripSession directive to true fix this problem? JkStripSession will remove the ;jsessionid=... parameter that is added to certain URLs that application servers generate. Apache httpd is too stupid to realize that this is a /parameter/ and not part of the actual URL, and so it stupidly goes around the disk looking for a file called myLogo.gif;jsessionid=ABCD1234. JkStripSession removes the jsessionid parameter from URLs that will /not/ be handled by Apache. I'm not sure what the mod_proxy_[whatever] analog is. This can also be done using mod_rewrite, but we'll only get to that if you actually have this problem. Check your httpd error logs to see if you're getting requests failing because it can't find a file with the session id attached. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkndJhkACgkQ9CaO5/Lv0PDN3ACbBOG50MFuvIHIIiQpg7XCvSga bX4AoJ0wf8lNSa/AFMSSmCamBJvRLCQ8 =jGTr -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: tomcat process memory
On 08.04.2009 16:25, Andrew Hole wrote: I'm using Your Kit Java Profiler: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 32121 tomcat25 0 1649m 962m 15m S 76 24.3 23:00.14 java At this moment I've 501 heap allocated memory and 146 non-heap. The total memory used by the process is increasing and has the value of 962MB. You might also want to look at the file /proc/32121/maps or more generally /proc/MYPID/maps There you can do a little exercise with hex address difference calculations to find out how much native memory is used (roughly) for what. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
how mod_jk load balancer auto recover node?
Hi all, I'm using httpd 2.2 mod_jk as the load balancer server. There 2 tomcat nodes under the cluster. After run a long time, the tomcat node need to restart (because I found that the system resource's usage is too high, the cpu usage is almost 100%). During the restarting period, httpd load balancer maybe treat the node in error status and try to recover it. I set each node retry times to 2 (the default value): worker.node1.retries = 2 but it seems that the httpd try to recover the error node for a while, if fail it will never try to recover the node any more. HERE IS MY QUESTION: is there any way let the httpd load balancer try to recover the fail nodes per several minutes (e.g. 5 mins), and not depend on the retries parameter. Thanks. -- Best Regards! Wen Dong - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
