Re: File descriptors peaks with latest stable build of Tomcat 7
Am 20.04.2015 um 17:40 schrieb Thomas Boniface: Hi, Both nginx and tomcat are hosted on the same server when listing the connections I see both the connections from nginx to tomcat (the first one create) and the one from tomcat to nginx used to reply. I may have presented things the bad way though (I'm not too good regarding system level). I do agree the high number of close wait seems strange, I really feel like nginx closed the connection before tomcat did (what I think leads to the broken pipe expections observed in the catalina.out). In case someone want to have a look I uploaded a netstat log here: http://www.filedropper.com/netsat The connection statistics between clients and nginx http port is: Count IP:Port ConnectionState 45467 178.32.101.62:80 TIME_WAIT 44745178.33.42.6:80 TIME_WAIT 26093178.33.42.6:80 ESTABLISHED 25667 178.32.101.62:80 ESTABLISHED 6898 178.32.101.62:80 FIN_WAIT2 6723178.33.42.6:80 FIN_WAIT2 800 178.32.101.62:80 FIN_WAIT1 792178.33.42.6:80 FIN_WAIT1 712 178.32.101.62:80 LAST_ACK 656178.33.42.6:80 LAST_ACK 234178.33.42.6:80 SYN_RECV 232 178.32.101.62:80 SYN_RECV 18178.33.42.6:80 CLOSING 8 178.32.101.62:80 CLOSING 1178.33.42.6:80 CLOSE_WAIT 10.0.0.0:80 LISTEN So lots of connections in TIME_WAIT state which is kind of expected for a web server doing lots of short time client connections, but slows down the IP stack. Also quite a lot of established connections (about 50.000!), which means that you probably want to check, whether you can reduce your keep alive timeout for nginx. The same statistics for the https port: Count IP:Port ConnectionState 2283 178.32.101.62:443 TIME_WAIT 2125 178.33.42.6:443 TIME_WAIT 1585 178.32.101.62:443 ESTABLISHED 1493 178.33.42.6:443 ESTABLISHED 484 178.32.101.62:443 FIN_WAIT2 420 178.33.42.6:443 FIN_WAIT2 47 178.32.101.62:443 FIN_WAIT1 46 178.33.42.6:443 FIN_WAIT1 25 178.32.101.62:443 LAST_ACK 17 178.33.42.6:443 SYN_RECV 16 178.32.101.62:443 SYN_RECV 16 178.33.42.6:443 LAST_ACK 10 178.33.42.6:443 CLOSING 4 178.32.101.62:443 CLOSING 1 0.0.0.0:443 LISTEN About the same relative picture but only about 5% of the http connection counts. The incoming connection statistics for Tomcat (port 8080) is: Count IP:Port ConnectionState 8381127.0.0.1:8080 CLOSE_WAIT 1650127.0.0.1:8080 ESTABLISHED 127127.0.0.1:8080 SYN_RECV 65127.0.0.1:8080 TIME_WAIT 1 172.16.1.3:8080 LISTEN 1127.0.0.1:8080 LISTEN The many CLOSE_WAIT mean, that the remote side (nginx) has already closed the connection, but not yet Tomcat. Probably the idleconnection timeout / keep alive timeout for connectiosn between nginx and Tomcat is lower on the nginx side, than on the tomcat side. Interestingly the same connections but viewed from the opposite side of the connection (nginx) have totally different statistics: Count IP:Port ConnectionState 20119127.0.0.1:8080 SYN_SENT 4692127.0.0.1:8080 ESTABLISHED 488127.0.0.1:8080 FIN_WAIT2 122127.0.0.1:8080 TIME_WAIT 13127.0.0.1:8080 FIN_WAIT1 I wonder why we have 4692 established connections here, but on 1650 in the table above. In a static situation, the numbers should be the same. It indicates, that the is so much dynamics, taht the numbers vary a lot even while netstat runs. We see a lot of SYN_SENT, so nginx wants to open many more connections to Tomcat but doesn't get them as quickly as it wants. Finally there's a bunch of connections to remote web services: CountIP:Port ConnectionState 286 95.85.3.86:80 CLOSE_WAIT 255 46.228.164.12:80 ESTABLISHED 209 188.125.82.65:80 CLOSE_WAIT 172 176.74.173.230:80 ESTABLISHED 170 54.171.53.252:80 CLOSE_WAIT 136 188.125.82.65:80 LAST_ACK 129 95.85.3.86:80 LAST_ACK 128 23.212.108.209:80 CLOSE_WAIT 106 46.137.157.249:80 CLOSE_WAIT 10181.19.244.69:80 ESTABLISHED 86 146.148.30.94:80 CLOSE_WAIT 8346.137.83.90:80 CLOSE_WAIT 80 188.125.82.65:80 ESTABLISHED 78 37.252.163.221:80 CLOSE_WAIT 7746.137.83.90:80 ESTABLISHED 73 46.137.157.121:80 CLOSE_WAIT 6454.246.89.98:80 CLOSE_WAIT 63 173.194.40.153:80 ESTABLISHED 6193.176.80.69:80 ESTABLISHED 55 23.212.108.198:80 CLOSE_WAIT 5354.72.204.78:80 CLOSE_WAIT 51 37.252.162.230:80 CLOSE_WAIT 51 173.194.40.154:80 ESTABLISHED 50 54.247.113.157:80 CLOSE_WAIT 50 37.252.170.98:80 CLOSE_WAIT 49 23.212.108.191:80 CLOSE_WAIT 47 54.154.23.133:80 CLOSE_WAIT 43 176.34.179.135:80 CLOSE_WAIT 39 146.148.21.73:80 CLOSE_WAIT 36 46.137.87.196:80 CLOSE_WAIT 34 173.194.40.154:80 CLOSE_WAIT 30 46.137.87.163:80
Re: SEVERE: All threads (200) are currently busy
Am 12.04.2015 um 02:20 schrieb HG: Every once in a while we get the following (Tomcat 6) Mar 31, 2015 7:24:32 PM org.apache.tomcat.util.threads.ThreadPool logFull SEVERE: All threads (200) are currently busy, waiting. Increase maxThreads (200) or check the servlet status in the Catalina logs and server stops responding. The server is used very lightly and number of concurrent users not anywhere near 200 (2-3 on a busy day) The thread dump shows 199 of these: TP-Processor200 daemon prio=10 tid=0x2b513c31b000 nid=0x1c44 runnable [0x2b514a9a7000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read1(BufferedInputStream.java:258) at java.io.BufferedInputStream.read(BufferedInputStream.java:317) - locked 0x0007873208a0 (a java.io.BufferedInputStream) at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:628) at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:566) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:693) at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690) at java.lang.Thread.run(Thread.java:662) and one of TP-Processor4 daemon prio=10 tid=0x2b513c21a000 nid=0x7470 in Object.wait() [0x2b513552] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x000784a7a208 (a org.apache.tomcat.util.threads.ThreadPool) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.threads.ThreadPool.findControlRunnable(ThreadPool.java:339) - locked 0x000784a7a208 (a org.apache.tomcat.util.threads.ThreadPool) at org.apache.tomcat.util.threads.ThreadPool.runIt(ThreadPool.java:314) at org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:676) at org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:879) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690) at java.lang.Thread.run(Thread.java:662) Can anybody figure out why it complaints about busy threads when they are clearly not? The first 199 are connected to a fronting web server and wait for requests on their connections. You are using a blocking connector implementation. That means each web server to Tomcat connection needs on eexclusive thread to handle it. It does not matter whether the connection already has an in-flight request on it or is completely idle. The thread is exclusively working for this single connection. Once your Tomcat has as manyconnections open in parallel as your thread pool size (or n-1), it can not accept any more connections and will log the observed message. Your can choose a mix of the following optimizations: - increase your thread pool size - configure Tomcat and for web server (Apache/mod_jk, Apache/mod_proxy_ajp, ...) to shut down idle connections more aggressively to free Tomcat threads - lower idle connection timeout (Tomcat keep alive timeout and e.g. mod_jk connection_pool_timeout) - lower the number of connections, that will be kept alive even if they are idle for a long time (mod_jk connectiopn_pool_min_size) - switch to a non blocking connector implementation like NIO. There the threads are not blocked during the time the connections are idle. Idle connections are monitored by (few, e.g. 1) poller threads and only connections with in-flight requests need a normal thread pool thread. - Make sure your web servers are not oversized in terms of their own capability on how many connections in parallel they will provide Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AJP Connector : question on mod_proxy_ajp
Am 01.04.2015 um 09:43 schrieb André Warnier: Rainer Jung wrote: Am 31.03.2015 um 22:47 schrieb Christopher Schultz: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 André, On 3/31/15 3:41 PM, André Warnier wrote: I have a question of my own. ??! +1 Tomcat 6.x/7.x/8.x. Until now, we have been using mostly the Apache httpd mod_jk connector to Tomcat. And we have been using the Tomcat AJP Connector's 'tomcatAuthentication=false' setting, to propagate the authenticated user from httpd to Tomcat. Now we have a case where we must use the Apache httpd mod_proxy_ajp connector instead of mod_jk, and I want to make sure that the working of the AJP Connector's attribute tomcatAuthentication remains the same in that context. Does it ? Not automatically IIRC. Mostly the same. And do we have to specify anything special in the httpd configuration for mod_proxy_ajp to make it so, or is the authenticated user always passed to Tomcat by default, as seems to be implied here : http://httpd.apache.org/docs/2.2/mod/mod_proxy_ajp.html Attributes ?remote_user0x03String mod_jk sends the user's authentication information over in the REMOTE_USER field, so you'd just make sure that the REMOTE_USER field is being sent using mod_proxy_ajp. I'm not exactly sure what the incantation is for telling mod_proxy_ajp to send that field. You could try to see if you get a REMOTE_USER without any additional configuration on the httpd side. Note that REMOTE_USER will not show up if you call request.getAttributeNames -- you have to explicitly call request.getAttribute(REMOTE_USER) if you want to get it back. mod_jk and mod_proxy_ajp by default forward the user member of the request_rec struct as the AJP attribute SC_A_REMOTE_USER to Tomcat. If tomcatAuthentication is false, Tomcat uses this attribute instead of doing its own authentication. With mod_jk you are a bit more flexible, because you can e.g. configure it to take the user name from an Apache environment variable you choose instead of where Apache puts it when its builtin authentication procedures are used. But for the typical setup it should work for both modules out of the box. Regards, Rainer Many thanks for the confirmation. @Chris : it looks like you are thinking a bit more complicated than it is.. But your thinking may come in handy for my follow-up question : Now suppose that the front-end Apache authenticates the user, and in addition to an authenticated user-id, it is also able to collect a number of groups to which this user belongs. And suppose that being a mod_perl guru at the Apache httpd level, I could collect these groups and could use just about any scheme known to man, to pass them on to Tomcat in the proxied request. In what form would I need to do that, so that Tomcat could use these groups as roles for that user ? And how would Tomcat (or a Tomcat-based application) get to know about these roles/groups for that user ? (for example if the webapp were to call : request.isUserInRole(x) ?) (and by the way, apart from calling isUserInRole() and getting a true/false answer, is there a way for the webapp to get a list of all such roles for the user, without knowing them in advance ?) I think there's no trivial way to do that. By default the Principal retrieved via AJP13 will have no roles. But Mark just committed for 8.0.21 and the forthcoming 7.0.61 a new feature tomcatAuthorization, see http://tomcat.apache.org/tomcat-8.0-doc/config/ajp.html. If it is set to true, then the name of the principal will be used to look up the roles in whatever realm is configured in Tomcat. That is not the same as setting them on the proxy side though. We could add a custom attribute to AJP like we did before to transport roles. But where would that attribute get its content from? As far as I can see, there is no official place in Apache httpd structures where group or role info is kept around. The same modules that retrieve group info (from files, ldap, etc.) are also responsible for checking authorization, so they keep that info in module individual private data. Some of them use env vars to publish info, especialy the ldap module and it seems also the dbd one. Adding a custom AJP attribute would therefore be of limited (but non-void) use. You would have to fill a custom env var with the list of roles using mod_setenvif, mod_rewrite, mod_perl or hopefully ldap or dbd already provide the data you'd want to pass along. Do you think that would be useful? Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Post Session Id
Am 31.03.2015 um 15:19 schrieb Wesley Acheson:
Currently I'm trying to use tracking-modeSSL/tracking-mode in web.xml
but just running some local tests it appears that there are a number of
problems when using the JK connector and using this mechanism.
First issue: Even though the requests are going through AJP which supports
the SSL context information propegation, It appears I need to add a second
connector to serve over https. This is because the logic in
ApplicationConnector.java
for (Connector connector : connectors) {
if (Boolean.TRUE.equals(connector.getAttribute(SSLEnabled))) {
supportedSessionTrackingModes.add(SessionTrackingMode.SSL);
break;
}
}
Looks like the AJP connector doesn't accept that attribute.
Something we could fix, but you found a workaround.
Second issue: This is the actual issue that blocks us.
When going over mod_jk to a tomcat instance it appears that the request
attribute SSL_SESSION_ID isn't populated on the first few requests to the
server. However it is populated on subsequent requests.
This is causing the following exception.
java.lang.NullPointerException
at
org.apache.catalina.connector.CoyoteAdapter.parseSessionSslId(CoyoteAdapter.java:985)
at
org.apache.catalina.connector.CoyoteAdapter.postParseRequest(CoyoteAdapter.java:765)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:416)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:190)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Closing and repopening the browser causes the same issue to occur again.
Which means that I'm going to have to go back to posting the id, after
places where we don't control redirects back to our domain, I'm going to
have to issue a one time session lookup token to lookup the session Id.
This means sharing a data source with the Valve and the web applications.
(basically a string-string hashmap) Hopefully I can use JNDI or similar
for a local map if not its going to be needed to be backed by a database.
So remaining questions are two: How to get the SSL_SESSION_ID populated on
initial requests? Can I share some object in memory with tomcat as the
container(in a valve) and the web application?
I currently see no reason, why it shouldn't be populated right from the
start. Could you please check
- whether Apache always provides SSL_SESSION_ID:
Set SSLOptions +StdEnvVars and add %{SSL_SESSION_ID}X to your access
log (using CustomLog and a LogFormat).
This will log the ssl session id with every request that is handled by
Apache in the Apache access log. If the field does not contain an id,
then mod_jk has no chance of forwarding it and we need to solve that part.
- whether Tomcat sees the right IDs:
You can add %{javax.servlet.request.ssl_session}r to the pattern of the
AccessLogValve to add the ssl session id to your Tomcat access log.
If you can see the ID for the problematic cases in the httpd access log
but not in the Tomcat one, I'll do a little test here to reproduce.
Regards,
Rainer
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AJP Connector : question on mod_proxy_ajp
Am 01.04.2015 um 12:22 schrieb André Warnier: Rainer Jung wrote: Am 01.04.2015 um 09:43 schrieb André Warnier: Rainer Jung wrote: Am 31.03.2015 um 22:47 schrieb Christopher Schultz: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 André, On 3/31/15 3:41 PM, André Warnier wrote: I have a question of my own. ??! +1 Tomcat 6.x/7.x/8.x. Until now, we have been using mostly the Apache httpd mod_jk connector to Tomcat. And we have been using the Tomcat AJP Connector's 'tomcatAuthentication=false' setting, to propagate the authenticated user from httpd to Tomcat. Now we have a case where we must use the Apache httpd mod_proxy_ajp connector instead of mod_jk, and I want to make sure that the working of the AJP Connector's attribute tomcatAuthentication remains the same in that context. Does it ? Not automatically IIRC. Mostly the same. And do we have to specify anything special in the httpd configuration for mod_proxy_ajp to make it so, or is the authenticated user always passed to Tomcat by default, as seems to be implied here : http://httpd.apache.org/docs/2.2/mod/mod_proxy_ajp.html Attributes ?remote_user0x03String mod_jk sends the user's authentication information over in the REMOTE_USER field, so you'd just make sure that the REMOTE_USER field is being sent using mod_proxy_ajp. I'm not exactly sure what the incantation is for telling mod_proxy_ajp to send that field. You could try to see if you get a REMOTE_USER without any additional configuration on the httpd side. Note that REMOTE_USER will not show up if you call request.getAttributeNames -- you have to explicitly call request.getAttribute(REMOTE_USER) if you want to get it back. mod_jk and mod_proxy_ajp by default forward the user member of the request_rec struct as the AJP attribute SC_A_REMOTE_USER to Tomcat. If tomcatAuthentication is false, Tomcat uses this attribute instead of doing its own authentication. With mod_jk you are a bit more flexible, because you can e.g. configure it to take the user name from an Apache environment variable you choose instead of where Apache puts it when its builtin authentication procedures are used. But for the typical setup it should work for both modules out of the box. Regards, Rainer Many thanks for the confirmation. @Chris : it looks like you are thinking a bit more complicated than it is.. But your thinking may come in handy for my follow-up question : Now suppose that the front-end Apache authenticates the user, and in addition to an authenticated user-id, it is also able to collect a number of groups to which this user belongs. And suppose that being a mod_perl guru at the Apache httpd level, I could collect these groups and could use just about any scheme known to man, to pass them on to Tomcat in the proxied request. In what form would I need to do that, so that Tomcat could use these groups as roles for that user ? And how would Tomcat (or a Tomcat-based application) get to know about these roles/groups for that user ? (for example if the webapp were to call : request.isUserInRole(x) ?) (and by the way, apart from calling isUserInRole() and getting a true/false answer, is there a way for the webapp to get a list of all such roles for the user, without knowing them in advance ?) I think there's no trivial way to do that. By default the Principal retrieved via AJP13 will have no roles. But Mark just committed for 8.0.21 and the forthcoming 7.0.61 a new feature tomcatAuthorization, see http://tomcat.apache.org/tomcat-8.0-doc/config/ajp.html. If it is set to true, then the name of the principal will be used to look up the roles in whatever realm is configured in Tomcat. That is not the same as setting them on the proxy side though. We could add a custom attribute to AJP like we did before to transport roles. But where would that attribute get its content from? As far as I can see, there is no official place in Apache httpd structures where group or role info is kept around. The same modules that retrieve group info (from files, ldap, etc.) are also responsible for checking authorization, so they keep that info in module individual private data. Some of them use env vars to publish info, especialy the ldap module and it seems also the dbd one. Adding a custom AJP attribute would therefore be of limited (but non-void) use. You would have to fill a custom env var with the list of roles using mod_setenvif, mod_rewrite, mod_perl or hopefully ldap or dbd already provide the data you'd want to pass along. Do you think that would be useful? Let's say that from my point of view, that new tomcatAuthorization feature is good, because it is an additional step in the right direction. In effect, it (optionally) splits the authentication from the authorization phase, which makes the whole thing more flexible. The underlying issue here is that the concept of groups used in most of the non-Tomcat area, is distinct of the concept of role in Tomcat. In general terms, what
Re: AJP Connector : question on mod_proxy_ajp
Am 31.03.2015 um 22:47 schrieb Christopher Schultz: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 André, On 3/31/15 3:41 PM, André Warnier wrote: I have a question of my own. ??! +1 Tomcat 6.x/7.x/8.x. Until now, we have been using mostly the Apache httpd mod_jk connector to Tomcat. And we have been using the Tomcat AJP Connector's 'tomcatAuthentication=false' setting, to propagate the authenticated user from httpd to Tomcat. Now we have a case where we must use the Apache httpd mod_proxy_ajp connector instead of mod_jk, and I want to make sure that the working of the AJP Connector's attribute tomcatAuthentication remains the same in that context. Does it ? Not automatically IIRC. Mostly the same. And do we have to specify anything special in the httpd configuration for mod_proxy_ajp to make it so, or is the authenticated user always passed to Tomcat by default, as seems to be implied here : http://httpd.apache.org/docs/2.2/mod/mod_proxy_ajp.html Attributes ?remote_user0x03String mod_jk sends the user's authentication information over in the REMOTE_USER field, so you'd just make sure that the REMOTE_USER field is being sent using mod_proxy_ajp. I'm not exactly sure what the incantation is for telling mod_proxy_ajp to send that field. You could try to see if you get a REMOTE_USER without any additional configuration on the httpd side. Note that REMOTE_USER will not show up if you call request.getAttributeNames -- you have to explicitly call request.getAttribute(REMOTE_USER) if you want to get it back. mod_jk and mod_proxy_ajp by default forward the user member of the request_rec struct as the AJP attribute SC_A_REMOTE_USER to Tomcat. If tomcatAuthentication is false, Tomcat uses this attribute instead of doing its own authentication. With mod_jk you are a bit more flexible, because you can e.g. configure it to take the user name from an Apache environment variable you choose instead of where Apache puts it when its builtin authentication procedures are used. But for the typical setup it should work for both modules out of the box. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 8 on Solaris 10/11
Am 27.03.2015 um 16:15 schrieb Rainer Jung:
I'm thinking about adding client port as a loggable item to
the Tomcat AccessLog but that won't help you right now.
Done and will be available starting with TC 8.0.22 and 7.0.62. Log
pattern format is %{remote}p like for Apache httpd.
Regards,
Rainer
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 8 on Solaris 10/11
Am 27.03.2015 um 15:48 schrieb Andrew Seales: Yes, we do, on Solaris 10. I don't know of any such problems, but I can't introduce the slow network condition here to test. Good to know. In case it wasn't clear, the network limiting is done on the client side, not on the server. Is the file really truncated, i.e. too short, or is it corrupt? Can the truncation also be seen in the Tomcat access log? If so, could you replace the curl/md5sum based test with another HTTP client like LWP::Simple in perl or ab coming with Apache httpd. Just to rule out the client side of the picture. Yes the file is definitely truncated rather than corrupted. Users of our services with normal browsers are noticing the problem, it's what prompted me to use the Perl+Curl test script. Is truncation always happening at the same byte? Any pattern? Which connector are you using? NIO? APR? I've tried both AJP13 and the standard HTTP1/1 connectors, both have the same problem. When using AJP the Apache log shows the file size as being truncated, I'll check the Tomcat log when using HTTP1/1 to see if it agrees. I personally would try the following to provide additional analysis data: Find a setup, where you can log the client port. Use this setup and snoop network traffic during the test on the client and server side. Once the problem happens, use the local port number and timestamp to extract the communication pattern on the server and client side. That way you can see, which side closed/aborted the connection - or whether it is something in between client and server. Thanks, I'll give Wireshark or something like that a go to see if I can see any TCP problems. Not necessarily a problem from the TCP layer point f view. But once you can find a request that's truncated in the TCP log, you can look out - whether it was a normal connection shutdown or a reset - whether there were unusual pauses between packets triggering timeouts etc. That's why you would benefit from your test client being able to log the client port when a failure arises so you can filter easily in Wireshark. I'm thinking about adding client port as a loggable item to the Tomcat AccessLog but that won't help you right now. I vaguely remember problems with TCP checksum offloading, but they should be fixed long ago. See e.g. http://compgroups.net/comp.unix.solaris/disable-e1000-tcp-checksum-offloading-t5220/472801 I also once at a customer saw a problem not of truncation, but the last packet of a response as delayed quite noticable. That was fixed by applying the current Solaris patch cluster at that time. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 8 on Solaris 10/11
Am 26.03.2015 um 10:54 schrieb Andrew Seales: Hi, We are having a problem on our production servers where downloads of certain files are getting randomly truncated. This includes static Javascript files, file downloads via servlets, etc, where the file is more than about 100K. Most of the time the file downloads successfully, but some randomly get truncated. The truncation doesn't happen in exactly the same place every time. I've been able to recreate the issue on our development servers using Tomcat 8.0.20 with Java 1.8.20 and 1.8.40. I've tried Solaris 10, 11, SPARC and x64 CPUs and the same issue occurs. I've tested on a fresh install of Tomcat 8 and dropped in one of our larger Javascript files into the webapps/ROOT directory and made no other changes. I'm using a Perl script to continuously download the file and test an md5 hash against a known good value to test if the download breaks. It also seems to only occur when the network speed isn't very good. I use the following command to limit the speed of my network interface: sudo tc qdisc add dev eth0 root handle 1:0 netem rate 128kbit I've also tested the same Tomcat on a Redhat 6 server but that appears to work fine. If I revert to Tomcat 7.0.59, then Solaris works fine. The problem appears to only occur with Tomcat 8 on Solaris. I've tried v8.0.14 and 8.0.20 and they both have the problem. The Perl script is available from http://dlib-bauer.ucs.ed.ac.uk/testdata.pl The Javascript file is available from http://dlib-bauer.ucs.ed.ac.uk/ext-datadownload-20150323_1157.js Is anyone else running Tomcat 8 on Solaris 10 or 11 with Java 8, or know of any problems on the platform? Yes, we do, on Solaris 10. I don't know of any such problems, but I can't introduce the slow network condition here to test. Is the file really truncated, i.e. too short, or is it corrupt? Can the truncation also be seen in the Tomcat access log? If so, could you replace the curl/md5sum based test with another HTTP client like LWP::Simple in perl or ab coming with Apache httpd. Just to rule out the client side of the picture. Is truncation always happening at the same byte? Any pattern? Which connector are you using? NIO? APR? I personally would try the following to provide additional analysis data: Find a setup, where you can log the client port. Use this setup and snoop network traffic during the test on the client and server side. Once the problem happens, use the local port number and timestamp to extract the communication pattern on the server and client side. That way you can see, which side closed/aborted the connection - or whether it is something in between client and server. Unfortunately logging the client port often is not trivial to achieve. On the Tomcat side (access log), currently there is only the server port available, not the remote port, although this would be very simple to add. On the short hand it would maybe work to switch to perl plus LWP and try to get the local port from LWP. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 7 (7.0.54) memory consuption is very high(3 times) than Tomcat 6 (6.0.28)
Am 24.03.2015 um 23:40 schrieb André Warnier: Rainer Jung wrote: Some numbers from a test here on RHEL 6, using Java 1.7.0_76 and TC 6.0.43, 7.0.59 and 8.0.20. Measurement is taken directly after start (a) plus once after one request to a non-existing page and two full GCs (b). Only manager was deployed, not example webapps or docs. GC was run using jcmd PID GC.run Numbers from ps RSSa RSSb SZaSZbVSZaVSZb tc6 62372 68336 272952 273532 1091808 1094128 tc7 63608 70456 271710 271978 1086840 1087912 tc8 72576 79140 272257 272525 1089028 1090100 Differences between TC6 and 7 marginal, differences between tc7 and 8 only noticable in RSS, around 9MB. Numbers from jstat -gc. First Capacity: Semi Spaces start with 512KB and grow to 768.0KB (TC6), 896.0KB (tc7) and 1024.0 (tc8). Those indicate increasing allocations, but are not relevant for total memory use. Edena EdenbOldaOldb Perma Permb tc6 4288.0 6656.0 10688.0 16320.0 21248.0 21248.0 tc7 4288.0 7168.0 10688.0 17904.0 21248.0 21248.0 tc8 6144.0 8640.0 15316.0 21316.0 21248.0 21248.0 Again this is capacity so including garbage and unused. We see that Perm is unchanged. For all versions Eden grows by 2.4-2.9 MB due to allocation activity. Numbers for tc6 and 7 are again very similar, tc8 numbers are slightly higher already after startup. Old (Tenured) grows by about 6-7MB, again very similar for tc 6 and tc 7 and slightly higher for TC 8. Now for the used numbers after GC, which are more relevant (allocation rates are another topic): Edena EdenbOldaOldb Perma Permb tc6 2910.8 69.3 7231.7 7984.1 13923.0 14429.1 tc7 2326.1 73.8 8504.4 9661.2 13910.1 15340.1 tc8 203.7 60.9 10577.6 12599.7 16183.3 17653.8 So the live objects are Edenb+Oldb: Edenb+Oldb tc6 8053.4 tc7 9735.0 tc8 12660.6 And here we see some increase but the total amount of about 2MB between tc 6 and 7 and about another 3 MB between 7 and 8 seems to be not really problematic. The same holds true for perm, there's an increase of about 1MB between 6 and 7 and 2 MB between 7 and 8. Finally: where does the difference between RSS, Sz and the sum of heap and eden come from? Example for TC 8 case b: RSS: 79140 SZ: 272525 Sum of RSS due to smaps: 79088 so roughly consistent. S0+S1+E+O+P capacity: 53252, but Rss 40872, so a delta of 38MB to RSS. smaps entries that can be identified: TypeSize Rss Perm 21248 17656 Old21316 15040 Eden+S0+S1 10688 8176 libjvm.so 11732 7712 (read-only) Then about 21 thread stack reservations, total Size 21676, total Rss 2804. So the delta goes down to 38 - 7.7 - 2.8 = 28MB. Some more segments, that I can't fully interprete are: Size Rssfrom -to Perm File 8852 7648 7f281800-7f28188a5000 rw-p 8940 6496 7f281400-7f28148bb000 rw-p 51116 4064 7f283527-7f283845b000 rw-p 2496 2048 7f283500-7f283527 rwxp 1788 1732 7f283d443000-7f283d602000 r--s .../lib/rt.jar 3304 1444 7f282c00-7f282c33a000 rw-p 784 784 7f283f1d9000-7f283f29d000 rw-p .../lib/amd64/server/libjvm.so 1564 648 3f6d20-3f6d387000 r-xp /lib64/libc-2.12.so 536 524 7f283000-7f2830086000 rw-p 272 208 7f283f29d000-7f283f2e1000 rw-p and those nearly make up the missing 28MB Rss (whatever they are). ... But now, for the mere humans among us, what does it mean in terms of the OP and his original question : why does Tomcat 7 seem to be using 70 MB more memory at startup than Tomcat 6 ? Is it : - it doesn't matter. The numbers shown are wrong, and if you run 10 instances of Tomcat 7 at the same time, you will see that they are not really using 700 MB more than before. or - it is normal and expected. Tomcat 7 - because of the new Servlet Spec - needs to borogrove the watchamecalits, and this is using 70 MB more heap than before. In return, you get a 25% performance improvement later.. or - we have no clue. It does not happen on other machines, so there must be something special on your machine, and to find out what we need heap dumps. or - obviously some cleverer and definitive answer derived from Rainer's exhaustive analysis abobe, and which is ? From the above analysis, I get the impression that there is only really a couple of MB additional memory used as one goes from Tomcat 6 to Tomcat 7 and then to Tomcat 8. Yes. Things might change if a real app comes into play. But again I would not expect big differences if it is an app that only uses old features, e.g. runs in old and new TC versions unchanged. Note that I didn't do allocation measurements (bytes per second), only memory demand. Allocation is a different story. And that this can easily be explained by additional things/functionality which each version does, compared to the previous one. And it is so little memory, that IMHO it is not worth trying to break it down into individual explanations. But then, what could explain the 70 MB difference as shown by top
Re: Tomcat 7 (7.0.54) memory consuption is very high(3 times) than Tomcat 6 (6.0.28)
Am 25.03.2015 um 15:14 schrieb Christopher Schultz: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Rainer, On 3/25/15 6:18 AM, Rainer Jung wrote: Am 24.03.2015 um 23:40 schrieb André Warnier: Rainer Jung wrote: Some numbers from a test here on RHEL 6, using Java 1.7.0_76 and TC 6.0.43, 7.0.59 and 8.0.20. Measurement is taken directly after start (a) plus once after one request to a non-existing page and two full GCs (b). Only manager was deployed, not example webapps or docs. GC was run using jcmd PID GC.run Numbers from ps RSSa RSSb SZaSZbVSZaVSZb tc6 62372 68336 272952 273532 1091808 1094128 tc7 63608 70456 271710 271978 1086840 1087912 tc8 72576 79140 272257 272525 1089028 1090100 Differences between TC6 and 7 marginal, differences between tc7 and 8 only noticable in RSS, around 9MB. Numbers from jstat -gc. First Capacity: Semi Spaces start with 512KB and grow to 768.0KB (TC6), 896.0KB (tc7) and 1024.0 (tc8). Those indicate increasing allocations, but are not relevant for total memory use. Edena EdenbOldaOldb Perma Permb tc6 4288.0 6656.0 10688.0 16320.0 21248.0 21248.0 tc7 4288.0 7168.0 10688.0 17904.0 21248.0 21248.0 tc8 6144.0 8640.0 15316.0 21316.0 21248.0 21248.0 Again this is capacity so including garbage and unused. We see that Perm is unchanged. For all versions Eden grows by 2.4-2.9 MB due to allocation activity. Numbers for tc6 and 7 are again very similar, tc8 numbers are slightly higher already after startup. Old (Tenured) grows by about 6-7MB, again very similar for tc 6 and tc 7 and slightly higher for TC 8. Now for the used numbers after GC, which are more relevant (allocation rates are another topic): Edena EdenbOldaOldb Perma Permb tc6 2910.8 69.3 7231.7 7984.1 13923.0 14429.1 tc7 2326.1 73.8 8504.4 9661.2 13910.1 15340.1 tc8 203.7 60.9 10577.6 12599.7 16183.3 17653.8 So the live objects are Edenb+Oldb: Edenb+Oldb tc6 8053.4 tc7 9735.0 tc8 12660.6 And here we see some increase but the total amount of about 2MB between tc 6 and 7 and about another 3 MB between 7 and 8 seems to be not really problematic. The same holds true for perm, there's an increase of about 1MB between 6 and 7 and 2 MB between 7 and 8. Finally: where does the difference between RSS, Sz and the sum of heap and eden come from? Example for TC 8 case b: RSS: 79140 SZ: 272525 Sum of RSS due to smaps: 79088 so roughly consistent. S0+S1+E+O+P capacity: 53252, but Rss 40872, so a delta of 38MB to RSS. smaps entries that can be identified: TypeSize Rss Perm 21248 17656 Old21316 15040 Eden+S0+S1 10688 8176 libjvm.so 11732 7712 (read-only) Then about 21 thread stack reservations, total Size 21676, total Rss 2804. So the delta goes down to 38 - 7.7 - 2.8 = 28MB. Some more segments, that I can't fully interprete are: Size Rssfrom -to Perm File 8852 7648 7f281800-7f28188a5000 rw-p 8940 6496 7f281400-7f28148bb000 rw-p 51116 4064 7f283527-7f283845b000 rw-p 2496 2048 7f283500-7f283527 rwxp 1788 1732 7f283d443000-7f283d602000 r--s .../lib/rt.jar 3304 1444 7f282c00-7f282c33a000 rw-p 784 784 7f283f1d9000-7f283f29d000 rw-p .../lib/amd64/server/libjvm.so 1564 648 3f6d20-3f6d387000 r-xp /lib64/libc-2.12.so 536 524 7f283000-7f2830086000 rw-p 272 208 7f283f29d000-7f283f2e1000 rw-p and those nearly make up the missing 28MB Rss (whatever they are). ... But now, for the mere humans among us, what does it mean in terms of the OP and his original question : why does Tomcat 7 seem to be using 70 MB more memory at startup than Tomcat 6 ? Is it : - it doesn't matter. The numbers shown are wrong, and if you run 10 instances of Tomcat 7 at the same time, you will see that they are not really using 700 MB more than before. or - it is normal and expected. Tomcat 7 - because of the new Servlet Spec - needs to borogrove the watchamecalits, and this is using 70 MB more heap than before. In return, you get a 25% performance improvement later.. or - we have no clue. It does not happen on other machines, so there must be something special on your machine, and to find out what we need heap dumps. or - obviously some cleverer and definitive answer derived from Rainer's exhaustive analysis abobe, and which is ? From the above analysis, I get the impression that there is only really a couple of MB additional memory used as one goes from Tomcat 6 to Tomcat 7 and then to Tomcat 8. Yes. Things might change if a real app comes into play. But again I would not expect big differences if it is an app that only uses old features, e.g. runs in old and new TC versions unchanged. The OP reported that be was using Tomcat's examples application. Presumably, he was using the appropriate examples application for each of the two Tomcats and not the same one in both. More apples and pears. That's why I didn't do that. We are not really interested in examples. But I doubt
Re: Tomcat 7 (7.0.54) memory consuption is very high(3 times) than Tomcat 6 (6.0.28)
Some numbers from a test here on RHEL 6, using Java 1.7.0_76 and TC 6.0.43, 7.0.59 and 8.0.20. Measurement is taken directly after start (a) plus once after one request to a non-existing page and two full GCs (b). Only manager was deployed, not example webapps or docs. GC was run using jcmd PID GC.run Numbers from ps RSSa RSSb SZaSZbVSZaVSZb tc6 62372 68336 272952 273532 1091808 1094128 tc7 63608 70456 271710 271978 1086840 1087912 tc8 72576 79140 272257 272525 1089028 1090100 Differences between TC6 and 7 marginal, differences between tc7 and 8 only noticable in RSS, around 9MB. Numbers from jstat -gc. First Capacity: Semi Spaces start with 512KB and grow to 768.0KB (TC6), 896.0KB (tc7) and 1024.0 (tc8). Those indicate increasing allocations, but are not relevant for total memory use. Edena EdenbOldaOldb Perma Permb tc6 4288.0 6656.0 10688.0 16320.0 21248.0 21248.0 tc7 4288.0 7168.0 10688.0 17904.0 21248.0 21248.0 tc8 6144.0 8640.0 15316.0 21316.0 21248.0 21248.0 Again this is capacity so including garbage and unused. We see that Perm is unchanged. For all versions Eden grows by 2.4-2.9 MB due to allocation activity. Numbers for tc6 and 7 are again very similar, tc8 numbers are slightly higher already after startup. Old (Tenured) grows by about 6-7MB, again very similar for tc 6 and tc 7 and slightly higher for TC 8. Now for the used numbers after GC, which are more relevant (allocation rates are another topic): Edena EdenbOldaOldb Perma Permb tc6 2910.8 69.3 7231.7 7984.1 13923.0 14429.1 tc7 2326.1 73.8 8504.4 9661.2 13910.1 15340.1 tc8 203.7 60.9 10577.6 12599.7 16183.3 17653.8 So the live objects are Edenb+Oldb: Edenb+Oldb tc6 8053.4 tc7 9735.0 tc8 12660.6 And here we see some increase but the total amount of about 2MB between tc 6 and 7 and about another 3 MB between 7 and 8 seems to be not really problematic. The same holds true for perm, there's an increase of about 1MB between 6 and 7 and 2 MB between 7 and 8. Finally: where does the difference between RSS, Sz and the sum of heap and eden come from? Example for TC 8 case b: RSS: 79140 SZ: 272525 Sum of RSS due to smaps: 79088 so roughly consistent. S0+S1+E+O+P capacity: 53252, but Rss 40872, so a delta of 38MB to RSS. smaps entries that can be identified: TypeSize Rss Perm 21248 17656 Old21316 15040 Eden+S0+S1 10688 8176 libjvm.so 11732 7712 (read-only) Then about 21 thread stack reservations, total Size 21676, total Rss 2804. So the delta goes down to 38 - 7.7 - 2.8 = 28MB. Some more segments, that I can't fully interprete are: Size Rssfrom -to Perm File 8852 7648 7f281800-7f28188a5000 rw-p 8940 6496 7f281400-7f28148bb000 rw-p 51116 4064 7f283527-7f283845b000 rw-p 2496 2048 7f283500-7f283527 rwxp 1788 1732 7f283d443000-7f283d602000 r--s .../lib/rt.jar 3304 1444 7f282c00-7f282c33a000 rw-p 784 784 7f283f1d9000-7f283f29d000 rw-p .../lib/amd64/server/libjvm.so 1564 648 3f6d20-3f6d387000 r-xp /lib64/libc-2.12.so 536 524 7f283000-7f2830086000 rw-p 272 208 7f283f29d000-7f283f2e1000 rw-p and those nearly make up the missing 28MB Rss (whatever they are). Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 7 (7.0.54) memory consuption is very high(3 times) than Tomcat 6 (6.0.28)
Am 23.03.2015 um 16:26 schrieb André Warnier: Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Chuck, On 3/23/15 10:33 AM, Caldarale, Charles R wrote: From: Christopher Schultz [mailto:ch...@christopherschultz.net] Subject: Re: Tomcat 7 (7.0.54) memory consuption is very high(3 times) than Tomcat 6 (6.0.28) Really? The Tomcat ROOT web application is taking up 3 times as much heap space in Tomcat 6 as Tomcat 7? Just remember that the numbers out of top are at best approximations, and, as Rainer pointed out, not taking measurements immediately after a GC is a guarantee of an apples versus oranges comparison. The appropriate tools (e.g., VisualVM) must be used for any rational analysis. +1 The output of top and ps are completely irrelevant. The very minimum would be the output of jmap -heap, and only after a full GC were to have been run. The appropriate java-specific tools must certainly be used to find out /what/ is using this memory inside the JVM. But qualifying the output of top or ps as irrelevant is probably a bit over the top. After all, they do indicate how much the JVM is (approximately) using from an OS perspective, and that is probably not totally irrelevant here. I wanted to see the respective startup commands to check if there wasn't some change in the default startup script switches (like -Xms/-Xmx) which would explain the difference. But apparently not. Even if a GC would make the two look less different, the question would remain as to why one Tomcat would need a GC for that, and the other not. Interpretation of memory measurement in a system with shared libraries is non-trivial. E.g. if both measurements are not taken on the same system, and on the TC 6 system another Java process with the JVM is already running, parts of the total memory needed by Tomcat could be assigned by OS tools to the other process, because they are shared. So I would look at OS reported sizes *and* at JVM reported sizes to get an idea, which part is atually that different. OS reported values are e.g. using ps: rss, sz and vsz Also interesting is cat /proc/PID/maps but here one would need to calculate sizes per line from the two hex addresses given at the start of each line. Something like: cat /proc/PID/maps | perl -n -e '($a,$b)=split(/[- ]/);print hex($b)-hex($a), , $_;' | sort -n (replace PID by the current Tomcat java process id). JVM reported values to be taken after two times GC (the first one might be needed if the first one just runs finalizers) and then either taking it from a GC log (to be added before starting the process) or from the textual output of a thread dump (kill -QUIT) or from jstat -gc PID. A heap dump IMHO is to cumbersome at this stage. The gc+measurements should also probably only be done after accessing Tomcat via HTTP at least once, to increase probability of similar state of initialization. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 7 (7.0.54) memory consuption is very high(3 times) than Tomcat 6 (6.0.28)
Am 23.03.2015 um 10:02 schrieb André Warnier: Rahul Kumar Singh wrote: Dear Thomas, Thanks for your quick response, Tomcat doesn't use anything like that much memory on its own. I suspect it is the application although the difference between Tomcat 6 and Tomcat 7 is unexpected. Ok, I understand, But could you please confirm us theoretically that is there any difference in initial memory requirement of tomcat6 and tomcat 7? That should be much easier and quicker to check this by yourself, as you obviously already have the 2 Tomcat versions up and running. Remove your own application, deploy one of the standard Tomcat-examples applications, and have a look. If there is still a big difference between v6 and v7, then come back here. At least then there will be a reproducible test case for someone here to have a closer look. And if there is not a big difference, then obviously it must have something to with your application. And then, follow Mark's already-given advice to find out what. In addition: run a Garbage Collection before doing the measurement, so you only measure live objects, not garbage. As you didn't tel us how you did your measurement, we can't know. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 7 (7.0.54) memory consuption is very high(3 times) than Tomcat 6 (6.0.28)
Am 23.03.2015 um 17:16 schrieb Caldarale, Charles R:
From: Rainer Jung [mailto:rainer.j...@kippdata.de]
Subject: Re: Tomcat 7 (7.0.54) memory consuption is very high(3 times) than
Tomcat 6 (6.0.28)
Also interesting is cat /proc/PID/maps but here one would need to
calculate sizes per line from the two hex addresses given at the start
of each line.
/proc/pid/smaps gives a bit more info (including various sizes), but produces
multiple lines per record, making sorting more difficult.
At least on my variant of Linux the following puts the data in one line
and sorts by first size (Size). Choosing another columnwould mean
changing the 2 in the sort command.
cat /proc/PID/smaps | \
perl -n -e 'END {print $line, \n} chomp(); if ($_ =~ /^[0-9a-f]+-/)
{print $line, \n; $line=$_} else {$_ =~ s/.* ([0-9]+ )/$1/; $line =
$line . | . $_;}' | \
sort -t '|' -n -k 2
Regards,
Rainer
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Migration from Tomcat6-Cluster to Tomcat7-Cluster: Digest Authentication problem
Am 17.03.2015 um 15:40 schrieb Sascha Skorupa: Hi Rainer, currently not (Apache 2.2) but it might be an option to upgrade the OS and the Apache if it leads to a solution. OK. But think twice, whether it is better to just compile mod_jk from sources or do the big update. Updating to 2.4 will bring many interesting achievements, but just for fixing this issue quickly it would be better to update mod_jk, even if this means switching to a non-OS-provided variant. If you seriously plan the 2.4 update and you have a test system, I could provide you with the non-trivial workaround letting Apache set the cookie. You would need to thoroughly test this though. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Migration from Tomcat6-Cluster to Tomcat7-Cluster: Digest Authentication problem
Hi Sascha, Am 17.03.2015 um 13:02 schrieb Sascha Skorupa: Rainer, thank you for this hint, but unfortunately, this feature is too new to be included in any current mod_jk linux package and building it from source it not an option for a production environment. Too bad because that is exactly what we need that. are you using Apache 2.4? In that case I might have an alternative recipe for you. Regards, Rainer Christopher, your suggestion sounds interesting. Would it be an option for future releases of tomcat? Sascha -Ursprüngliche Nachricht- Von: Christopher Schultz [mailto:ch...@christopherschultz.net] Gesendet: Freitag, 13. März 2015 19:24 An: Tomcat Users List Betreff: Re: Migration from Tomcat6-Cluster to Tomcat7-Cluster: Digest Authentication problem -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Rainer, On 3/13/15 12:15 PM, Rainer Jung wrote: Am 13.03.2015 um 16:28 schrieb Christopher Schultz: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Mark, On 3/12/15 1:13 PM, Mark Thomas wrote: On 12/03/2015 15:20, Sascha Skorupa wrote: Hi, here: http://grokbase.com/t/tomcat/users/13bvsbwb8s/multiple-servers-and- digest-authentication the same problem is described and the recommended solution is to use sticky load balancing. But, the problem in a tomcat cluster is that the session ID is generated after a successful authentication. The first http response (401 with Authentication Header) does not contain a session ID. How should sticky load balancing be configured or how to enforce session id generation before authentication? Most load-balancers have various options for doing this that don't depend on the back-end server at all. Perhaps an option in Tomcat that will force the creation of a session when a DIGEST authentication is requested might be useful. This would tie e.g. mod_jk to the proper back-end server. I'm not sure how this could be done using mod_jk without such a feature, or changes to mod_jk itself to annotate the request with the chosen worker, which could then be converted into a cookie in order to keep the node-hint associated with the client. Yes, mod_jk can help since version 1.2.38: Look for set_session_cookie on http://tomcat.apache.org/connectors-doc/reference/workers.html. Using that attribute you can let mod_jk set the cookie, if it doesn't find one already set by Tomcat. You need to also set session_cookie=JSESSIONID and session_cookie_path=/myapp where you adjust myapp to your context path. Oh, good: I had missed that feature. Thanks for pointing it out! - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJVAytTAAoJEBzwKT+lPKRYO9EP/1NnmcKmXYwYYvtnb97dMZmz NI7GIaZYDhx1NLb7w5UFDrPNhaAvBDSvzmNxnhZPfdPcwUK93k95+KVymrpI49xh wJ7wbBlFPJcB6coK35jwp0oZnfKbUyHFZ6Qr/r4fbrd57PDqCKGLc/6YicY11nm+ 3EGCYHKe/B2orPNJvw+B5ZCm4cJFwh/VWespkAylCWbqJV1k882zG4MJEwZWgXdn FBLggaCW1N5V6LNAhKiwyrrZrcV9TY3zoMI4Dd8M8yzq8MoTbUK2g2sYZh0LNm0d 0ZbcE07uBUgO5NamNk049vSK+yx0gfHmnL1Gst/jdQ23I8876cTYNCkJKyb0/uDq x5nv2K+dYj1rDeak3j4PS35W+Z6C/SCyp8n3W2L16xtLYtDRQTYv4OTOv1oVTuA2 UueFMwnqRoJV24nLathp29RnjODK7shYce1JqShiAVXzyKSgAkWsu2J8V07txUjP 4v+E0LrWOGEimvHfrOYqdTmH+Ed6YEcttrYU2ddH1g2z4Hz9n+S+fsO2fQgLhY/S V6RluOXfAlg6+IFEtW7DW2PzXuQxOHfKfIX3dlBDGrJGoYNFdYY+uaZO6TPyp7qR UVO9BRA0Roxtpvn7TpJJjygjNXM7uac5MMeCJtA4KPd29PT0sxAnJv+NYpYJ1F35 XROKEh5OIpcNKOPxBoof =w+jN -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Migration from Tomcat6-Cluster to Tomcat7-Cluster: Digest Authentication problem
Am 13.03.2015 um 16:28 schrieb Christopher Schultz: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Mark, On 3/12/15 1:13 PM, Mark Thomas wrote: On 12/03/2015 15:20, Sascha Skorupa wrote: Hi, here: http://grokbase.com/t/tomcat/users/13bvsbwb8s/multiple-servers-and-digest-authentication the same problem is described and the recommended solution is to use sticky load balancing. But, the problem in a tomcat cluster is that the session ID is generated after a successful authentication. The first http response (401 with Authentication Header) does not contain a session ID. How should sticky load balancing be configured or how to enforce session id generation before authentication? Most load-balancers have various options for doing this that don't depend on the back-end server at all. Perhaps an option in Tomcat that will force the creation of a session when a DIGEST authentication is requested might be useful. This would tie e.g. mod_jk to the proper back-end server. I'm not sure how this could be done using mod_jk without such a feature, or changes to mod_jk itself to annotate the request with the chosen worker, which could then be converted into a cookie in order to keep the node-hint associated with the client. Yes, mod_jk can help since version 1.2.38: Look for set_session_cookie on http://tomcat.apache.org/connectors-doc/reference/workers.html. Using that attribute you can let mod_jk set the cookie, if it doesn't find one already set by Tomcat. You need to also set session_cookie=JSESSIONID and session_cookie_path=/myapp where you adjust myapp to your context path. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk causing stuck Apache processes
Am 03.03.2015 um 23:11 schrieb Jesse Defer: any update on the behavior of the patched nodes? Did it behave better, ie. did the problem reoccur on other nodes but not on the patched ones, or were all nodes mostly without problems during the last 2 days? Any strange things on the patched ones? I'd like to add the patch for 1.2.41 but your feedback would be very valuable. Thanks! Rainer It's been quiet, no stuck processes anywhere in the farm. The patched jk is working fine, no issues found so far. I'll continue to monitor and let you know. Just to make sure: no stuck processes anywhere in the farm means we can't be sure, that the patch helps, because the unpatched nodes didn't have a problem either during that time. So we have to wait until the problem happens a few more times so we can see whether the patch helps, or the patched nodes have run long enough so that at least we don't expect a negative impact from the patch. Rainer, The patch has not completely fixed the issue, I had a host today exhibit the behavior. It does seem to have better behavior: stuck process don't hang around as long though there are still some 300 second ones. It also seemed to recover by itself, though it took about 40 minutes. It also had more idle processes on average, though it did hit 90% in W for a few minutes. I have also converted 4 (different) hosts to the event mpm and was tweaking the mpm settings all last week. I think they are in a good state now and so far I haven't seen the issue on them. I now have 8 on stock mod_jk/prefork, 4 on your patched mod_jk/prefork, and 4 on stock mod_jk/event. We are upgrading to httpd 2.4.12 on Friday and I am also gone on vacation all next week. We haven't decided yet if we are switching to event at the same time, but it seems likely. I am willing to continue testing your patch but I won't be able to resume until the week of the 16th and it may not be under prefork. Thanks for your efforts. I have committed the changes to mod_jk in 1665454. They will be part of version 1.2.41. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [URGENT] Content-Encoding: gzip not set
Hi Chris, Am 10.03.2015 um 15:09 schrieb Christopher Schultz: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Rainer, On 3/9/15 7:01 PM, Rainer Jung wrote: Am 09.03.2015 um 23:11 schrieb Christopher Schultz: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Igor, On 3/9/15 6:01 PM, Igor Cicimov wrote: On 10/03/2015 6:14 AM, Victor Rodriguez victropo...@gmail.com wrote: Greetings, I have some ALREADY gzipped files that I'm trying to serve up. I have the following in my web.xml. mime-mapping extensionjson/extension mime-typeapplication/gzip/mime-type /mime-mapping And, I have the following in my server.xml: Context docBase=/path/to/already-gzipped-json path=/already-gzipped-json / From the command line, I can curl the files and gunzip them just fine, so they are coming across gzipped: curl http://localhost:8082/already-gzipped-json/fie.json | gunzip - However, requests coming from a web browser aren't handled correctly and aren't legible in the browser, and I believe it's because Content-Encoding: gzip is not in the response headers. You mean Accept-Encoding, right? Is tomcat fronted by apache, nginx or something else that can add this header for you? Ironically, getting this to work as requested in Apache httpd is a complete nightmare. The Tomcat solution basically works *exactly* as a user would want it to work. Agreed, that the feature in the default servlet is much much easier to use than configure pre-compressed content in httpd. But nightmare might be a bit to strong. It is tricky. See for example: http://mail-archives.apache.org/mod_mbox/httpd-users/201110.mbox/%3c4e8e51c0.4050...@kippdata.de%3E Would you be up for the creation of mod_already_gzipped? It could act just like the DefaultServlet's behavior: 1. Did the client say it could Accept-Encoding: gzip? 2. Does the file [translated-path].gz exist? 3. Then send those bytes with Content-Encoding: gzip and Content-Length: size(file.gz) It seems like that should be a no-brainer in terms of functionality. Using MultiViews and re-naming your files to bizarre filenames, etc. is a big hack that I finally decided wasn't worth it. Note that the linked recipe I provided for httpd does not use MultiViews and also does not force you to rename the existing (uncompressed) files. It contains 5 lines of Apache config, of which 2 are most likely already present. So it boils down to one RewriteRule plus RewriteCond and one mod_headers directive. How and why it works is the long part of the description. The only other thing is the maybe uncommon policy for the names of the *compressed* files, keeping the .gz in front of the original file name suffix. You could even move it completely in front of the file name like gz.myfile.suf to make it even more uncommon ;) But yes, the recipe still needs to be adjusted depending on where your static files are, for which files you want to use it etc. So packing it into a simple module could be worthwhile. I'll post it if I get to it. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [URGENT] Content-Encoding: gzip not set
Am 09.03.2015 um 23:11 schrieb Christopher Schultz: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Igor, On 3/9/15 6:01 PM, Igor Cicimov wrote: On 10/03/2015 6:14 AM, Victor Rodriguez victropo...@gmail.com wrote: Greetings, I have some ALREADY gzipped files that I'm trying to serve up. I have the following in my web.xml. mime-mapping extensionjson/extension mime-typeapplication/gzip/mime-type /mime-mapping And, I have the following in my server.xml: Context docBase=/path/to/already-gzipped-json path=/already-gzipped-json / From the command line, I can curl the files and gunzip them just fine, so they are coming across gzipped: curl http://localhost:8082/already-gzipped-json/fie.json | gunzip - However, requests coming from a web browser aren't handled correctly and aren't legible in the browser, and I believe it's because Content-Encoding: gzip is not in the response headers. You mean Accept-Encoding, right? Is tomcat fronted by apache, nginx or something else that can add this header for you? Ironically, getting this to work as requested in Apache httpd is a complete nightmare. The Tomcat solution basically works *exactly* as a user would want it to work. Agreed, that the feature in the default servlet is much much easier to use than configure pre-compressed content in httpd. But nightmare might be a bit to strong. It is tricky. See for example: http://mail-archives.apache.org/mod_mbox/httpd-users/201110.mbox/%3c4e8e51c0.4050...@kippdata.de%3E Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Fwd: File upload fails after upgrade to 7.0.59
Am 03.03.2015 um 13:45 schrieb Umesh Sehgal: On Mon, Mar 2, 2015 at 6:20 PM, Rainer Jung rainer.j...@kippdata.de wrote: Am 02.03.2015 um 13:34 schrieb Umesh Sehgal: On Mon, Mar 2, 2015 at 5:25 PM, Umesh Sehgal umesh.seh...@gmail.com wrote: On Mon, Mar 2, 2015 at 3:52 PM, Rainer Jung rainer.j...@kippdata.de wrote: Am 02.03.2015 um 11:02 schrieb Umesh Sehgal: Thanks for the quick reply, I tried using the maxSwallowSize with increased value but to no effect. The max size that I have been able to upload is ~16 KB. I also see that the maxSwallowSize got introduced with update 55 but the behavior I'm seeing is 50 update onwards, is there any other param too? is there any logging that can be turned on tomcat to help debug? Please do not top post. For the rest see below. On Mon, Mar 2, 2015 at 2:32 PM, Rainer Jung rainer.j...@kippdata.de wrote: Am 02.03.2015 um 09:34 schrieb Umesh Sehgal: Hi, We recently upgraded our application's tomcat from 7.0.30 to 7.0.59. After upgrade the file upload feature has broken. I have been able to nail it down to the point that the problem manifests 7.0.50 onwards. Here is the exception that I see inside logs: Caused by: java.net.SocketException: Connection reset by peer: socket write error at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(Unknown Source) at java.net.SocketOutputStream.write(Unknown Source) at sun.security.ssl.OutputRecord.writeBuffer(Unknown Source) at sun.security.ssl.OutputRecord.write(Unknown Source) Also, I notice that the problem doesn't happen with a 2KB file but 2MB. I don't see anything obvious in the 7.0.50 changelog which could explain this behavior. Can someone please provide some pointer as what could be causing this? https://bz.apache.org/bugzilla/show_bug.cgi?id=57617 Fixed for next 7.0.60 in http://svn.apache.org/r1659295 The original change can be found looking for maxSwallowSize in the changelog Could it be If a request that includes an Expect: 100-continue header receives anything other than a 2xx response, close the connection This protects against misbehaving clients that may not sent the request body in that case and send the next request instead. (markt) ? It was changed in 7.0.49, but 49 was not released, so 50 was the first version with this change. Regards, Rainer I did see this in changelog but in the captured traffic don't see any expect 100 header request. Any other way I can confirm this on the tomcat side? Thanks, Umesh Can you please point me to SVN change for : If a request that includes an Expect: 100-continue header receives anything other than a 2xx response, close the connection This protects against misbehaving clients that may not sent the request body in that case and send the next request instead. (markt) ? http://svn.apache.org/r1540689 Hi Rainer, Thanks, I think the problem is indeed caused by this change. I downloaded the tomcat source, removed the above change from AbstractHttp11Processor and delpoyed the updated jar. The file upload didn't work right away but at least now maxSwallowSize is honored and I can upload the files per the size specified. I did the above work to confirm, but of course I don't want to ship it carrying modified code. Can you please suggest as what could be done in this case? OK, good to know. I'd say now it would be good to find out why your webapp sends a non-2xx response code and hwich it is. Since you already suceeded to build tomcat, simply add a custom log or System.out.println() statement printing response.getStatus() where the change in r1540689 was added and tell us what it is for the failing uploads. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk dealy sendng request to backend tomcat
Please don't top post, it makes following the communication very hard. Am 02.03.2015 um 12:48 schrieb Rajesh Cherukuri: for telnet immediateconnection refused is given telnet 10.xxx.xxx.xx 8911 Trying 10.xx.xx.x... telnet: connect to address 10.xx.x.xx: Connection refused Then I would expect that mod_jk gets exactly the same quick error. Maybe the network situation between your mod_jk server and the Tomcat worker was different, when the problem occurred. But ... here is the error log for the specifed time the logs looks strange to me. It could be, that mod_jk did not really run into a 10 second timeout. Error 111 on Linux is connection refused, not a timeout. Is that 20 second delay always the case? Could it be that your http server was overloaded? Are you using the latest mod_jk version? [Fri Feb 27 02:26:14.463 2015] [31713:140059770595072] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Fri Feb 27 02:27:30.458 2015] [3471:140059728635648] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:27:30.560 2015] [3471:140059728635648] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:27:30.560 2015] [3471:140059728635648] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Fri Feb 27 02:28:14.585 2015] [31713:140059550308096] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:28:14.686 2015] [31713:140059550308096] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:28:14.687 2015] [31713:140059550308096] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Fri Feb 27 02:29:14.401 2015] [31713:140059728635648] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:29:14.503 2015] [31713:140059728635648] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:29:14.503 2015] [31713:140059728635648] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Fri Feb 27 02:30:40.149 2015] [31713:140059739125504] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:30:40.250 2015] [31713:140059739125504] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:30:40.251 2015] [31713:140059739125504] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Fri Feb 27 02:31:15.442 2015] [31713:140059644716800] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Fri Feb 27 02:31:15.543 2015] [31713:140059644716800] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend failed. Tomcat is probably not started or is listening on On Mon, Mar 2, 2015 at 4:28 PM, Rainer Jung rainer.j...@kippdata.de wrote: Am 02.03.2015 um 11:34 schrieb Rajesh Cherukuri: rainer looks like what you said is correct , but not sure why the Mod_jk has to wait for 10 seconds when the backend tomcat servers is down Because your network layer behaves like that. It simply hangs for (more than) 10 seconds. You should be able to observer that yourself e.g. using telnet tomcatserverip tomcatajpport It should hang that long as well. here is the error log i don't see that any place where it is aitng for 20 sec The situation you want to discuss happened at 02:28:14, the log snippet is from 01:28:35 to 01:33:31. So they do not match. error log [Thu Feb 26 01:28:35.432 2015] [19535:140214433396480] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:28:35.534 2015] [19535:140214433396480] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111
Re: Fwd: File upload fails after upgrade to 7.0.59
Am 02.03.2015 um 09:34 schrieb Umesh Sehgal: Hi, We recently upgraded our application's tomcat from 7.0.30 to 7.0.59. After upgrade the file upload feature has broken. I have been able to nail it down to the point that the problem manifests 7.0.50 onwards. Here is the exception that I see inside logs: Caused by: java.net.SocketException: Connection reset by peer: socket write error at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(Unknown Source) at java.net.SocketOutputStream.write(Unknown Source) at sun.security.ssl.OutputRecord.writeBuffer(Unknown Source) at sun.security.ssl.OutputRecord.write(Unknown Source) Also, I notice that the problem doesn't happen with a 2KB file but 2MB. I don't see anything obvious in the 7.0.50 changelog which could explain this behavior. Can someone please provide some pointer as what could be causing this? https://bz.apache.org/bugzilla/show_bug.cgi?id=57617 Fixed for next 7.0.60 in http://svn.apache.org/r1659295 The original change can be found looking for maxSwallowSize in the changelog Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Fwd: File upload fails after upgrade to 7.0.59
Am 02.03.2015 um 13:34 schrieb Umesh Sehgal: On Mon, Mar 2, 2015 at 5:25 PM, Umesh Sehgal umesh.seh...@gmail.com wrote: On Mon, Mar 2, 2015 at 3:52 PM, Rainer Jung rainer.j...@kippdata.de wrote: Am 02.03.2015 um 11:02 schrieb Umesh Sehgal: Thanks for the quick reply, I tried using the maxSwallowSize with increased value but to no effect. The max size that I have been able to upload is ~16 KB. I also see that the maxSwallowSize got introduced with update 55 but the behavior I'm seeing is 50 update onwards, is there any other param too? is there any logging that can be turned on tomcat to help debug? Please do not top post. For the rest see below. On Mon, Mar 2, 2015 at 2:32 PM, Rainer Jung rainer.j...@kippdata.de wrote: Am 02.03.2015 um 09:34 schrieb Umesh Sehgal: Hi, We recently upgraded our application's tomcat from 7.0.30 to 7.0.59. After upgrade the file upload feature has broken. I have been able to nail it down to the point that the problem manifests 7.0.50 onwards. Here is the exception that I see inside logs: Caused by: java.net.SocketException: Connection reset by peer: socket write error at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(Unknown Source) at java.net.SocketOutputStream.write(Unknown Source) at sun.security.ssl.OutputRecord.writeBuffer(Unknown Source) at sun.security.ssl.OutputRecord.write(Unknown Source) Also, I notice that the problem doesn't happen with a 2KB file but 2MB. I don't see anything obvious in the 7.0.50 changelog which could explain this behavior. Can someone please provide some pointer as what could be causing this? https://bz.apache.org/bugzilla/show_bug.cgi?id=57617 Fixed for next 7.0.60 in http://svn.apache.org/r1659295 The original change can be found looking for maxSwallowSize in the changelog Could it be If a request that includes an Expect: 100-continue header receives anything other than a 2xx response, close the connection This protects against misbehaving clients that may not sent the request body in that case and send the next request instead. (markt) ? It was changed in 7.0.49, but 49 was not released, so 50 was the first version with this change. Regards, Rainer I did see this in changelog but in the captured traffic don't see any expect 100 header request. Any other way I can confirm this on the tomcat side? Thanks, Umesh Can you please point me to SVN change for : If a request that includes an Expect: 100-continue header receives anything other than a 2xx response, close the connection This protects against misbehaving clients that may not sent the request body in that case and send the next request instead. (markt) ? http://svn.apache.org/r1540689 - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk dealy sendng request to backend tomcat
Am 02.03.2015 um 10:45 schrieb Rajesh Cherukuri: here is what i can see in modJk log file , we have stopped tomcat 11 a week back , 172.x [27/Feb/2015:*02:28:14* +] GET /xx/deal-finder HTTP/1.1200 15172 20 *lb* tomcat--11** 0 26712 12118 **tomcat-03** 0 820679 4 *20.352845* – worker configuration *here is my work.property file* *worker.lb.common.type=ajp13worker.lb.common.lbfactor=1worker.lb.common.ping_mode=Aworker.lb.common.socket_connect_timeout=1# worker.lb.common.connection_pool_size=6worker.lb.common.connection_pool_timeout=600worker.lb.common.socket_keepalive=True* That is not a production ready config. - update to the recent version 1.2.40 (if your version is older) - build your configuration starting from the *source* download linked on http://tomcat.apache.org/download-connectors.cgi. In the source download (tarball or zip) there's a conf folder containing a workers.properties and a httpd-jk.conf file. Look at those and adjust to your environment. They contain generally useful timeouts etc. The 20 seconds you are noticing are likely 10 seconds (connect_timeout) times 2 (retries). If the server for a Tomcat worker is up and Tomcat itself is down, the server should typically reject a new connect very fast, because nothing is listening to the port. If the server is down it depends on the network situation, whether you get a fast error (e.g. no route to host etc.) or whether it takes long. If the server is up and Tomcat is up, but your Tomcat jvm doesn't respond, it again depends on the details. If you know, that a node is down, then you should set its activation to stopped to let mod_jk know, that it should no longer try to use it. You mod_jk log should show more information than what you posted. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk dealy sendng request to backend tomcat
Am 02.03.2015 um 11:34 schrieb Rajesh Cherukuri: rainer looks like what you said is correct , but not sure why the Mod_jk has to wait for 10 seconds when the backend tomcat servers is down Because your network layer behaves like that. It simply hangs for (more than) 10 seconds. You should be able to observer that yourself e.g. using telnet tomcatserverip tomcatajpport It should hang that long as well. here is the error log i don't see that any place where it is aitng for 20 sec The situation you want to discuss happened at 02:28:14, the log snippet is from 01:28:35 to 01:33:31. So they do not match. error log [Thu Feb 26 01:28:35.432 2015] [19535:140214433396480] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:28:35.534 2015] [19535:140214433396480] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:28:35.534 2015] [19535:140214433396480] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Thu Feb 26 01:30:26.182 2015] [19536:140214611724032] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:30:26.284 2015] [19536:140214611724032] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:30:26.284 2015] [19536:140214611724032] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Thu Feb 26 01:30:32.670 2015] [19538:140214548784896] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:30:32.772 2015] [19538:140214548784896] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:30:32.772 2015] [19538:140214548784896] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Thu Feb 26 01:31:37.441 2015] [19535:140214318008064] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:31:37.543 2015] [19535:140214318008064] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:31:37.543 2015] [19535:140214318008064] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Thu Feb 26 01:32:29.323 2015] [19534:140214338987776] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:32:29.425 2015] [19534:140214338987776] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:32:29.425 2015] [19534:140214338987776] [error] ajp_service::jk_ajp_common.c (2643): (tomcat-live-11) connecting to tomcat failed. [Thu Feb 26 01:33:31.431 2015] [19536:140214569764608] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port (errno=111) [Thu Feb 26 01:33:31.532 2015] [19536:140214569764608] [error] ajp_send_request::jk_ajp_common.c (1630): (tomcat-live-11) connecting to backend fai led. Tomcat is probably not started or is listening on the wrong port ( On Mon, Mar 2, 2015 at 3:47 PM, Rainer Jung rainer.j...@kippdata.de wrote: Am 02.03.2015 um 10:45 schrieb Rajesh Cherukuri: here is what i can see in modJk log file , we have stopped tomcat 11 a week back , 172.x [27/Feb/2015:*02:28:14* +] GET /xx/deal-finder HTTP/1.1200 15172 20 *lb* tomcat--11** 0 26712 12118 **tomcat-03** 0 820679 4 *20.352845* – worker configuration *here is my work.property file* *worker.lb.common.type=ajp13worker.lb.common.lbfactor=1worker.lb.common. ping_mode=Aworker.lb.common.socket_connect_timeout=1# worker.lb.common.connection_pool_size=6worker.lb.common. connection_pool_timeout=600worker.lb.common.socket_keepalive=True* That is not a production ready config. - update to the recent version 1.2.40 (if your version is older) - build your configuration starting from the *source* download linked on http
Re: Fwd: File upload fails after upgrade to 7.0.59
Am 02.03.2015 um 11:02 schrieb Umesh Sehgal: Thanks for the quick reply, I tried using the maxSwallowSize with increased value but to no effect. The max size that I have been able to upload is ~16 KB. I also see that the maxSwallowSize got introduced with update 55 but the behavior I'm seeing is 50 update onwards, is there any other param too? is there any logging that can be turned on tomcat to help debug? Please do not top post. For the rest see below. On Mon, Mar 2, 2015 at 2:32 PM, Rainer Jung rainer.j...@kippdata.de wrote: Am 02.03.2015 um 09:34 schrieb Umesh Sehgal: Hi, We recently upgraded our application's tomcat from 7.0.30 to 7.0.59. After upgrade the file upload feature has broken. I have been able to nail it down to the point that the problem manifests 7.0.50 onwards. Here is the exception that I see inside logs: Caused by: java.net.SocketException: Connection reset by peer: socket write error at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(Unknown Source) at java.net.SocketOutputStream.write(Unknown Source) at sun.security.ssl.OutputRecord.writeBuffer(Unknown Source) at sun.security.ssl.OutputRecord.write(Unknown Source) Also, I notice that the problem doesn't happen with a 2KB file but 2MB. I don't see anything obvious in the 7.0.50 changelog which could explain this behavior. Can someone please provide some pointer as what could be causing this? https://bz.apache.org/bugzilla/show_bug.cgi?id=57617 Fixed for next 7.0.60 in http://svn.apache.org/r1659295 The original change can be found looking for maxSwallowSize in the changelog Could it be If a request that includes an Expect: 100-continue header receives anything other than a 2xx response, close the connection This protects against misbehaving clients that may not sent the request body in that case and send the next request instead. (markt) ? It was changed in 7.0.49, but 49 was not released, so 50 was the first version with this change. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk dealy sendng request to backend tomcat
Am 01.03.2015 um 17:57 schrieb Rajesh Cherukuri: hi today i have noticed that mod_jk took 20 seconds delay in sending the request to backend tomcat servers from apache , lookslike the request was assigned to a tomcat 11and it has replicate the request to the other tomcat-3 here are my findings Deal Finder request * Apache request access logs * 172.xx - - [27/Feb/2015:*02:28:14* +] GET /x/deal-finder HTTP/1.1 200 15172 - 195..xxx.xx, - - Mod_Jk apache log Mod Jk logs show that the request was assigned to tomcat live 11 but it was reassigned to tomcat live 03 since it was down so there might be delay due to this 172.x [27/Feb/2015:*02:28:14* +] GET /xx/deal-finder HTTP/1.1 200 15172 20 *lb tomcat--11* 0 26712 12118 *tomcat-03* 0 820679 4 20.352845 – *Tomcat access logs - Delay of 20 sec from mod JK * 172.xxx - - [27/Feb/2015:02:*28:34* +] 'GET /xx/deal-finder HTTP/1.1' 200 15172 TP-Processor243 D45BF91DBAF9D2CBFFBC2D6B19D3CC61.tomcat-03 0.127 Tomcat response in 0.127 millseconds quick What's in your mod_jk log file? What is your worker configuration? Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk causing stuck Apache processes
Am 27.02.2015 um 18:15 schrieb Jesse Defer: Hi Jesse, any update on the behavior of the patched nodes? Did it behave better, ie. did the problem reoccur on other nodes but not on the patched ones, or were all nodes mostly without problems during the last 2 days? Any strange things on the patched ones? I'd like to add the patch for 1.2.41 but your feedback would be very valuable. Thanks! Rainer It's been quiet, no stuck processes anywhere in the farm. The patched jk is working fine, no issues found so far. I'll continue to monitor and let you know. Just to make sure: no stuck processes anywhere in the farm means we can't be sure, that the patch helps, because the unpatched nodes didn't have a problem either during that time. So we have to wait until the problem happens a few more times so we can see whether the patch helps, or the patched nodes have run long enough so that at least we don't expect a negative impact from the patch. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: when idle tomcat runs on 3.9% CPU
Am 27.02.2015 um 18:07 schrieb Christopher Schultz:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Peter,
On 2/27/15 11:21 AM, Peter Irbizon wrote:
Hi Chris, here are my dumps
http://www.filedropper.com/jvmthreaddump. It looks like it has
something to do with memcached.
There are a bunch of RUNNABLE threads, but sometimes the term
RUNNABLE in a Java thread dump doesn't actually mean taking CPU
time right now. For example, the main thread is RUNNABLE but it's
waiting on accepting a socket connection (for a SHUTDOWN command) and
so it's effectively blocked, taking zero CPU time.
I'm not entirely sure what the Memcached threads actually do, but I
think they are in a similar situation: waiting for code to request
some communication with Memcached.
I agree, hard to say from those dumps what's consuming the CPU, could be
even GC threads. It's easier if threads are actually doing something,
not just hanging around and waiting for work.
If you have commandline access, use
top -H -p PID
to show the list of threads of your process (replace PID by the process
ID of your Java process) and check, what the thread numbers that are
active on the CPU are. Hopefully it is only a few threads. Then take the
thread number and translate it from decimal to hexadecimal and look for
it in the header lines of your thread dumps.
Example:
The thread
Memcached IO over {MemcachedConnection to localhost/127.0.0.1:11215}
prio=10 tid=0x025558d8 nid=0x34d3 runnable [0x9f4fe000]
from the dump will be shown on some Linuxes in top -H as thread id
0x34d3 = 13523
on others as
0x025558d8 = 39147736
If the thread ID doesn't occur in the thread dump at all, then it is
native threads (unlikely but possible).
Regards,
Rainer
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk causing stuck Apache processes
Hi Jesse, any update on the behavior of the patched nodes? Did it behave better, ie. did the problem reoccur on other nodes but not on the patched ones, or were all nodes mostly without problems during the last 2 days? Any strange things on the patched ones? I'd like to add the patch for 1.2.41 but your feedback would be very valuable. Thanks! Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk causing stuck Apache processes
Hi Jesse, Am 24.02.2015 um 19:27 schrieb Jesse Defer: -Original Message- From: Rainer Jung [mailto:rainer.j...@kippdata.de] Sent: Monday, February 23, 2015 5:00 PM To: Tomcat Users List Subject: Re: mod_jk causing stuck Apache processes Am 24.02.2015 um 00:29 schrieb Jesse Defer: I suggest: - check whether there's a performance problem on the webapp side when it happens next time - check thread dumps - check GC log No long pauses that I can see. - activate Tomcat access log and add %D to the pattern, so that you can analyze request duration on the Tomcat side post-mortem - using the request duration logs, you can also check, whether a potential performance problem happens on all nodes or only on some. Depending on the result, the GC log check and thread dump creation and check would be easier or harder (more work). - run strace with time stamps plus system call duration next time the problem occurs http://pastebin.com/raw.php?i=X09ZTW4V - also add %D to the LogFormat of your Apache access log. Do also add the name of the load balancer member that handled the request to the access log. Mod_jk provides notes, that can be logged in the access log (look for JK_LB_LAST_NAME in http://tomcat.apache.org/connectors-doc/reference/apache.html) - prepare switching to a threaded MPM, be it worker or event. Use JkWatchdogInterval there. - put JkShmFile to a local file system - check whether your system clock on the Apache servers works correctly and doesn't jump (common virtualization problem) NTP is configured on all hosts and we monitor drift with collectd every 10 seconds, no anomalies. - configure the jk status worker and check, whether many of your workers are in ERROR state. You can also retrieve info from the JK status worker in a simple to parse text format and automate retrieval and extraction of data (similar to what one can do with the Apache server-status) I haven't seen the status worker show any workers in error. Here are some logs with request duration. First entry in each pair is httpd, second is tomcat. It is interesting that the tomcat response is very fast but in two of the cases the apache response is very slow. I wasn't able to correlate the JVM stack traces for these, maybe because they executed quickly and weren't blocked. 10.142.64.251 - - [24/Feb/2015:10:34:37 -0700] GET /my/ HTTP/1.1 200 72209 - Mozilla/5.0 pid=25649 tid=47017797431248 time_s=1 time_us=1174645 x_id=VOy2LYHbdfEAAGQxXQkAAAC0 content_type=text/html last_modified=- cache_control=no-cache worker=mylb first_name=my27 first_busy=0 last_name=my27 last_busy=0 ssl_protocol=TLSv1 ssl_cipher=AES256-SHA 10.142.64.251 - - [24/Feb/2015:10:34:37 -0700] GET /my/ HTTP/1.1 200 72209 - Mozilla/5.0 time_ms=115 tomcat_thread=ajp-62014-31 x_id=VOy2LYHbdfEAAGQxXQkAAAC0 app=my 10.142.64.251 - - [24/Feb/2015:10:31:02 -0700] GET /my/ HTTP/1.1 200 8184 - Mozilla/5.0 pid=25697 tid=47017797431248 time_s=246 time_us=246422721 x_id=VOy1VoHbdfEAAGRhuqQAAADq content_type=text/html last_modified=- cache_control=- worker=mylb first_name=my27 first_busy=0 last_name=my27 last_busy=0 ssl_protocol=TLSv1 ssl_cipher=AES256-SHA 10.142.64.251 - - [24/Feb/2015:10:35:09 -0700] GET /my/ HTTP/1.1 200 72209 - Mozilla/5.0 time_ms=53 tomcat_thread=ajp-62014-32 x_id=VOy1VoHbdfEAAGRhuqQAAADq app=my 10.142.64.251 - - [24/Feb/2015:10:30:41 -0700] GET /my/ HTTP/1.1 200 8184 - Mozilla/5.0 pid=25772 tid=47017797431248 time_s=371 time_us=371698757 x_id=VOy1QYHbdfEAAGSsv-oAAAEy content_type=text/html last_modified=- cache_control=- worker=mylb first_name=my27 first_busy=0 last_name=my27 last_busy=0 ssl_protocol=TLSv1 ssl_cipher=AES256-SHA 10.142.64.251 - - [24/Feb/2015:10:36:52 -0700] GET /my/ HTTP/1.1 200 72209 - Mozilla/5.0 time_ms=57 tomcat_thread=ajp-62014-5 x_id=VOy1QYHbdfEAAGSsv-oAAAEy app=my I let one server stay busy for about an hour to collect some logs. There are some slow requests for this app in the tomcat logs, but few over 60 seconds and only 1 over 120 seconds (for all 17 of this app's tomcats). Meanwhile there are thousands of httpd requests over 120s from just the one web server with the stuck processes. Plotting the request time for the entire httpd farm shows the affected server has a large jump but no change in the other servers. So here is a patch for version 1.2.40: http://people.apache.org/~rjung/patches/jk_shm_lock.patch The patch is non-trivial though. It reduces the load on the lock by - moving some trivial stuff done for each lb member in a separate lock/unlock block (per member) to the lb itself, using its own lock/unlock block. That doesn't reduce the total time locks are held, but the number of lock/unlock ops. - using a different mechanism to check, whether the global maintenance should actually run (based on atomics and a global decision instead of a per lb decision). That should be more efficient and again reduces lock/unlock ops, because the check can now be done
Re: mod_jk causing stuck Apache processes
Am 23.02.2015 um 19:03 schrieb Jesse Defer:
I have a farm of Apache httpd servers proxying to Tomcat with mod_jk and I am
having issues with Apache processes getting stuck (as seen by the W state in
server-status). I am sending to this list because the stack traces show httpd
gets stuck in mod_jk.
httpd is configured for prefork and with 512 servers on start and maximum.
When the problem occurs we end up with nearly all 512 processes in the W state
until we restart it. The problem occurs more often when load is high but is
not restricted to high load. The problem started occuring more often when we
increased the servers from 384 to 512. The hosts have enough memory and do not
swap. The issue occurs intermitently and is not tied to a particular host or
instance Tomcat (there are ~150 Tomcat instances). The JkShmFile is on tmpfs.
Why on tmpfs?
Environment: RHEL5.11, Apache 2.4.10 (prefork), JK 1.2.40, APR 1.5.1, APR-UTIL
1.5.4
The stuck httpd processes all show the same stack and strace:
pstack:
#0 0x2b3439026bff in fcntl () from /lib64/libpthread.so.0
#1 0x2b3440911656 in jk_shm_lock () from
/usr/local/apache2/modules/mod_jk.so
#2 0x2b3440917805 in ajp_maintain () from
/usr/local/apache2/modules/mod_jk.so
#3 0x2b3440906cac in maintain_workers () from
/usr/local/apache2/modules/mod_jk.so
#4 0x2b3440901140 in wc_maintain () from
/usr/local/apache2/modules/mod_jk.so
#5 0x2b34408f40b6 in jk_handler () from
/usr/local/apache2/modules/mod_jk.so
#6 0x00448eca in ap_run_handler ()
#7 0x0044cc92 in ap_invoke_handler ()
#8 0x0045e24f in ap_process_async_request ()
#9 0x0045e38f in ap_process_request ()
#10 0x0045ab65 in ap_process_http_connection ()
#11 0x004530ba in ap_run_process_connection ()
#12 0x0046423a in child_main ()
#13 0x00464544 in make_child ()
#14 0x004649ae in prefork_run ()
#15 0x00430634 in ap_run_mpm ()
#16 0x0042ad97 in main ()
So mod_jk tries to get a lock on the shared memory before reading and
updating some shared memory data. That as one of many things mod_jk does
is normal. It would be not normal, if most processes seem to almost
always sit in this stack.
strace:
fcntl(19, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=1}) = 0
fcntl(19, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=1}) = 0
time(NULL) = 1424711498
fcntl(19, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=1}) = 0
fcntl(19, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=1}) = 0
fcntl(19, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=1}) = 0
fcntl(19, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=1}) = 0
fcntl(19, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=1}) = 0
fcntl(19, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=1}) = 0
Any help tracking this issue down would be appreciated.
This doesn't look like mod_jk is hanging in the jk_shm_lock() but
instead it looks like normal processing, locking and then unlocking.
Locking and unlocking suceeds (return code 0).
You didn't provide time stamps to get an idea, whether it is normal
behavior or not. What is your request throughput (requests per second
forwarded by mod_jk as long as it is running well)?
I suspect something else is wrong. Have you checked, whether in fact
mod_jk waits for the response from requests it has send to the backend
and which do not return quickly, e.g. looking at a Java thread dump
(not: heap dump) of the backend during the times you have problems?
What are the error or warn messages in your mod_jk log file?
Did you use apachectl graceful shortly before the problems start, or
change configuration via the mod_jk status worker?
What is special here, is the use of many processes plus many workers. So
the lock is used quite a lot. Still the global maintain functionality
which uses the jk_shm_lock() in the stack above should be called by each
process only every worker.maintain seconds, by default every 60 seconds.
And each project should quickly detect whether another process already
did the global maintain. During the global maintain, for any ajp13
worker there is really just a few simply local code statements. For any
lb (load balancer) worker, there is a little more to do, especialy
checking whether any members have failed long ago and should now be
recovered. ut still those operations are all local, not going on the
network.
What is your workr struture? Are the 150 workers part of few load
balancers or are they all in the worker.list? Can you show significant
parts of your workers.properties? Are you using the non-default
pessimistic locking?
If your Apache uses a threaded APR library, you could try using
JkWatchdogInterval. That moved the maintenance from the normal request
processing code to a separate thread per Apache process.
Regards,
Rainer
Re: mod_jk causing stuck Apache processes
Am 23.02.2015 um 20:34 schrieb Rainer Jung:
Am 23.02.2015 um 19:03 schrieb Jesse Defer:
I have a farm of Apache httpd servers proxying to Tomcat with mod_jk
and I am having issues with Apache processes getting stuck (as seen by
the W state in server-status). I am sending to this list because the
stack traces show httpd gets stuck in mod_jk.
httpd is configured for prefork and with 512 servers on start and
maximum. When the problem occurs we end up with nearly all 512
processes in the W state until we restart it. The problem occurs more
often when load is high but is not restricted to high load. The
problem started occuring more often when we increased the servers from
384 to 512. The hosts have enough memory and do not swap. The issue
occurs intermitently and is not tied to a particular host or instance
Tomcat (there are ~150 Tomcat instances). The JkShmFile is on tmpfs.
Why on tmpfs?
Environment: RHEL5.11, Apache 2.4.10 (prefork), JK 1.2.40, APR 1.5.1,
APR-UTIL 1.5.4
The stuck httpd processes all show the same stack and strace:
pstack:
#0 0x2b3439026bff in fcntl () from /lib64/libpthread.so.0
#1 0x2b3440911656 in jk_shm_lock () from
/usr/local/apache2/modules/mod_jk.so
#2 0x2b3440917805 in ajp_maintain () from
/usr/local/apache2/modules/mod_jk.so
#3 0x2b3440906cac in maintain_workers () from
/usr/local/apache2/modules/mod_jk.so
#4 0x2b3440901140 in wc_maintain () from
/usr/local/apache2/modules/mod_jk.so
#5 0x2b34408f40b6 in jk_handler () from
/usr/local/apache2/modules/mod_jk.so
#6 0x00448eca in ap_run_handler ()
#7 0x0044cc92 in ap_invoke_handler ()
#8 0x0045e24f in ap_process_async_request ()
#9 0x0045e38f in ap_process_request ()
#10 0x0045ab65 in ap_process_http_connection ()
#11 0x004530ba in ap_run_process_connection ()
#12 0x0046423a in child_main ()
#13 0x00464544 in make_child ()
#14 0x004649ae in prefork_run ()
#15 0x00430634 in ap_run_mpm ()
#16 0x0042ad97 in main ()
So mod_jk tries to get a lock on the shared memory before reading and
updating some shared memory data. That as one of many things mod_jk does
is normal. It would be not normal, if most processes seem to almost
always sit in this stack.
strace:
fcntl(19, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=1}) = 0
fcntl(19, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=1}) = 0
time(NULL) = 1424711498
fcntl(19, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=1}) = 0
fcntl(19, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=1}) = 0
fcntl(19, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=1}) = 0
fcntl(19, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=1}) = 0
fcntl(19, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=1}) = 0
fcntl(19, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=1}) = 0
Any help tracking this issue down would be appreciated.
This doesn't look like mod_jk is hanging in the jk_shm_lock() but
instead it looks like normal processing, locking and then unlocking.
Locking and unlocking suceeds (return code 0).
You didn't provide time stamps to get an idea, whether it is normal
behavior or not. What is your request throughput (requests per second
forwarded by mod_jk as long as it is running well)?
I suspect something else is wrong. Have you checked, whether in fact
mod_jk waits for the response from requests it has send to the backend
and which do not return quickly, e.g. looking at a Java thread dump
(not: heap dump) of the backend during the times you have problems?
What are the error or warn messages in your mod_jk log file?
Did you use apachectl graceful shortly before the problems start, or
change configuration via the mod_jk status worker?
What is special here, is the use of many processes plus many workers. So
the lock is used quite a lot. Still the global maintain functionality
which uses the jk_shm_lock() in the stack above should be called by each
process only every worker.maintain seconds, by default every 60 seconds.
And each project should quickly detect whether another process already
did the global maintain. During the global maintain, for any ajp13
worker there is really just a few simply local code statements. For any
lb (load balancer) worker, there is a little more to do, especialy
checking whether any members have failed long ago and should now be
recovered. ut still those operations are all local, not going on the
network.
What is your workr struture? Are the 150 workers part of few load
balancers or are they all in the worker.list? Can you show significant
parts of your workers.properties? Are you using the non-default
pessimistic locking?
If your Apache uses a threaded APR library, you could try using
JkWatchdogInterval. That moved the maintenance from the normal request
processing code to a separate thread per Apache process.
Two more things:
- are you running
Re: mod_jk causing stuck Apache processes
Am 23.02.2015 um 22:53 schrieb Jesse Defer:
-Original Message-
Am 23.02.2015 um 19:03 schrieb Jesse Defer:
I have a farm of Apache httpd servers proxying to Tomcat with mod_jk and
I am having issues with Apache processes getting stuck (as seen by the W
state in server-status). I am sending to this list because the stack
traces show httpd gets stuck in mod_jk.
httpd is configured for prefork and with 512 servers on start and
maximum. When the problem occurs we end up with nearly all 512 processes
in the W state until we restart it. The problem occurs more often when
load is high but is not restricted to high load. The problem started
occuring more often when we increased the servers from 384 to 512. The
hosts have enough memory and do not swap. The issue occurs intermitently
and is not tied to a particular host or instance Tomcat (there are ~150
Tomcat instances). The JkShmFile is on tmpfs.
Why on tmpfs?
Not sure, might be because of poor local IO performance (hosts are VMs) or
historic use of network filesystems.
I haven't checked the details, but I don't expect the shared memory
activity to be reflected by FS activity. I don't have an indication,
that tmpfs is a problem for the locking either, but once we run out of
ideas, you could try whether moving away from tmpfs and network
filesystems for the JkShmFile and instead using a path to a local file
for JkShmFile helps.
Environment: RHEL5.11, Apache 2.4.10 (prefork), JK 1.2.40, APR 1.5.1,
APR-UTIL 1.5.4
The stuck httpd processes all show the same stack and strace:
pstack:
#0 0x2b3439026bff in fcntl () from /lib64/libpthread.so.0
#1 0x2b3440911656 in jk_shm_lock () from
/usr/local/apache2/modules/mod_jk.so
#2 0x2b3440917805 in ajp_maintain () from
/usr/local/apache2/modules/mod_jk.so
#3 0x2b3440906cac in maintain_workers () from
/usr/local/apache2/modules/mod_jk.so
#4 0x2b3440901140 in wc_maintain () from
/usr/local/apache2/modules/mod_jk.so
#5 0x2b34408f40b6 in jk_handler () from
/usr/local/apache2/modules/mod_jk.so
#6 0x00448eca in ap_run_handler ()
#7 0x0044cc92 in ap_invoke_handler ()
#8 0x0045e24f in ap_process_async_request ()
#9 0x0045e38f in ap_process_request ()
#10 0x0045ab65 in ap_process_http_connection ()
#11 0x004530ba in ap_run_process_connection ()
#12 0x0046423a in child_main ()
#13 0x00464544 in make_child ()
#14 0x004649ae in prefork_run ()
#15 0x00430634 in ap_run_mpm ()
#16 0x0042ad97 in main ()
So mod_jk tries to get a lock on the shared memory before reading and
updating some shared memory data. That as one of many things mod_jk does
is normal. It would be not normal, if most processes seem to almost always
sit in this stack.
strace:
fcntl(19, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=1}) =
0 fcntl(19, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=1}) =
0
time(NULL) = 1424711498
fcntl(19, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=1}) =
0 fcntl(19, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=1})
= 0 fcntl(19, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0,
len=1}) = 0 fcntl(19, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET,
start=0, len=1}) = 0 fcntl(19, F_SETLKW, {type=F_WRLCK,
whence=SEEK_SET, start=0, len=1}) = 0 fcntl(19, F_SETLKW,
{type=F_UNLCK, whence=SEEK_SET, start=0, len=1}) = 0
Any help tracking this issue down would be appreciated.
This doesn't look like mod_jk is hanging in the jk_shm_lock() but instead
it looks like normal processing, locking and then unlocking.
Locking and unlocking suceeds (return code 0).
You didn't provide time stamps to get an idea, whether it is normal
behavior or not. What is your request throughput (requests per second
forwarded by mod_jk as long as it is running well)?
Next time it happens I will capture timestamps, but it scrolls pretty
quickly and doesn't seem to get hung up on any calls. We have very seasonal
traffic, but normally around 30/sec. The high period peaks at around 200/sec.
OK, strace should be able to print time stamps plus the time that the
call took.
Load around 30/s up to 200/s is not extreme.
I suspect something else is wrong. Have you checked, whether in fact
mod_jk waits for the response from requests it has send to the backend and
which do not return quickly, e.g. looking at a Java thread dump
(not: heap dump) of the backend during the times you have problems?
I haven't, next time it happens I'll look.
What are the error or warn messages in your mod_jk log file?
Here is one instance of each type of warn/error in the logs of an affected host.
All of these are fairly common but we have tomcats going up and down all the
time and I think most are related to that. Log level is info.
[Mon Feb 23 03:05:37 2015] [25833:47348544151936] [error]
ajp_service::jk_ajp_common.c (2693): (my27) connecting to tomcat failed.
[Mon Feb
Re: mod_jk causing stuck Apache processes
Am 24.02.2015 um 00:29 schrieb Jesse Defer: -Original Message- Am 23.02.2015 um 22:53 schrieb Jesse Defer: -Original Message- Am 23.02.2015 um 19:03 schrieb Jesse Defer: ... How many Apache web servers sit in front of the Tomcats? Is each of the Apaches configured with 512 processes? All Tomcats are configured the same, maxThreads is 4096, apr, Tomcat 6. 16 apaches, all configured identically with 512 processes, the same worker.properties, etc. OK. So less threads than 16*512. But the apr connector - like the nio one - can handle more connections than it has threads. Not more in-flight requests, but for both connectors idle connections (keep-alive connections) are cheap and do not need a pool thread. Our busiest Tomcats peak at about 2000 connections to their AJP port from the web farm. I'm not sure how closely connections relate to thread usage, I haven't looked too closely at the JVM yet. For BIO one connection one thread. For APR and NIO it's roughly one in-flight request one thread. Connections waiting for the next request to arrive are cheap and handled by few separate poller threads. ... And only the lb is added to the worker.list? Or also the individual AJP workers? Just the LBs. OK If your Apache uses a threaded APR library, you could try using JkWatchdogInterval. That moved the maintenance from the normal request processing code to a separate thread per Apache process. I suspect that once your app (one or some of your apps) gets slow, requests pile up in Apache and Apache can no longer connect to Tomcat, because it doesn't have free threads. Some Apache children still can connect, because the connect is handled by the OS underneath Tomcat, but those connections do not get the awaited cping reply. Shouldn't the ping timeout cause the processes to exit the wait state if the cping reply hasn't been received? I have seen the SS column in server-status grow into the thousands. Yes it would. But it could be a mixture: lots of Apache processes waiting for hanging or slow Tomcat responses due to an aplication or GC bottleneck, and some on-top requests failing in cping. In addition to the Thread Dumps, you should also activate GC-Logs for your Tomcat JVMs. I have gc logging enabled already. ACK - using the worker or prefork MPM in combination with JkWatchdogInterval would reduce the amount of shm locking due to global maintenance calls, because it is done per process, not per thread. You might have other reasons for using prefork though. Have you? Use of prefork is historical. We have been looking at going to event, but Haven't made the switch yet. If you are using many Apaches each with 512 children, you will most likely need to use an NIO connector on the Tomcat processes and would also benefit from worker or event MPM for Apache. Each process e.g. with 50 or 100 threads and a relatively short connection_pool_timeout like e.g. 60 seconds. I am going to bring up accelerating the switch to the event MPM with my team. Will NIO give us more scalability than APR? Not really, both are good at connection scalability. Maybe don't change to much at the same time. I'd say NIO is a bit more mainstream than APR, but your mileage my vary. What was the reason, that you started suspecting the fcntl() underneath jk_shm_lock? Are we sure, that the stack you provided is really the typical one? And was the strace dumping all system calls or only the fcntl calls? I can't be sure it's the typical one, I only know every time I sample the stack of a stuck process that's what I get, plus the strace shows no other system calls except time(). It seems like it's stuck in a fairly tight loop somewhere. I haven't looked too deeply in the mod_jk code though and haven't collected any system call statistics. I don't necessarily suspect fcntl, but it's getting called a lot (the strace is not filtered). System time jumps when the issue is happening and the fcntl looks like the vast majority of system calls. Since the issue got worse when we added more processes it seems plausible it could be lock contention. Hmm, but lock contention should result in the fcntl call to take more and more time. Could it be a problem with the system time? That somehow the don't run the maintenance for the next 60 seconds rule doesn't get effective because the clock jumps? I suggest: - check whether there's a performance problem on the webapp side when it happens next time - check thread dumps - check GC log - activate Tomcat access log and add %D to the pattern, so that you can analyze request duration on the Tomcat side post-mortem - using the request duration logs, you can also check, whether a potential performance problem happens on all nodes or only on some. Depending on the result, the GC log check and thread dump creation and check would be easier or harder (more work). - run strace with time stamps plus system call duration next time the
Re: AccessLogValve - add a suffix to log file on rotation
Am 22.02.2015 um 17:12 schrieb Campbell, Lance: Tomcat 8.x Valve: org.apache.catalina.valves.AccessLogValve When all of my log files rotate I like to add the suffix “.old”. This is so that if there is an issue with file sizes I can very quickly delete all old log files by doing: rm *.old Using the above AccessLogValve how can I append “.old” to the end of the file name when it rotates? Try: renameOnRotate=true fileDateFormat=.-MM-dd.old and no suffix attribute. See: http://tomcat.apache.org/tomcat-8.0-doc/config/valve.html#Access_Log_Valve Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Issue with RewriteValve and folders (tomcat 8.0.15)
Am 19.02.2015 um 22:13 schrieb Felix Schumacher:
Am 19.02.2015 um 21:41 schrieb André Warnier:
Jérémie Barthés wrote:
...
Make a file rewrite.config in conf/Catalina/localhost/ that contains :
RewriteRule^/mypath/(.*)$/examples/jsp/$1
copy the line
Valve
className=org.apache.catalina.valves.rewrite.RewriteValve /
in the conf/server.xml file, line 131
Since this is a Valve, it will run before Tomcat attempts to match the
URL to an actual directory or webapp.
try the followings URLs :
1) http://localhost:8080/mypath/async
This matches the rewrite rule, so it will be rewritten to URL
/examples/jsp/async
Then Tomcat will attempt to match this to a directory or webapp, and
find that (catalina_base)/webapps/examples/jsp/async is a directory.
It will thus respond to the browser with a 302 re-direct to
http://localhost:8080/examples/jsp/async/
which is actually the correct URL.
And this is what will be shown in the browser URL bar. This, in my
view, is expected behaviour. The server does that, so that when an
actual response is generated (for the correct URL
http://localhost:8080/examples/jsp/async/), the browser can cache this
response under the correct URL.
Then the browser re-issues a request for
http://localhost:8080/examples/jsp/async/
and that is when Tomcat will actually generate a real response,
because this time it is a correct URL. So the response appears to the
browser, as coming from
http://localhost:8080/examples/jsp/async/
which is correct.
2) http://localhost:8080/mypath/async/
This also matches the rewrite rule, so it gets rewritten to
http://localhost:8080/examples/jsp/async/
which is a correct URL.
Thus Tomcat will immediately generate a real response (without an
intermediate 302 redirect), which will be appear in the browser URL
bar as a response to
http://localhost:8080/mypath/async/
This is also expected behaviour.
I believe that if you do not want to see the first redirect URL
http://localhost:8080/examples/jsp/async/
in the browser, you have to modify your rewrite rules, perhaps by
using a RewriteCond with the -d flag, to check first if the URL points
to an existing directory, and if yes add the terminating / yourself
(with a RewriteRule) before other rewrite tests/rules take place.
This rewrite.config
...
will do the trick. I think -d will not work, since /mypath/async is not
existant, it only feels like a directory.
Not clear: the implementation for -d is (case 0 below, from file
ResolverImpl.java):
148 @Override
149 public boolean resolveResource(int type, String name) {
150 WebResourceRoot resources = request.getContext().getResources();
151 WebResource resource = resources.getResource(name);
152 if (!resource.exists()) {
153 return false;
154 } else {
155 switch (type) {
156 case 0:
157 return (resource.isDirectory());
158 case 1:
159 return (resource.isFile());
160 case 2:
161 return (resource.isFile()
resource.getContentLength() 0);
162 default:
163 return false;
164 }
165 }
166 }
Since it checks resources and the OP was actually talking about a path
that is a folder in his webapp, -d could work.
Regards,
Rainer
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Re-enabling SSLv3
Am 18.02.2015 um 10:11 schrieb Mark Thomas: On 18/02/2015 09:04, André Warnier wrote: snip/ Also, while we are at it : this whole SSL area is - I believe - hopelessly confusing for anyone (aka me) who does not spend a considerable amount of time dealing with that kind of setup. Do you know of any reasonably short and concise introductory article on the www ? Something which really explains the basics (and the why's) of what you need to set up a HTTPS webserver, be it Tomcat or something else ? I believe that some kind of wrap-up article in the FAQ would really help, and you seem to be our resident expert here. Maybe this? https://www.feistyduck.com/books/bulletproof-ssl-and-tls/ You can download a PDF of the first chapter that explains a lot of the basics for free. I'm in the middle of reading the whole book and it is really a good read. But there's quite a lot of terminology to get used to for beginners, so probably not easy to write a short but useful SSL primer in the Tomcat wiki. Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat Access Logging Valve ?
Am 14.02.2015 um 18:44 schrieb Jeff Kohut: Sent from Google Nexus Phone On Feb 14, 2015 9:54 AM, Rainer Jung rainer.j...@kippdata.de wrote: Am 14.02.2015 um 12:46 schrieb Jeff Kohut: On Fri, Feb 13, 2015 at 10:29 AM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Jeff, On 2/13/15 10:28 AM, Jeff Kohut wrote: I am running Tomcat 7.0.54 on a Windows 2008 R1 with SP1 platform. I would like to control the contents of what gets logged to the Tomcat localhost_access_log specifically , I would like to remove logging of entries like : 10.239.54.8 - - [13/Feb/2015:00:00:07 -0600] GET /atb HTTP/1.0 200 573 10.239.58.29 - - [13/Feb/2015:00:00:08 -0600] GET /atb HTTP/1.0 200 561 we have some external load balancers and server monitoring software periodically hitting the application web page to see if it is responding. I would like to know how remove these entries to reduce the amount of noise that is in the localhost_access_log log file, but allow other entries (i.e. from OTHER IP addresses or with other functions like POST or other strings like HTTP/1.1 I have examined the information at the below link (unfortunately there is no example there with detail on what I am attempting to do). http://tomcat.apache.org/tomcat-7.0-doc/config/valve.html Seems like I want to use the valve Access Log Valve Attributes: conditionIf or conditionUnless options from valve documentation, but it is not clear to me how to do this (if this is the correct way ?). Yep, this is what you are looking for. Server.xml currently has the following setup configuration: !-- Access log processes all example. Documentation at: /docs/config/valve.html Note: The pattern used is equivalent to using pattern=common -- Valve className=org.apache.catalina.valves.AccessLogValve directory=logs prefix=localhost_access_log. suffix=.txt pattern=%h %l %u %t quot;%rquot; %s %b / Is it possible to do this without programming (i.e. can I set the valve components to simply filter on an ip address or key word string (i.e. Get /atb HTTP/1.0 and if that value is found in the request, can I chose to then NOT write that entry to the AccessLogValve). I have searched pretty thoroughly for some time on how to do this, but the documentation simply does not provide enough of an example on how to do this. I am not a programmer, but would think that filtering entries out of logs based on criteria would be a valuable feature to have. You have to have a way to set (or not) a request attribute in order to use conditionIf/conditionUnless. One way to do it without writing any of your own code is to use url-rewrite, which is more like url-do-whatever: http://tuckey.org/urlrewrite/ You can configure that to set request attributes under certain criteria. Just have it set (or not) whatever attribute name you use for your conditional logging and you should be good to go. Hope that helps, - -chris Chris, Looks a bit complicated (but then I guess life is a bit complicated ;-) I was hoping for something built in. In looking at the explanation and a little at the support websites, it looks as if this is something to rewrite what makes it to the actual webapps applications. I did not see where it was used to be able to filter out the main tomcat access logs (i.e being able to control something so early as the Request and built in Tomcat logging), but I guess it may be possible to do that. I will look into this further if I have the time. As Andre pointed out in a follow up post, I could just filter the access log into another file externally and do my cleanup that way. Thank you for taking the time to answer and provide something to look into further. I basically agree with André. And the URL rewrite filter is a nice and powerful tool that's useful to learn about, although it can't be easily integrated into a webapp from the outside (not moving it into the webapp itself). If it happens that between your load balancer and Tomcat there is already an Apache web server and the forwarding from Apache to Tomcat is done by mod_jk, then you can set Tomcat request attributes from mod_jk by using JkSetEnv. You would use a RewriteRule probably with a RewriteCond to set an Apache environment variable for the requests that come from the LB, then use JkEnvVar to forward that variable, then use the conditionUnless for the Tomcat access log valve to suppress those requests. Using a seconds AccessLogValve with conditionIf would let you log those requests to a second log file. I'm not saying that I would inject a new Apache/mod_jk into the picture just for that feature. But if it were already there ... Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org All good thoughts, but no Apache. If it can't be done in Tomcat, I will likely
Re: Tomcat Access Logging Valve ?
Am 14.02.2015 um 12:46 schrieb Jeff Kohut: On Fri, Feb 13, 2015 at 10:29 AM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Jeff, On 2/13/15 10:28 AM, Jeff Kohut wrote: I am running Tomcat 7.0.54 on a Windows 2008 R1 with SP1 platform. I would like to control the contents of what gets logged to the Tomcat localhost_access_log specifically , I would like to remove logging of entries like : 10.239.54.8 - - [13/Feb/2015:00:00:07 -0600] GET /atb HTTP/1.0 200 573 10.239.58.29 - - [13/Feb/2015:00:00:08 -0600] GET /atb HTTP/1.0 200 561 we have some external load balancers and server monitoring software periodically hitting the application web page to see if it is responding. I would like to know how remove these entries to reduce the amount of noise that is in the localhost_access_log log file, but allow other entries (i.e. from OTHER IP addresses or with other functions like POST or other strings like HTTP/1.1 I have examined the information at the below link (unfortunately there is no example there with detail on what I am attempting to do). http://tomcat.apache.org/tomcat-7.0-doc/config/valve.html Seems like I want to use the valve Access Log Valve Attributes: conditionIf or conditionUnless options from valve documentation, but it is not clear to me how to do this (if this is the correct way ?). Yep, this is what you are looking for. Server.xml currently has the following setup configuration: !-- Access log processes all example. Documentation at: /docs/config/valve.html Note: The pattern used is equivalent to using pattern=common -- Valve className=org.apache.catalina.valves.AccessLogValve directory=logs prefix=localhost_access_log. suffix=.txt pattern=%h %l %u %t quot;%rquot; %s %b / Is it possible to do this without programming (i.e. can I set the valve components to simply filter on an ip address or key word string (i.e. Get /atb HTTP/1.0 and if that value is found in the request, can I chose to then NOT write that entry to the AccessLogValve). I have searched pretty thoroughly for some time on how to do this, but the documentation simply does not provide enough of an example on how to do this. I am not a programmer, but would think that filtering entries out of logs based on criteria would be a valuable feature to have. You have to have a way to set (or not) a request attribute in order to use conditionIf/conditionUnless. One way to do it without writing any of your own code is to use url-rewrite, which is more like url-do-whatever: http://tuckey.org/urlrewrite/ You can configure that to set request attributes under certain criteria. Just have it set (or not) whatever attribute name you use for your conditional logging and you should be good to go. Hope that helps, - -chris Chris, Looks a bit complicated (but then I guess life is a bit complicated ;-) I was hoping for something built in. In looking at the explanation and a little at the support websites, it looks as if this is something to rewrite what makes it to the actual webapps applications. I did not see where it was used to be able to filter out the main tomcat access logs (i.e being able to control something so early as the Request and built in Tomcat logging), but I guess it may be possible to do that. I will look into this further if I have the time. As Andre pointed out in a follow up post, I could just filter the access log into another file externally and do my cleanup that way. Thank you for taking the time to answer and provide something to look into further. I basically agree with André. And the URL rewrite filter is a nice and powerful tool that's useful to learn about, although it can't be easily integrated into a webapp from the outside (not moving it into the webapp itself). If it happens that between your load balancer and Tomcat there is already an Apache web server and the forwarding from Apache to Tomcat is done by mod_jk, then you can set Tomcat request attributes from mod_jk by using JkSetEnv. You would use a RewriteRule probably with a RewriteCond to set an Apache environment variable for the requests that come from the LB, then use JkEnvVar to forward that variable, then use the conditionUnless for the Tomcat access log valve to suppress those requests. Using a seconds AccessLogValve with conditionIf would let you log those requests to a second log file. I'm not saying that I would inject a new Apache/mod_jk into the picture just for that feature. But if it were already there ... Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk and session stickiness
Am 30.01.2015 um 03:06 schrieb Christopher Schultz: On 7/23/14 5:14 PM, Christopher Schultz wrote: Rainer, On 7/23/14, 4:10 PM, Rainer Jung wrote: Am 23.07.2014 um 21:03 schrieb Christopher Schultz: On 7/23/14, 1:49 PM, Rainer Jung wrote: The other case, a request with an invalid session ID accessing a tomcat instance with activation disabled can IMHO be handled by a filter that - checks whether the request has a valid session using getSession(false), if it has one, let the request proceed - checks activation state, if active, let the request proceed - checks the request method, if not GET, let the request proceed - otherwise: - set the session cookie, e.g. JSESSIONID the an empty value - issue an external redirect to the same request URL - optional redirect loop detection: add a query string param or cookie that gets the local jvmRoute appended during each redirect. Before doing the redirect, check that the local jvmRoute is not already part of that token (we have already been here before) This would not really interfere with your saved requests: they would get a redirect which the browser follwos automatically and after that you will observe the normal behavior. This is exactly what I have implemented -- as a Filter since we can insert it before SecurityFilter intercepts the requests -- and my tests suggest that it will work correctly. I added code to strip-out any ;jsessionid path parameter from the URL ACK if it exists, but haven't done any of the redirect loop detection (yet). I think the loop detection is going to have to keep a running list of visited nodes which, in large clusters, could grow very large especially if the node names are long. I'll post my code when it's a little more featureful. As long as nothing goes wrong, the first redirect - having no more session info - should not end up in getting redirected as well, because it should only be send to an active node by mod_jk. So you can even stop redirecting if you detect, that it already happened once. For a large cluster that might be better. Multiple might happen (didn't check the code), if all active members are in error state. Even then one would not like to produce many of those redirects for one request, so again, only allowing one redirect might be the right solution. That's a good thought, and simplifies things. The request parameter could then just be a boolean flag and not actually identify the node that produced the redirect: any node considering redirecting simply would not redirect if that flag had been set. So, I've implemented all of the above and just given it a real test-drive in production. It seems to work well on the first shot. Good! I'll be testing it a bit more as time does on, actually. I also converted it to a Valve that I will propose to include in Tomcat once I get some real mileage on it. Was a Valve more useful than a filter? Did you need to access Tomcat internals? I also discovered that I'd like to have a new feature: the ability for a whitelisted client to ignore the re-balance rules and continue to contact the server. So, if you have a cluster member called, say, foo and you want to force your way into foo, you can make a request to any page with a jsessionid parameter: http://host/some_page;jsessionid=1234.foo mod_jk will send you to the foo node, even though it is in the DISABLED state. Normally, my Filter/Valve would see that you have no valid session (1234.foo is not valid) and perform a redirect without the jsessionid path parameter added, thus re-balancing you to another node. Instead, I've added the ability to configure an ignore cookie and cookie value (like ignore=SECRETCODE). If the client presents this cookie to the Filter/Valve, then it's business as usual. This allows a trusted user to log into the DISABLED node through the load-balancer to take a look around, make sure that things are working properly, etc. before re-enabling the node in the cluster. I'll give this a shot in the next few days, clean it all up, and publish it (soon?). That sounds useful. Thanks! Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Problem with mod_jk JKStatus: where is the worker state property?
Hi Lorenzo, Am 30.01.2015 um 12:22 schrieb Lorenzo Maurizi - C.S.I.A. UniMC: Dear All, I need help on an issue I'm facing. I've tried to find an answer with the online documentation but I didn't succeed. I am trying to fix a munin plugin for monitoring the mod_jk ajp13 connector for apache. The mod_jk is version 1.2.37 (the default version for the Debian Wheezy package). The munin plugin reads the http://localhost/jk-status?mime=prop output and searches for the worker.workername.state value. But my jk-status mime=prop output does not contains that state row, while the State is correctly shown on HTML output of the jk-status. I looked at the changelog of mod-jk trying to find something about a change in the output of JKStatus, without success. Is it a bug of mod_jk ver. 1.2.37? state is only printed of the worker is a member of a load balancer. Otherwise there is no such thing as state. It is good practise to wrap even a single worker in a load balancer because of its enhanced failure detection and reporting. Even if you don't have a second worker to fail over. HTH Rainer Output of my jk-status properties (I changed some values to anonymize it): 8- worker.server_name=193.x.x.x worker.server_port=80 worker.time_datetime=20150130121721 worker.time_tz=CET worker.time_unix=1422616641 worker.web_server=Apache/2.2.22 (Debian) mod_jk/1.2.37 mod_ssl/2.2.22 OpenSSL/1.0.1e worker.jk_version=mod_jk/1.2.37 worker.ajp_count=1 worker.ajp13.list=ajp13 worker.ajp13.type=ajp13 worker.ajp13.host=localhost worker.ajp13.port=8009 worker.ajp13.address=127.0.0.1:8009 worker.ajp13.connection_pool_timeout=1300 worker.ajp13.ping_timeout=1 worker.ajp13.connect_timeout=0 worker.ajp13.prepost_timeout=0 worker.ajp13.reply_timeout=0 worker.ajp13.retries=2 worker.ajp13.connection_ping_interval=0 worker.ajp13.recovery_options=0 worker.ajp13.max_packet_size=8192 worker.ajp13.used=174 worker.ajp13.errors=0 worker.ajp13.client_errors=0 worker.ajp13.reply_timeouts=0 worker.ajp13.transferred=5197555 worker.ajp13.read=121336154 worker.ajp13.busy=0 worker.ajp13.max_busy=1 worker.ajp13.connected=2 worker.ajp13.map_count=9 worker.ajp13.last_reset_at=1422616434 worker.ajp13.last_reset_ago=207 worker.ajp13.map.1.server=x.x.x [*:443] worker.ajp13.map.1.uri=/olat/raw/extensions/* worker.ajp13.map.1.type=Unmount Wildchar worker.ajp13.map.1.source=JkMount worker.ajp13.map.1.reply_timeout=-1 worker.ajp13.map.1.sticky_ignore=0 worker.ajp13.map.1.stateless=0 worker.ajp13.map.1.fail_on_status= worker.ajp13.map.1.active= worker.ajp13.map.1.disabled= worker.ajp13.map.1.stopped= worker.ajp13.map.1.use_server_errors=0 worker.ajp13.map.2.server=x.x.x [*:443] worker.ajp13.map.2.uri=/olat/raw/images/* worker.ajp13.map.2.type=Unmount Wildchar worker.ajp13.map.2.source=JkMount worker.ajp13.map.2.reply_timeout=-1 worker.ajp13.map.2.sticky_ignore=0 worker.ajp13.map.2.stateless=0 worker.ajp13.map.2.fail_on_status= worker.ajp13.map.2.active= worker.ajp13.map.2.disabled= worker.ajp13.map.2.stopped= worker.ajp13.map.2.use_server_errors=0 worker.ajp13.map.3.server=x.x.x [*:443] worker.ajp13.map.3.uri=/olat/raw/css/* worker.ajp13.map.3.type=Unmount Wildchar worker.ajp13.map.3.source=JkMount worker.ajp13.map.3.reply_timeout=-1 worker.ajp13.map.3.sticky_ignore=0 worker.ajp13.map.3.stateless=0 worker.ajp13.map.3.fail_on_status= worker.ajp13.map.3.active= worker.ajp13.map.3.disabled= worker.ajp13.map.3.stopped= worker.ajp13.map.3.use_server_errors=0 worker.ajp13.map.4.server=x.x.x [*:443] worker.ajp13.map.4.uri=/olat/raw/js/* worker.ajp13.map.4.type=Unmount Wildchar worker.ajp13.map.4.source=JkMount worker.ajp13.map.4.reply_timeout=-1 worker.ajp13.map.4.sticky_ignore=0 worker.ajp13.map.4.stateless=0 worker.ajp13.map.4.fail_on_status= worker.ajp13.map.4.active= worker.ajp13.map.4.disabled= worker.ajp13.map.4.stopped= worker.ajp13.map.4.use_server_errors=0 worker.ajp13.map.5.server=x.x.x [*:443] worker.ajp13.map.5.uri=/olat/raw/extensions* worker.ajp13.map.5.type=Unmount Wildchar worker.ajp13.map.5.source=JkMount worker.ajp13.map.5.reply_timeout=-1 worker.ajp13.map.5.sticky_ignore=0 worker.ajp13.map.5.stateless=0 worker.ajp13.map.5.fail_on_status= worker.ajp13.map.5.active= worker.ajp13.map.5.disabled= worker.ajp13.map.5.stopped= worker.ajp13.map.5.use_server_errors=0 worker.ajp13.map.6.server=x.x.x [*:443] worker.ajp13.map.6.uri=/olat/raw/images* worker.ajp13.map.6.type=Unmount Wildchar worker.ajp13.map.6.source=JkMount worker.ajp13.map.6.reply_timeout=-1 worker.ajp13.map.6.sticky_ignore=0 worker.ajp13.map.6.stateless=0 worker.ajp13.map.6.fail_on_status= worker.ajp13.map.6.active= worker.ajp13.map.6.disabled= worker.ajp13.map.6.stopped= worker.ajp13.map.6.use_server_errors=0 worker.ajp13.map.7.server=x.x.x [*:443] worker.ajp13.map.7.uri=/olat/raw/js* worker.ajp13.map.7.type=Unmount Wildchar worker.ajp13.map.7.source=JkMount worker.ajp13.map.7.reply_timeout=-1 worker.ajp13.map.7.sticky_ignore=0 worker.ajp13.map.7.stateless=0
Re: Mod_jk Configuration
Am 22.01.2015 um 01:03 schrieb Chris Arnold: +1 And also, could you specify again what URL you are requesting in the browser, which you would expect to be proxied to Tomcat ? https://share2.domain.tld Looking at the log you just showed, it seemed that the only requests ever passed through mod_jk for evaluation, where things like /error/* and mod_jk (rightly so) decides that they are not for Tomcat, declines to process them, and returns the request to Apache, to tell it to find another victim to handle that URL. Agreed! Which is why i need some guidance. You have set JkMount /share2/* worker1 which means: forward any request that hits the VirtualHost in which you have put that JkMount and where the request has a URI starting with /share2/ via the worker worker1. Then you test with the request http://share2.domain.tld/ which has URI /. This URI does not start with /share2/ so it does not match the JkMount and the request is not being forwarded. Attempting to map URI '/' from 1 maps Attempting to map context URI '/share2/*=worker1' source 'JkMount' no match for / found Instead Apache tries to produce an error page using a new interal request with an URI starting with /error/ etc. So either you test with the request http://share2.domain.tld/share2/what-ever-makes-sense-here or - if your really need to forward anything under http://share2.domain.tld/ you need to use JkMount /* worker1 instead of the JkMount you are currently using. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [ANN] Apache Tomcat 8.0.17 available
Am 21.01.2015 um 04:24 schrieb Leo Donahue: On Tue, Jan 20, 2015 at 5:09 PM, Mark Thomas ma...@apache.org wrote: The Apache Tomcat team announces the immediate availability of Apache Tomcat 8.0.17. - The RemoteAddrValve and RemoteHostValve can now optionally include the port when filtering along with a new option to trigger authentication rather than denying access There are no links on the changelog page for these and I was hoping to see some details about why this option was added. Optionally trigger authentication instead of denial in RemoteAddrValve and RemoteHostValve http://tomcat.apache.org/tomcat-8.0-doc/config/valve.html#Remote_Address_Filter The behavior when a request is refused can be changed to not deny but instead set an invalid authentication header Example #3 To allow unrestricted access to port 8009, but trigger basic authentication if the application is accessed on another port: I'm trying to understand this kind of setup. If an IP has been allowed to pass through via a Filter to a restricted resource, wouldn't the user get the container configured authentication dialog anyway? The original use case was: - the app does not have authentication configured - the app is officially only available via an AJP connector - for admin/testing purposes the app should be made available via an additional http connector but only for authorized people. Normal people must go via reverse proxy / AJP. You can use the above for this kind of setup without editing the app itself Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Mod_jk Configuration
Am 17.01.2015 um 00:31 schrieb Chris Arnold: Current working setup: apache 2.2 using mod_jk to pass 443 requests to tomcat on 8443. We are migrating from SLES 11 SP3 to SLEs 12. On SLES 11 we use alfresco 5.0.c which includes tomact 7.x i believe. SLES 11 has apache 2.2.10. SLES 12 has apache 2.4 and we use the same version of alfresco on SLES 12 (tomcat 7.x). So i installed SLES 12, apache 2.4 and alfresco 5.0.c on a test server. I then followed the upgrading guide for apache 2.2 to 2.4. I copied the existing working config files from the alfresco tomact to the SLES 12 alfresco tomact. Restarted all apache/tomcat services and try to go to https://share2.domain.tld.https://share2.domain.tld./ Chrome reports Error code: ERR_CONNECTION_REFUSED When i look at the apache log for that request, i dont see where the request is even making it to apache or tomcat. Here is mod_jk: # simple configuration for apache (for AJP connector, modul mod_jk.so) IfModule mod_jk.c JkWorkersFile /opt/alfresco/tomcat/workers.properties JkLogFile /var/log/alfresco/mod_jk.log JkShmFile /var/log/alfresco/shm # Log level to be used by mod_jk JkLogLevel error # The following line prohibits users from directly accessing WEB-INF Location /share/WEB-INF/ #AllowOverride None Require all denied /Location #Location /servlets-examples/WEB-INF/ #AllowOverride None #deny from all #/Location /IfModule No JkMount? mod_jk uses the JkMount directive to decide, which requests should be forwarded. Something like JkMount /myapp|/* balancer The directive should be put into the VirtualHost that is used in your Apache web server config to serve the requests for /myapp. Here is worker.properties: worker.list=jk-status worker.jk-status.type=status worker.jk-status.read_only=true worker.list=jk-manager worker.list=worker1 worker.jk-manager.type=status worker.list=balancer worker.balancer.type=lb worker.balancer.max_reply_timeouts=10 worker.balancer.balance_workers=worker1 worker.worker1.reference=worker.template worker.worker1.host=localhost worker.worker1.port=8009 worker.worker1.activation=A worker.template.type=ajp13 worker.template.socket_keepalive=true worker.template.connection_pool_minsize=0 worker.template.connection_pool_timeout=600 worker.template.reply_timeout=30 worker.template.recovery_options=3 I have mod_jk loaded in apache. Can anyone identify where the issue is? Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Mod_jk Configuration
Am 17.01.2015 um 01:51 schrieb Konstantin Kolinko: 2015-01-17 2:31 GMT+03:00 Chris Arnold carn...@electrichendrix.com: Current working setup: apache 2.2 using mod_jk to pass 443 requests to tomcat on 8443. We are migrating from SLES 11 SP3 to SLEs 12. On SLES 11 we use alfresco 5.0.c which includes tomact 7.x i believe. SLES 11 has apache 2.2.10. SLES 12 has apache 2.4 and we use the same version of alfresco on SLES 12 (tomcat 7.x). So i installed SLES 12, apache 2.4 and alfresco 5.0.c on a test server. I then followed the upgrading guide for apache 2.2 to 2.4. I copied the existing working config files from the alfresco tomact to the SLES 12 alfresco tomact. Restarted all apache/tomcat services and try to go to https://share2.domain.tld.https://share2.domain.tld./ Chrome reports Error code: ERR_CONNECTION_REFUSED When i look at the apache log for that request, i dont see where the request is even making it to apache or tomcat. Try to request a static file from DocumentRoot directory. If you cannot, then your HTTPS is not configured correctly. Get that working first. Yeah, I overlooked that in my first response. As long as you don't see the request in the Apache access log, you'll not make progress. Here is mod_jk: # simple configuration for apache (for AJP connector, modul mod_jk.so) IfModule mod_jk.c JkWorkersFile /opt/alfresco/tomcat/workers.properties JkLogFile /var/log/alfresco/mod_jk.log JkShmFile /var/log/alfresco/shm # Log level to be used by mod_jk JkLogLevel error # The following line prohibits users from directly accessing WEB-INF Location /share/WEB-INF/ #AllowOverride None Require all denied /Location #Location /servlets-examples/WEB-INF/ #AllowOverride None #deny from all #/Location /IfModule Here is worker.properties: worker.list=jk-status worker.jk-status.type=status worker.jk-status.read_only=true worker.list=jk-manager worker.list=worker1 worker.jk-manager.type=status worker.list=balancer So, what is the value for worker.list? You set the same worker.list property 4 times, but with different values. Konstantin: that's actually OK. Some of the properties are allowed multiple times. The values are cumulative. That aloows a modular layout of the config file. In the docs under http://tomcat.apache.org/connectors-doc/reference/workers.html it is explicitly mentioned if a property is allowed multiple times. Currently this is true e.g. for the worker.list property and the balance_workers property of an lb worker. worker.balancer.type=lb worker.balancer.max_reply_timeouts=10 worker.balancer.balance_workers=worker1 worker.worker1.reference=worker.template worker.worker1.host=localhost worker.worker1.port=8009 worker.worker1.activation=A worker.template.type=ajp13 worker.template.socket_keepalive=true worker.template.connection_pool_minsize=0 worker.template.connection_pool_timeout=600 worker.template.reply_timeout=30 worker.template.recovery_options=3 Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Mod_jk Configuration
Am 17.01.2015 um 03:43 schrieb Chris Arnold:
No JkMount?
mod_jk uses the JkMount directive to decide, which requests should be
forwarded. Something like
JkMount /myapp|/* balancer
The directive should be put into the VirtualHost that is used in your
Apache web server config to serve the requests for /myapp.
When i look at the existing server that works, i cant tell if we are using
mod_proxy or mod_jk. In the SSL-virtualhost, i see:
#This rewrites https://share.anydomain.tld to our share server
RewriteEngine On
RewriteCond %{HTTP_HOST} ^share\.
RewriteCond %{HTTPS} on
RewriteCond %{REQUEST_URI} !^/share/
RewriteRule ^/(.*) https://share.domain.tld:8443/share/ [P]
That will forward any https request with a URI *not* starting with
/share/ using mod_proxy. If there are no ProxyPass directives, it will
not use persistent backend connections though.
JkMount /share/* worker1
And this will use a worker named worker1 to forward anything with a URI
starting with /share/ using mod_jk.
Neither will work as long as your request don't actually hit the Apache
web server.
Rainer
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Mod_jk Configuration
Am 17.01.2015 um 04:05 schrieb Chris Arnold: When i look at the apache log for that request, i dont see where the request is even making it to apache or tomcat. Try to request a static file from DocumentRoot directory. If you cannot, then your HTTPS is not configured correctly. Get that working first. Yeah, I overlooked that in my first response. As long as you don't see the request in the Apache access log, you'll not make progress. I meant to answer that in my last answer: i can get to a site from HTTPD and i can also get to the https://192.168.xx.3:8443 Tomcat SSL with no problems. I just cant make it from HTTPD to Tomcat. Note that mod_jk will not use your 8443 port on Tomcat. It will use the 8009 AJP port (and connector in server.xml). If your mod_jk log doesn't contain any errors and if the needed JkMount directive is in place, increase the log level of mod_jk: JkLogLevel debug That will produce lots of info, but since your system is not already doing production, you can increase the log level, restart Apache and then try a single request that you expect to be forwarded from Apache to Tomcat. The log of that request (and of the startup) will help us to find the reason for your problems. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk status worker returns 500 error with no messages, logs
Am 13.01.2015 um 18:59 schrieb Christopher Schultz: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 All, On 1/10/15 2:56 PM, Christopher Schultz wrote: I'm having a problem with mod_jk 1.2.40 under Apache 2.4.10. After a successful start up, accessing the mod_jk status page gives me a 500 error with no errors on the screen or in the logs. Solved. The problem turned out to be authnz_ldap, which does not produce any errors when it can't reach the LDAP server. A bad iptables rule prevented this web server from reaching our LDAP server, and the behavior was 500 server error with no messages in any log files for any resource protected by the module. I spent a great deal of time trying to figure out how this web server was configured differently than any of the others, which were of course still working just fine. Since the problem was with an external server (the LDAP server), it took quite a long time to track down, as the web servers were in fact configured identically :) Thanks for letting us know! Hopefully the status worker activation update problem in the other thread will be fixed by a valid shm path. Otherwise analysis will get nasty ... Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Scripting mod_jk load-balancer member changes
Hi Chris, Am 12.01.2015 um 20:12 schrieb Christopher Schultz: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 All, On 1/12/15 2:01 PM, Christopher Schultz wrote: I'm running into a bit of difficulty scripting updates for load-balancer members and thought I'd ask here before slogging through the code to see if there are any requirements I'm missing. Long story short, I'd like to script setting the activation status for a particular load-balanced worker to something, like ACT or DIS. I've written a python script to do this for me and I can confirm with Wireshark that it's sending what I expect. Here is the HTTP request that is being sent: POST /jk-status HTTP/1.1 Accept-Encoding: identity Content-Length: 43 Host: localhost Content-Type: application/x-www-form-urlencoded Connection: close User-Agent: mod_jk.py / Python-urllib from=listcmd=updatesw=myworkervwa=1w=lb That's the whole request. The response I get is a 200 response, but it's got the form content in it again, and not the you will be redirected in 3 seconds response that I get from the web UI. I can think of a few things that I just wanted to sanity check: 1. I need to provide all of the various values and not just the vwa value (which is the short-code request parameter value for the activation status). 2. The status worker can't handle POST requests (the web interface uses GET). 3. I've broken something else. Can anyone confirm either 2 or 3 above, or both? I tried switching to GET and so I'm using this URL, now: GET /jk-status?from=listcmd=updatesw=myworkervwa=0w=lb HTTP/1.1 Accept-Encoding: identity Host: localhost Connection: close User-Agent: mod_jk.py / Python-urllib (Note that the load balancer worker's name is lb and the balanced worker is called myworker) I'm getting a 200 response with the expected You will be redirected in 3 seconds page, but the value for the activation of this worker is not actually being updated. So, it seems that the data must be in the URL (i.e. GET) but something is still not working as I expect. Can anyone shed some light on what I might be missing? I can probably change the order of the parameters if that's what's required... I'm using a python dictionary (an unordered name/value hash) to build my request parameters which is automatically converted to URL parameters, but they are unordered. I could put them in order if that is likely to change anything. Only GET works currently. It is easiest to do what you want in the browser GUI and then take the request URL including query string from the web server access log. And vice versa, does the URL do what you want, if you enter it in the browser? At first sight, the query string looks good. Value 0 means active. The from=list part should not be necessary. It will define, which redirect URL will be used in the response (in which you are not really interested). In case of problems, you can try the JkLogLevel debug for the status worker request. Status worker also logs quite verbosely what it does. Any changes done will be logged on info log level, problems as errors or warnings. Are you using 1.2.40 or an older version? Some older versions had problems with updating data via status worker. It should be in the changelog of the more recent version. Finally: make sure the status worker is not configured as read-only. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 8, Apache 2.4, Tomcat Connector 1.2.40, Windows 7 home basic issue
Hi Sandip, Am 10.01.2015 um 12:56 schrieb Sandip Gaikwad: Hi Andre/Rainer, Please find below httpd.conf file's content: I think you need to get used to the Apache web server configuration basics. For instance learn what the lines Include conf/extra/httpd-vhosts.conf and Include conf/extra/httpd-sni.conf in your config actually do. Then get back to my original mail about JkMountCopy and look for existing VirtualHost... blocks in any config file which is part of your complete Apache web server configuration. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 8, Apache 2.4, Tomcat Connector 1.2.40, Windows 7 home basic issue
Am 08.01.2015 um 09:45 schrieb André Warnier: Sandip Gaikwad wrote: Hi Terence, [ snip ] When i access http://localhost/jenkins/ i am getting following error: Not Found The requested URL http://jenkins/ was not found on this server. ... *httpd.conf* LoadModule jk_module C:/Apache24/modules/mod_jk.so JkWorkersFile C:/tomcat-connectors-1.2.40-src/conf/workers.properties JkLogFile C:/Apache24/logs/mod_jk.log JkLogLevel info JkLogStampFormat [%a %b %d %H:%M:%S %Y] JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories JkRequestLogFormat %w %V %T JkMount /jenkins/* worker1 Did you look at what is in the mod_jk.log file ? Are you sure that the call http://localhost/jenkins/; really gets proxied to Tomcat ? If you do not see anything in the mod_jk.log file when you request http://localhost/jenkins/;, then it is probably not happening, and then also, the error page which you get in the browser is coming from the front-end Apache httpd, not from Tomcat (these error pages have a quite distinctive style between httpd and Tomcat, so you should be able to tell - with a browser other than IE). In addition to the comments form André: JkMount is by default only active in the vhost in which you configure it. If your Apache config contains some VirtualHost which matches your request but doesn't contain the JkMount in its ocnfig, then the rule will not trigger. You can add JkMountCopy All in the above mod_jk config block, which will active the JkMounts in any VirtualHost. If it is working then, you should remove JkMountCopy again, find the correct VirtualHost and move the JkMount from the global config to the VirtualHost. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk monitoring
Hi Chris, Am 07.11.2014 um 23:04 schrieb Christopher Schultz: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 All, I've been playing around with mod_jk's status screens a lot lately. I also recently upgraded from Apache httpd 2.2 to Apache httpd 2.4 and the hostname shown at the top of the page changed from the real hostname to the masquerading hostname (more on that in a second) and I was wondering what else was going on. First off, I have a load-balancer with 3 web servers behind it. Each individual web server has its own distinct hostname (the real hostname) like foo.example.com, bar.example.com, etc. plus, when you contact it through the load-balancer, it thinks it's www.example.com of course. Everything seems to work fine. These days, the jk-status page always says JK Status Manager for www.example.com:443 when I visit the page, no matter which back-end server I get by entering https://foo.example.com/jk-status. When I connect to an individual server, I use HTTPS of course, so I get the _default_:443 virtual host which makes sense. My conf/hostname.conf has ServerName www.example.com and no server aliases, and my conf.d/ssl.conf has no ServerName and no ServerAlias. (This is on Amazon Linux, which is essentially CentOS, and I'm using package-managed versions of httpd so this is the layout they have). I'm assuming that, since I have no ServerAlias for my :443 VirtualHost, that's where the name at the top of the page comes from. Can anyone confirm that? It would be nice if the page advertised my real hostname instead of the generic one. That is, I'd like to see JK Status Manager for foo.example.com:443 instead of www.example.com:443. Would adding a ServerAlias in either hostname.conf (which I believe is loaded at the top-level configuration) or in ssl.conf inside the VirtualHost achieve that? The name that is shown is the server name for the request being handled. This name is influenced by - the host header send by the client - the ServerName configured in the VirtualHost or global server actually serving the jk-status request - the UseCanonicalName setting For details see: http://httpd.apache.org/docs/2.4/en/mod/core.html#usecanonicalname A ServerAlias does not directly change that name. But if setting a ServerAlias in a VirtualHost that does not handle jk-status currently results in that VirtualHost handling your jk-status afer setting the Alias, it can also change the name shown in the page. First check what your UseCanonicalName setting is. If it is off, then mod_jk will simply show what is in the host header of your request - as long as your LB forwards the host header correctly. That would typically the best situation. Everything else quickly gets messy. Next, I have a question about the workers. I believe - but I'm not sure - that each worker is unique to the server as a whole, and not to just a single VirtualHost, right? That is, my jk_workers.properties file contains a series of worker definitions and I can use them in any VirtualHost I like. If I reference the same worker from two different VirtualHosts, I just get two references to the same worker, right? The reason I ask is that I don't have the jk-status page available for users coming from the load-balancer. I want to make sure that when I disable a worker when looking at the :443 jk-status page that I'm disabling it across all VirtualHosts in the server. (The load-balancer uses a different VirtualHost, and it's tough to convince the load-balancer to pick a specific back-end server to direct my commands to, so... it's kind of important that the above be true.) Can anyone confirm that the workers are shared? I think it would be very difficult to administer if the workers were not shared, but I wanted to be absolutely sure. Yes, workers and their status are global (shared). Only Mounts are not. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk monitoring
Am 07.11.2014 um 23:58 schrieb Christopher Schultz: Great. We've been having some problems draining Tomcat nodes, and I was concerned that maybe I had overlooked this one possibility. Our next release will include a version of the recently-discussed load-balancer draining filter/valve so we'll see how that does. There are some tricks also in mod_jk to improve draining: - people using an old session id without an existing session, e.g. having a JSESSIONID cookie or who bookmarked a jsessionid-encoded URL, will hinder draining. Typically clearing the cookie during logout doesn't happen. So what you can do is twi things: - you can use mod_headers to force cookie clearing on a logout URI by setting the Set-Cookie header to an empty JSESSIONID cookie if the URI matches your logout URI - you can use mod_setenvif or mod_rewrite to set the Apache environment variable JK_STICKY_IGNORE to 1 for requests that might have a session id but can actually be handled non-sticky. E.g. a request for a login form from a user that might use an old cookie. If mod_jk sees JK_STICKY_IGNORE it will not handle the request sticky, so if you already disabled some worker, the request will not be send there, even if the cookie points to that worker. Finally you can use the Tomcat Manager webapp Expire Sessions feature. If you click it, it will not really expire sessions (actually only those that the automatic expiration would expire during the next run), but it will show a histogram of session idleness. Depending on the data you might be able to decide, when enough draining happened, e.g. the remaining sessions all or nearly all have a long idleness time (but of course shorter than the session timeout). Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Trying to filter noise from catalina.out.
Am 05.11.2014 um 00:12 schrieb Brandon Darbro: Looking for configuration help. Using tomcat7 7.0.34 from rpm package tomcat7-7.0.34-3.jpp6.noarch. Followed the instructions for using log4j for catalina.out found here: http://tomcat.apache.org/tomcat-7.0-doc/logging.html#Using_Log4j Took the example log4j.properties file from the instructions above, corrected the logging paths for /var/log/tomcat7, and put it through a properties to xml converter. Replaced log4j.properties with log4j.xml, and logging is working. Now we want to try and filter out an Exception we are willing to live with, but can't have overflowing our log. Added the following filter: filter class=org.apache.log4j.filter.ExpressionFilter param name=expression value=EXCEPTION ~= java.io.NotSerializableException / param name=acceptOnMatch value=false/ /filter Yet we continue to get the exception in the log: Nov 4, 2014 1:52:45 PM net.sf.ehcache.distribution.RMISynchronousCacheReplicator replicatePutNotification SEVERE: Exception on replication of putNotification. error marshalling arguments; nested exception is: java.io.NotSerializableException: com.fakename.services.cache.ehcache.EHCacheServiceImpl. Continuing... java.rmi.MarshalException: error marshalling arguments; nested exception is: java.io.NotSerializableException: com.fakename.services.cache.ehcache.EHCacheServiceImpl at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:138) ...snip... Caused by: java.io.NotSerializableException: com.fakename.services.cache.ehcache.EHCacheServiceImpl at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1164) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1518) ...snip... What am I doing wrong? Full xml and/or log of error available if requested. The formatting of this message looks like it is *not* written using Log4J, but instead using java.util.logging. To rule out mail reformatting: If the Date and timestamp is on one line, and the text starting with SEVERE on the second line, then my guess is correct. As others are indicating: these messages are not written by Tomcat, but instead by your application, which seems to use java.util.logging, which by default loggs to console = STDOUT which is redirected by Tomcat catalina.sh script to catalina.out. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Weird (apocryphal) reference to Tomcat in Wikipedia
Am 05.11.2014 um 16:39 schrieb Christopher Schultz: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 André, On 11/5/14 10:31 AM, André Warnier wrote: Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 All, I was reading the Wikipedia entry on .properties format yesterday (http://en.wikipedia.org/wiki/.properties) and I saw a mention of Apache Tomcat that doesn't make any kind of sense to me: In Apache Tomcat the exclamation mark denotes a Negation operator when used as the first non blank character in a line[citation needed]. I'm the one that added the [citation needed] with a note that I think this is false. Does anyone know if there was some kind of ancient version of Tomcat that read its own .properties files and added some kind of magic to do what the above says? I think it's a complete fabrication with no basis in reality. Any ideas? Maybe a very indirect reference to : http://tomcat.apache.org/connectors-doc/reference/uriworkermap.html (see : Exclusions and rule disabling) Good call. This actually might be the source of that text. but, like you, I think that this reference is quite irrelevant in that Wikipedia article. I agree. This is one single instance of one file that typically ends in .properties that has these semantics. It's not even correct to say that Apache Tomcat does this... it's really Apache mod_jk that does this, and mod_jk isn't a Java program (though of course, .properties isn't exclusive to the Java world). I'm sure that mod_jk doesn't follow the exact rules of Java .properties specifications like using \ as an escape character, trimming leading spaces, etc. I think I'll update the Wikipedia article. Rather than removing the text, I think I'll move it into another section and explain. If you don't want to remove it, you can add it as an example for existing custom variations of the properties format. Note though, that uriworkermap.properties does not implement most of the standard specifics of the Java properties file format, so it is not a superset. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: SecureRandom instance for session ID generation using [SHA1PRNG] took [510,962] milliseconds !
Am 03.10.2014 um 14:01 schrieb Christopher Schultz: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Martin, On 10/3/14 5:48 AM, Martin Hamant wrote: Le 03/10/2014 11:26, Martin Hamant a écrit : The virtual (qemu) server runs with 4GB RAM Sorry, The hypervisor is KVM. The VM is running on top of OpenStack So... This could lead somewhere as I am reading http://blog.dustinkirkland.com/2012/10/entropy-or-lack-thereof-in-openstack.html OpenStack or not, running on a VM usually means that the underlying OS is providing the source of entropy. If your physical machine is heavily virtualized, you may have multiple entropy sinks constantly draining your source(s() of entropy. If you wait for a while, things will recover. If you find you are constantly blocking waiting for more randomness to be available from your random source, you basically have 3 options: 1. Suffer through it. Just keep waiting. 2. Use a poor source of randomness, like /dev/urandom on Linux. I wouldn't recommend this for any kind of production deployment, since the entropy source is watered-down. You can't rely on it for important things like encryption (including SSL) and really anything that requires random numbers that are as random as possible (like session ids). 3. Get yourself a hardware entropy source. You can buy USB keys that do this kind of thing. Make sure whatever you get is compatible with your OS and accessible by Java (better yet, get one that will simply dump its randomness into /dev/random). ... and in case you are heading for the urandom solution and are sing JDK before 8, you should use e.g. -Djava.security.egd=file:/dev//urandom and *not* -Djava.security.egd=file:/dev/urandom For background info look at http://marc.info/?l=tomcat-devm=130182757504685w=2 or more officially http://bugs.java.com/view_bug.do?bug_id=6202721 and http://openjdk.java.net/jeps/123 This has been fixed in JDK8 though (finally). Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Why does mod_jk bypass Apache authorization?
Am 11.09.2014 um 22:15 schrieb Daniel Pfeiffer: On 2014-09-10 22:12, Mark Eggers wrote: I don't think that the trailing /* is valid for a simple Location directive. If you want regular expressions you'll have to use either LocationMatch or Location ~ (Location followed by the ~) This was the decisive hint! JkMount needs /*, but Location doesn't seem to handle it well. This makes the one-argument-form of JkMount quite useless. The solution was using the two-argument-form isolated with /* and Location without. I updated the docs in svn to make that behavior more clear und to warn against that type of use. See: http://svn.apache.org/viewvc?view=revisionrevision=1626582 Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
[ANN] New committer: Felix Schumacher
On behalf of the Tomcat committers I am pleased to announce that Felix Schumacher (fschumacher) has been voted in as a new Tomcat committer. Please join me in welcoming him. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AJP connector is blocking JSON responses
Am 10.08.2014 um 15:41 schrieb Omar Belkhodja: Hi, I'm adding HTTPS access to my Java site. For that, I have chosen to leave the management of HTTPS to an Apache server. The Apache server is connecting to tomcat through mod_jk and AJP 1.3. I'm logging all my Tomcat request/responses, and I'm facing the following problem: - If I access my URL through the standard Tomcat connector on port 8080, I have a correct response (JSON) - If I access my URL through Apache HTTPS, I have an empty response(no JSON) From the Tomcat logs below, the last parameter is the number of bytes returned, and you can see that they are different. 41.227.87.197:443 - - [10/Aug/2014:15:24:32 +0200] 1688 GET /url?format=json HTTP/1.1 200 33 41.227.87.197:8080 - - [10/Aug/2014:15:24:46 +0200] 734 GET /url?format=json HTTP/1.1 200 44 Another test I have done is to use an XML response instead of JSON response, and I can't see any more the problem. Any idea what could it be ? Can you set JkLogLevel trace for mod_jk and post the complete log output for a single request? We can then check the headers etc. For a first try we don't need the startup messages, only the log lines related to receiving and forwarding the request and reponse. Are there any other Apache directives active that might change the URI or headers? RewriteRules etc.? Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk and session stickiness
On 23.07.2014 16:37, Christopher Schultz wrote: On 7/18/14, 12:13 PM, Rainer Jung wrote: On 17.06.2014 16:43, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 All, I've been using sticky sessions with mod_jk and I can see that there is a bit of a problem when attempting to take a backend Tomcat server out of load-balanced rotation: a user who never (or rarely) restarts their web browser will keep the same JSESSIONID cookie forever and therefore end up with the same backend server whether it has been disabled or not. Quick series of events: 1. User visits load-balancer and gets a randomly-assigned backend server/route. We'll call this route X. The JSESSIONID cookie set by the backend server is therefore foo.X. 2. User's requests are routed by mod_jk to route X. 3. Route X is disabled using mod_jk's status worker 4. User's session on server X expires. [Technically, 3 and 4 can happen in either order] 5. User makes a new request to the load-balancer, and mod_jk sees the JSESSIONID cookie still set to foo.X. mod_jk sends the request to route X which allows the user to login, etc. Thus, it takes more time than necessary to bleed all the traffic from route X for maintenance, etc. Is there a way for mod_jk to ask route X if the session is *still* valid? It seems that mod_jk will not re-route a request that looks like it's got a valid session id to a new (active) backend server unless the backend server X is actually down. Any ideas? Not exactly what you want, but you can build something around it: 1) Switch off stickyness for specific URLs If you know that users will come via specific URLs, like a login page, and you want that page to be handled non-sticky to optimize load balancing even if users have an old cookie, you can do that by setting the Apache envvar JK_STICKY_IGNORE. Look for JK_STICKY_IGNORE on: http://tomcat.apache.org/connectors-doc/reference/apache.html This seems like a reasonable way to do things, except that we still want to support requests to protected resources being saved and redirected to the login page. If we did this (JK_STICKY_IGNORE), then we'd end up forgetting the saved request (because the client would be re-balanced to another node for the login page) and ending up with a (useless) session on the node we are trying to take down. We'd like to retain the request-saving capabilities of the container. Whatever saved exactly means: if you can identify that situation in terms of any part of the request (e.g. something in the referer header etc.), you can add that as a positive or negative condition via RewriteCond (or the 2.4 unified expression parser) and set JK_STICKY_IGNORE in a RewriteRule that does not change the URI, only sets the variable and also only of the RewriteConds apply. You'd have to analyze the request-response stream e.g. with a browser plugin to look for a useful header or similar which can be used to distinguish the saved and the normal case. One option we have here is to provide separate URLs for regular logins versus the saved-request logins mentioned above: that would probably solve both problems. 2) Improve handling of sessions for node with activation disabled If you switch a node to activation disabled, mod_jk will not send requests there, that have no session id (and of course also not when the session route points to another node). But the old cookie requests might still go there. Yes, this is what we would normally do. For that you can use the feature, that mod_jk forwards the activation state to the Tomcat node. The node can then decide on itself, whether it will handle a request coming in with an invalid session id, or for example clears the session cookie and does a self-referential redirect, which then by mod_jk is balanced on the fully enabled nodes. This sounds like a promising technique. I was thinking we'd just tell the Tomcat node directly (e.g. set a context-scoped flag) that it was disabled, but having AJP forward that information would be even better, so the state can be managed entirely by the status worker on the httpd side. This logic can be put into a servlet filter. I'm not sure it can be in a Filter because of the interaction with the saved-request features described above. If in a Filter, the request would be saved before the Filter has a chance to see it, then authentication would take place, etc. I think this has to be in a Valve, and it has to happen before the AuthenticatorValve sees the request. Do you see a way around using a Valve? Assuming that such a Valve would be required, I think we should provide one with Tomcat. I'd be happy to write such a Valve. Can't comment because I don't know what exactly the saved request feature means, resp. how it works. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h
Re: mod_jk and session stickiness
Am 23.07.2014 um 17:59 schrieb Christopher Schultz: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Rainer, On 7/23/14, 11:23 AM, Rainer Jung wrote: On 23.07.2014 16:37, Christopher Schultz wrote: On 7/18/14, 12:13 PM, Rainer Jung wrote: On 17.06.2014 16:43, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 All, I've been using sticky sessions with mod_jk and I can see that there is a bit of a problem when attempting to take a backend Tomcat server out of load-balanced rotation: a user who never (or rarely) restarts their web browser will keep the same JSESSIONID cookie forever and therefore end up with the same backend server whether it has been disabled or not. Quick series of events: 1. User visits load-balancer and gets a randomly-assigned backend server/route. We'll call this route X. The JSESSIONID cookie set by the backend server is therefore foo.X. 2. User's requests are routed by mod_jk to route X. 3. Route X is disabled using mod_jk's status worker 4. User's session on server X expires. [Technically, 3 and 4 can happen in either order] 5. User makes a new request to the load-balancer, and mod_jk sees the JSESSIONID cookie still set to foo.X. mod_jk sends the request to route X which allows the user to login, etc. Thus, it takes more time than necessary to bleed all the traffic from route X for maintenance, etc. Is there a way for mod_jk to ask route X if the session is *still* valid? It seems that mod_jk will not re-route a request that looks like it's got a valid session id to a new (active) backend server unless the backend server X is actually down. Any ideas? Not exactly what you want, but you can build something around it: 1) Switch off stickyness for specific URLs If you know that users will come via specific URLs, like a login page, and you want that page to be handled non-sticky to optimize load balancing even if users have an old cookie, you can do that by setting the Apache envvar JK_STICKY_IGNORE. Look for JK_STICKY_IGNORE on: http://tomcat.apache.org/connectors-doc/reference/apache.html This seems like a reasonable way to do things, except that we still want to support requests to protected resources being saved and redirected to the login page. If we did this (JK_STICKY_IGNORE), then we'd end up forgetting the saved request (because the client would be re-balanced to another node for the login page) and ending up with a (useless) session on the node we are trying to take down. We'd like to retain the request-saving capabilities of the container. Whatever saved exactly means: if you can identify that situation in terms of any part of the request (e.g. something in the referer header etc.), you can add that as a positive or negative condition via RewriteCond (or the 2.4 unified expression parser) and set JK_STICKY_IGNORE in a RewriteRule that does not change the URI, only sets the variable and also only of the RewriteConds apply. You'd have to analyze the request-response stream e.g. with a browser plugin to look for a useful header or similar which can be used to distinguish the saved and the normal case. When I say saved I mean this workflow: 1. User requests a protected resource 2. Container saves the request (which creates a new session), presents the login screen 3. User authenticates 4. User is redirected to request saved in step #1 So where's the interaction with putting a JK_STICKY_IGNORE on the login page if directly retrieved? The above sequence of steps does not retrieve the login page from the outside via its real URL, e.g. /login, does it? That env var is only useful for URIs for which you know that they either don't need a session (unprotected static content) or will start a new session (login page accessed by URI). The other case, a request with an invalid session ID accessing a tomcat instance with activation disabled can IMHO be handled by a filter that - checks whether the request has a valid session using getSession(false), if it has one, let the request proceed - checks activation state, if active, let the request proceed - checks the request method, if not GET, let the request proceed - otherwise: - set the session cookie, e.g. JSESSIONID the an empty value - issue an external redirect to the same request URL - optional redirect loop detection: add a query string param or cookie that gets the local jvmRoute appended during each redirect. Before doing the redirect, check that the local jvmRoute is not already part of that token (we have already been here before) This would not really interfere with your saved requests: they would get a redirect which the browser follwos automatically and after that you will observe the normal behavior. One option we have here is to provide separate URLs for regular logins versus the saved-request logins mentioned above: that would probably solve both problems. 2) Improve handling of sessions for node with activation disabled If you
Re: mod_jk and session stickiness
Am 23.07.2014 um 21:03 schrieb Christopher Schultz: On 7/23/14, 1:49 PM, Rainer Jung wrote: The other case, a request with an invalid session ID accessing a tomcat instance with activation disabled can IMHO be handled by a filter that - checks whether the request has a valid session using getSession(false), if it has one, let the request proceed - checks activation state, if active, let the request proceed - checks the request method, if not GET, let the request proceed - otherwise: - set the session cookie, e.g. JSESSIONID the an empty value - issue an external redirect to the same request URL - optional redirect loop detection: add a query string param or cookie that gets the local jvmRoute appended during each redirect. Before doing the redirect, check that the local jvmRoute is not already part of that token (we have already been here before) This would not really interfere with your saved requests: they would get a redirect which the browser follwos automatically and after that you will observe the normal behavior. This is exactly what I have implemented -- as a Filter since we can insert it before SecurityFilter intercepts the requests -- and my tests suggest that it will work correctly. I added code to strip-out any ;jsessionid path parameter from the URL ACK if it exists, but haven't done any of the redirect loop detection (yet). I think the loop detection is going to have to keep a running list of visited nodes which, in large clusters, could grow very large especially if the node names are long. I'll post my code when it's a little more featureful. As long as nothing goes wrong, the first redirect - having no more session info - should not end up in getting redirected as well, because it should only be send to an active node by mod_jk. So you can even stop redirecting if you detect, that it already happened once. For a large cluster that might be better. Multiple might happen (didn't check the code), if all active members are in error state. Even then one would not like to produce many of those redirects for one request, so again, only allowing one redirect might be the right solution. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Conditional logging
On 22.07.2014 16:22, Dames, Kristopher J wrote:
Kris,
On 7/21/14, 6:43 PM, Dames, Kristopher J wrote:
Can anyone point me to an example of conditional logging in Tomcat
6? Here is what I have tried to no avail. I want requests that
match /*/iaahb to not be logged.
In my server.xml:
Valve className=org.apache.catalina.valves.AccessLogValve
condition=DoNotLog directory=logs fileDateFormat=-MM-dd
pattern=%{X-Forwarded-For}i %h %l %u %t quot;%rquot; %s %b
quot;%{Referer}iquot; quot;%{User-Agent}iquot;
prefix=access_log/
in my web.xml:
!-- Custom filter to prevent logging health check requests --
filter filter-nameSet Do Not Log Attribute/filter-name
filter-classSetDoNotLogFilter/filter-class init-param
param-nameDoNotLog/param-name param-valuetrue/param-value
/init-param /filter filter-mapping filter-nameSet Do Not
Log Attribute/filter-name url-pattern/*/iaahb/url-pattern
/filter-mapping
On the face of it, this looks like it should work. What does your
SetDoNoLogFilter code look like?
As Mark Eggers already said: you can also use the Tuckey
UrlRewriteFilter to set the chosen request attribute DoNotLog for you.
See for example
http://urlrewritefilter.googlecode.com/svn/trunk/src/doc/manual/4.0/index.html#set
I haven't tried it but something like
rule
from casesensitive=true^/[^/]*/iaahb$/from
set name=clientDoNotLog/set
/rule
Note that for TC 7 there's also a conditionIf and conditionUnless, which
makes it a bit easier, if you need to revert the logic:
http://tomcat.apache.org/tomcat-7.0-doc/config/valve.html#Access_Log_Valve
Regards,
Rainer
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk and session stickiness
Hi Chris, late answer but at least an answer. See below. On 17.06.2014 16:43, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 All, I've been using sticky sessions with mod_jk and I can see that there is a bit of a problem when attempting to take a backend Tomcat server out of load-balanced rotation: a user who never (or rarely) restarts their web browser will keep the same JSESSIONID cookie forever and therefore end up with the same backend server whether it has been disabled or not. Quick series of events: 1. User visits load-balancer and gets a randomly-assigned backend server/route. We'll call this route X. The JSESSIONID cookie set by the backend server is therefore foo.X. 2. User's requests are routed by mod_jk to route X. 3. Route X is disabled using mod_jk's status worker 4. User's session on server X expires. [Technically, 3 and 4 can happen in either order] 5. User makes a new request to the load-balancer, and mod_jk sees the JSESSIONID cookie still set to foo.X. mod_jk sends the request to route X which allows the user to login, etc. Thus, it takes more time than necessary to bleed all the traffic from route X for maintenance, etc. Is there a way for mod_jk to ask route X if the session is *still* valid? It seems that mod_jk will not re-route a request that looks like it's got a valid session id to a new (active) backend server unless the backend server X is actually down. Any ideas? Not exactly what you want, but you can build something around it: 1) Switch off stickyness for specific URLs If you know that users will come via specific URLs, like a login page, and you want that page to be handled non-sticky to ptimize load balancing even if users have an old cookie, you can do that by setting the Apache envvar JK_STICKY_IGNORE. Look for JK_STICKY_IGNORE on: http://tomcat.apache.org/connectors-doc/reference/apache.html 2) Improve handling of sessions for node with activation disabled If you switch a node to activation disabled, mod_jk will not send requests there, that have no session id (and of course also not when the session route points to another node). But the old cookie requests might still go there. For that you can use the feature, that mod_jk forwards the activation state to the Tomcat node. The node can then decide on itself, whether it will handle a request coming in with an invalid session id, or for example clears the session cookie and does a self-referential redirect, which then by mod_jk is balanced on the fully enabled nodes. This logic can be put into a servlet filter. You have to be careful though to not produce redirecting cycles, e.g. in case of errors or all other nodes are down. You could add the name of the first node the the URL as a query param, and if you see it a second time, then do not redirect again. The request attribute is named JK_LB_ACTIVATION. Search for that name on http://tomcat.apache.org/connectors-doc/generic_howto/loadbalancers.html HTH Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk and session stickiness
On 08.07.2014 14:29, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 All, On 6/17/14, 10:43 AM, Christopher Schultz wrote: All, I've been using sticky sessions with mod_jk and I can see that there is a bit of a problem when attempting to take a backend Tomcat server out of load-balanced rotation: a user who never (or rarely) restarts their web browser will keep the same JSESSIONID cookie forever and therefore end up with the same backend server whether it has been disabled or not. Quick series of events: 1. User visits load-balancer and gets a randomly-assigned backend server/route. We'll call this route X. The JSESSIONID cookie set by the backend server is therefore foo.X. 2. User's requests are routed by mod_jk to route X. 3. Route X is disabled using mod_jk's status worker 4. User's session on server X expires. [Technically, 3 and 4 can happen in either order] 5. User makes a new request to the load-balancer, and mod_jk sees the JSESSIONID cookie still set to foo.X. mod_jk sends the request to route X which allows the user to login, etc. Thus, it takes more time than necessary to bleed all the traffic from route X for maintenance, etc. Is there a way for mod_jk to ask route X if the session is *still* valid? It seems that mod_jk will not re-route a request that looks like it's got a valid session id to a new (active) backend server unless the backend server X is actually down. Any ideas? Any takers? I was thinking that the following strategy might work: 1. Extend mod_jk's load-balancer worker to include a new directive: rebalance_statuses, then set the value for that directive to some very rare HTTP status code(s) like 502. 2. Add a Valve/Filter/Whatever to Tomcat that, when activated because you want to take a node out of service) returns a 502 when a session identifier is used that does not map to a currently-valid session and there is no force-new-session header included in the request with some unpredictable value to prevent end-users from sending their own valid headers. This could also be handled by the same component (Manager) that current creates sessions with a new configuration option. These two together would then bleed-off the aforementioned users: in step 5, the request would be routed to Tomcat X, then rejected with a 502. mod_jk would detect this and re-balance the worker. If mod_jk can't re-balance the worker to another backend (e.g. they are all disabled, stopped, or in a failure state), then mod_jk would send the request (back) to Tomcat X with the force-new-session header set. Other than the above (or something similar), or actively probing a backend Tomcat to see if a session is valid, does anyone have any ideas for how to get a backend Tomcat into a completely idle state before pulling the plug? The status code (plus header) idea could be done. Probably not to difficult. Will not necessarily work for POST, but will suffice in most cases. The check-valid-session stuff would be harder. First we need to run an http request on our own (which would be useful also for regular probing), but then it is probably also a performance issue. We can't simply piggyback the session id on cping/cpong, because the container needs to at least get the URL to determine the correct context and run the mapper. IMHO to much to force on every request, even for a node with activation disabled. See also my other response. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Read-only mod_jk jk-status?
On 13.06.2014 19:03, Christopher Schultz wrote: All, I'm interested in locking-down my jk-status page so that certain users can view the information but not modify it. Unfortunately, the jk-status page is implemented using a single URL as a controller with GET-parameters controlling what actually happens. Even the edit worker page uses GET instead of POST, so I can't just disable POST for all but some blessed set of users. Does anyone have any suggestions for how jk-status could be locked-down? I'm guessing that a whole lot of mod_rewrite rules could do the trick by looking for certain write operations and rejecting them, but that would mean being very careful about a lot of magic that's being sent-around in URL parameters. Has anyone done anything like this before? It's a build in feature, set the read_only attribute of that status worker to true. You can even have multiple status workers, like one read-write and one read-only. For instance the worker.properties in the source code release of mod_jk has: http://svn.apache.org/viewvc/tomcat/jk/trunk/conf/workers.properties?view=co # Define two status worker: # - jk-status for read-only use # - jk-manager for read/write use worker.list=jk-status worker.jk-status.type=status worker.jk-status.read_only=true worker.list=jk-manager worker.jk-manager.type=status That means whatever URL you mount to the worker jk-status will be read-only and whatever url you mount to jk-manager will be read-write. You can choose those names and also the URLs arbitrarily as long as that snippet stays consistent. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk warning about OPTIONS * requests
On 22.04.2014 17:32, André Warnier wrote:
Christopher Schultz wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
André,
On 4/22/14, 5:09 AM, André Warnier wrote:
Christopher,
Christopher Schultz wrote:
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256
All,
I'm running some load tests in a dev environment, and I'm seeing
acess log messages like these:
::1 - - [21/Apr/2014:14:15:03 +] OPTIONS * HTTP/1.0 200 -
- Apache/2.4.9 (Amazon) OpenSSL/1.0.1g-fips PHP/5.5.10
mod_jk/1.2.40 (internal dummy connection)
Paired with mod_jk errors like these:
[Mon Apr 21 14:15:03.359 2014] [23925:3072505728] [warn]
map_uri_to_worker_ext::jk_uri_worker_map.c (1096): Uri * is
invalid. Uri must start with /
This happens to be running Apache httpd 2.4.9 with mod_jk 1.2.40,
but I can see similar messages in an otherwise
identically-configured server running httpd 2.2.23 and mod_jk
1.2.37:
== access.log == ::1 - - [21/Apr/2014:14:20:24 +] OPTIONS
* HTTP/1.0 200 - - Apache/2 (internal dummy connection)
== mod_jk.log == [Mon Apr 21 14:20:24.159 2014]
[12384:3072042880] [warn] map_uri_to_worker_ext::jk_uri_worker_map.c
(1057): Uri * is
invalid. Uri must start with /
In my httpd 2.4 environment, I seem to be seeing them much more
regularly (once every second or two) than in my httpd 2.2
environment (maybe every minute or two). If possible, I'd like to
minimize the amount of stuff that goes into my mod_jk.log file,
and since this is logged at the WARN level, it's tough to justify
disabling it.
Now, I realize that the OPTIONS * messages are httpd's internal
monitoring queries to see if workers are serviceable, so that's
not my question. My question is why mod_jk complains about them.
Perhaps a URI of * itself is invalid, but since httpd is known
to issue internal queries like this, why doesn't mod_jk simply
ignore them?
Just to keep the church in the middle of the village, as they say
in my part of the world :
[OT] I'm not sure I can interpret the meaning of that. What does it mean?
1) OPTIONS * is a valid HTTP request. [...]
Yep, I get all this. My real question is why mod_jk is complaining
about the URI. It shouldn't.
Here's what happens in mod_jk:
httpd
- - mod_jk.c::jk_handler
- - jk_uri_worker_map.c::map_uri_to_worker_ext
Here's the code for map_uri_to_worker_ext, around the interesting part:
jk_uri_worker_map.c::map_uri_to_worker_ext
1056if (*uri != '/') {
1057jk_log(l, JK_LOG_WARNING,
1058Uri %s is invalid. Uri must start with /, uri);
1059JK_TRACE_EXIT(l);
1060return NULL;
1061}
When this happens, jk_handler ultimately returns DECLINED. So,
everything works the way it should work (except possibly a bug where
Tomcat can't receive requests for OPTIONS *)
exactly; that is most probably a bug.
except that mod_jk
complains about a URI that it should not complain about.
I'm wondering if there's a particular reason to complain about this
URI,
Maybe it just simplifies the code after that, which compares the mod_jk
URI-mappings to the received request URI.
especially given that httpd (to which mod_jk is rather tightly
bound) uses these requests internally for legitimate reasons. To me,
it just seems like this message should be suppressed entirely.
The real basic issue here seems to be that mod_jk /misinterprets/
these OPTIONS requests : it takes itself the decision to reject it
and log an error, because it misinterprets the fact that, for
OPTIONS, * is a valid path.
All correct, except that mod_jk returns DECLINED instead of ERROR.
I don't think it should do that.
+1
That it then logs an error is a side-effect of this
misinterpretation. Stopping mod_jk from logging this error would
just hide the underlying issue.
No. It would just shut it up and everything would still work fine.
It would be mildly interesting, in comparison, to know how the
mod_proxy_ajp developers handle this case.
(*) For example, if you used a Location(Match) section with a
SetHandler jakarta-servlet, and in that Location, excluded the
OPTIONS requests (via a filter?). Or if you used mod_rewrite to set
the no-jk httpd env value for OPTIONS requests. Unfortunately, I
think that there is no corresponding JkUnMount syntax which
allows to exclude some requests based on the HTTP request Verb.
I believe that to be correct. I exclusively use JkMount, so the
OPTIONS * should never have mod_jk explicitly set as a request
handler... httpd is just apparently consulting mod_jk for all
requests. That's a bit unexpected, honestly (I thought mod_jk could
register itself as a request handler for certain URIs, but in
retrospect that would be pretty chaotic), but the situation is fairly
clear.
I would simply like to remove the warning, or maybe place it at a
lower logging level. It would be nice if OPTIONS * requests could be
forwarded-through to Tomcat, but it appears that it not
Re: ColdFusion10 custom mod_jk difference
On 08.04.2014 23:44, Doug Strick wrote: We're moving from ColdFusion8 to CF10 where I work and ran into a strange issue. We tried using mod_jk-1.2.39 and it compiled fine. We were able to get the communication working, but ran into strange errors like below. Adobe provides their own customized version of mod_jk which appears to be built from 1.2.32. When I compiled their version from the source they provide our errors went away. I downloaded the source from here if anyone's interested: http://helpx.adobe.com/coldfusion/kb/rhel-connector-configuration.html. I'd like to avoid using their custom version as I don't know how it will play if other non-ColdFusion based apps want to use AJP in the future. Does anyone have any recommendations on how I might be able to figure out what was changed? I'm not a developer so I don't know much at the code level. [Fri Apr 04 15:22:49 2014] [9753:139964571830064] [debug] ajp_send_request::jk_ajp_common.c (1713): (cfusion) request body to send 0 - request body to resend 0 [Fri Apr 04 15:22:49 2014] [9753:139964571830064] [debug] ajp_connection_tcp_get_message::jk_ajp_common.c (1403): received from ajp13 pos=0 len=14 max=65536 [Fri Apr 04 15:22:49 2014] [9753:139964571830064] [debug] ajp_connection_tcp_get_message::jk_ajp_common.c (1403): 0F 00 0A 2F 69 6E 64 65 78 2E 68 74 6D 00 00 00 - .../index.htm ... [Fri Apr 04 15:22:49 2014] [9753:139964571830064] [error] ajp_process_callback::jk_ajp_common.c (2071): Unknown AJP protocol code: 0F [Fri Apr 04 15:22:49 2014] [9753:139964571830064] [info] ajp_service::jk_ajp_common.c (2669): (cfusion) sending request to tomcat failed (recoverable), because of server error (attempt=2) [Fri Apr 04 15:22:49 2014] [9753:139964571830064] [debug] jk_shutdown_socket::jk_connect.c (840): About to shutdown socket 25 [ 172.16.113.55:49315 - 10.9.49.245:51010] [Fri Apr 04 15:22:49 2014] [9753:139964571830064] [debug] jk_is_input_event::jk_connect.c (1287): timeout during poll on socket 25 [ 172.16.113.55:49315 - 10.9.49.245:51010] (timeout=2) [Fri Apr 04 15:22:49 2014] [9753:139964571830064] [debug] jk_shutdown_socket::jk_connect.c (922): Shutdown socket 25 [ 172.16.113.55:49315 - 10.9.49.245:51010] and read 23848 lingering bytes in 0 sec. [Fri Apr 04 15:22:49 2014] [9753:139964571830064] [error] ajp_service::jk_ajp_common.c (2689): (cfusion) connecting to tomcat failed. [Fri Apr 04 15:22:49 2014] [9753:139964571830064] [debug] ajp_reset_endpoint::jk_ajp_common.c (810): (cfusion) resetting endpoint with socket -1 (socket shutdown) [Fri Apr 04 15:22:49 2014] [9753:139964571830064] [debug] ajp_done::jk_ajp_common.c (3140): recycling connection pool for worker cfusion and socket -1 [Fri Apr 04 15:22:49 2014] [9753:139964571830064] [info] jk_handler::mod_jk.c (2806): Service error=-3 for worker=cfusion The code changes show that Adobe has decided to not only add a feature (max reuse count per connection), but also to extend the protocol which makes standard mod_jk incompatible. The Unknown AJP protocol code: 0F is due to that addition. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Configuring mod_jk with multiple Apache HTTPD Virtual Hosts
On 04.03.2014 23:09, Doug Strick wrote: The F5 issues were just due to poor environment configuration. Each F5 VIP was sending traffic to the same pool and that pool was only configured for 1 member. That 1 member IP/port was used by several apache virtual hosts. So basically I never knew which virtual host was getting the request which meant some requests were going to listeners not running mod_jk. Always fun joining a new company and having to piece together what someone else has done. Please remember that over 90% of these configs were created by the Adobe CF webserver config utility so I question some of it as well. Here is what's being used in the uriworkermap.properties: /cfformgateway/* = cfusion /CFFormGateway/* = cfusion /flex2gateway/* = cfusion /flex2gateway = cfusion /cffileservlet/* = cfusion /CFFileServlet/* = cfusion /cfform-internal/* = cfusion /flashservices/gateway/* = cfusion /flex-internal/* = cfusion /rest/* = cfusion /*.cfml/* = cfusion /*.mxml = cfusion /*.as = cfusion /*.cfm = cfusion /*.cfm/* = cfusion /*.swc = cfusion /*.cfml = cfusion /*.cfc = cfusion /*.cfc/* = cfusion /*.cfr = cfusion /*.cfswf = cfusion /*.sws = cfusion /*.jsp = cfusion /*.hbmxml = cfusion This is what I'm seeing in the mod_jk log. I've cut out a few sections of the Attempting to map context URI so there's less clutter. [Tue Mar 04 16:36:50 2014] [5763:140265396258560] [debug] map_uri_to_worker_ext::jk_uri_worker_map.c (1131): Attempting to map URI '/' from 24 maps [Tue Mar 04 16:36:50 2014] [5763:140265396258560] [debug] jk_translate::mod_jk.c (3723): no match for / found [Tue Mar 04 16:36:50 2014] [5763:140265396258560] [debug] map_uri_to_worker_ext::jk_uri_worker_map.c (1131): Attempting to map URI '/' from 24 maps [Tue Mar 04 16:36:50 2014] [5763:140265396258560] [debug] jk_map_to_storage::mod_jk.c (3798): no match for / found [Tue Mar 04 16:36:50 2014] [5763:140265396258560] [debug] map_uri_to_worker_ext::jk_uri_worker_map.c (1131): Attempting to map URI '/index.cfm' from 24 maps [ [Tue Mar 04 16:36:50 2014] [5763:140265396258560] [debug] find_match::jk_uri_worker_map.c (958): Found a wildchar match '/*.cfm=cfusion' [Tue Mar 04 16:36:50 2014] [5763:140265396258560] [debug] jk_handler::mod_jk.c (2621): Into handler jakarta-servlet worker=cfusion r-proxyreq=0 [Tue Mar 04 16:36:50 2014] [5763:140265396258560] [debug] wc_get_worker_for_name::jk_worker.c (115): found a worker cfusion [Tue Mar 04 16:36:50 2014] [5763:140265396258560] [debug] wc_get_name_for_type::jk_worker.c (292): Found worker type 'ajp13' [Tue Mar 04 16:36:50 2014] [5763:140265396258560] [debug] init_ws_service::mod_jk.c (1097): Service protocol=HTTP/0.9 method=GET ssl=false host=(null) addr=192.168.253.3 name=app1.dev5.abc.com port=80 auth=(null) user=(null) laddr=192.168.253.61 raddr=192.168.253.3 uri=/ [Tue Mar 04 16:36:50 2014] [5763:140265396258560] [debug] ajp_get_endpoint::jk_ajp_common.c (3161): acquired connection pool slot=0 after 0 retries [Tue Mar 04 16:36:50 2014] [5763:140265396258560] [debug] ajp_marshal_into_msgb::jk_ajp_common.c (626): ajp marshaling done [Tue Mar 04 16:36:50 2014] [5763:140265396258560] [debug] ajp_service::jk_ajp_common.c (2450): processing cfusion with 2 retries [Tue Mar 04 16:36:50 2014] [5763:140265396258560] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1184): sending to ajp13 pos=4 len=185 max=8192 [Tue Mar 04 16:36:50 2014] [5763:140265396258560] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1184): 12 34 00 B5 02 02 00 08 48 54 54 50 2F 30 2E 39 - .4..HTTP/0.9 [Tue Mar 04 16:36:50 2014] [5763:140265396258560] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1184): 001000 00 01 2F 00 00 0D 31 39 32 2E 31 36 38 2E 32 - .../...192.168.2 [Tue Mar 04 16:36:50 2014] [5763:140265396258560] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1184): 002035 33 2E 33 00 FF FF 00 22 63 6F 6D 6D 65 72 63 - 53.3app1 [Tue Mar 04 16:36:50 2014] [5763:140265396258560] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1184): 003065 2E 64 65 76 35 2E 6C 69 66 65 74 65 63 68 6E - dev5.abc [Tue Mar 04 16:36:50 2014] [5763:140265396258560] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1184): 00406F 6C 6F 67 69 65 73 2E 63 6F 6D 00 00 50 00 00 - .com..P.. [Tue Mar 04 16:36:50 2014] [5763:140265396258560] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1184): 005002 A0 09 00 27 54 4C 54 53 49 44 3D 31 38 36 35 - 'TLTSID=1865 [Tue Mar 04 16:36:50 2014] [5763:140265396258560] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1184): 006037 36 41 32 41 33 45 35 31 30 41 33 30 30 30 33 - 76A2A3E510A30003 [Tue Mar 04 16:36:50 2014] [5763:140265396258560] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1184): 007045 43 39 33 42 35 31 31 39 39 42 43 00 A0 08 00 -
Re: tomcat 6 refuses mod_jk connections after server runs for a couple of days
On 27.02.2014 23:06, Isaac Gonzalez wrote: Hi Christopher(and Konstantin), attached is a couple of thread dumps of when we experienced the issue again today. I also noticed we get this message right before the problem occurs: Feb 27, 2014 12:47:15 PM org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run SEVERE: Caught exception (java.lang.OutOfMemoryError: unable to create new native thread) executing org.apache.jk.common.ChannelSocket$SocketAcceptor@177ddea, terminating thread Is it a 32Bit system? You have 2GB of heap plus Perm plus native memory needed by the process plus thread stacks. Not unlikely, that you ran out of memory address space for a 32 bit process. The only fixes would then be: - switch to a 64 bit system - reduce heap if the app can work with less - improve performance or eliminate bottlenecks so that the app works with less threads - limit you connector thread pool size. That will still mean that if requests begin to queue because of performance problems, the web server can't create additional connections, but you won't get in an irregular situation as you experience now. In that case you would need to configure a low idle timeout for the connections on the JK and TC side. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Status of mod_jk-1.2.38
On 08.01.2014 18:25, Martin Knoblauch wrote: Hi, happy New year. Should be still early enough for this :-) I have one short question: what is the status of mod_jk-1.2.38? It has been over a year since 1.2.37 and I am just curious. I am specially interested in an official fix for *Bug 53762*https://issues.apache.org/bugzilla/show_bug.cgi?id=53762. The patch mentioned there works, but cannot be used due to an only official releases policy at the customer site. I did some more changes lately due to a user request which need some more testing. I expect that we'll do the 1.2.38 release during the next 4-8 weeks. I have to ask Mladen about the status of his IPv6 changes though. On a related note: is mod_jk still the preferred way for HA and load-balancing tomcats? Are there better ways? Our environment is currently httpd/mod_jk/tomcat on Linux. Better or worse depend on your requirements. I think mod_jk is still working very well and has some unique advantages. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: max_packet_size for data in mod_jk
On 02.01.2014 15:42, frenchc44 wrote: Thanks Rainer. To be honest, we don't really know what to expect from a larger packet size, but we think it could only help since it would reduce round trips between apache/tomcat. My main objective with this thread is to confirm my suspicion that the max_packet_size is not for request post data and rather for header. Confirmed, used for request header and for response packets. I added it today also for request bodies, but did not yet really test it. It will be part of version 1.2.38. It would be nice if you could give it a try: http://people.apache.org/~rjung/mod_jk-dev/tomcat-connectors-1.2.38-dev-1555414-src.tar.gz That is not a release but just a snapshot tarball, that can be build just like a release. Thanks for pushing it until here and possible further tests and feedback. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: max_packet_size for data in mod_jk
On 31.12.2013 18:44, frenchc44 wrote:
Due to large payloads, we wanted to increase the max_packet size of the
mod_jk/ajp layer in order to see if it will help with performance. However,
it appears the max_packet_size for non- header requests is capped at 8192
bytes despite the documentation stating that it should be 65536. I am using
mod_jk version 1.2.31
[Tue Dec 31 11:54:09 2013][12492:12640] [debug]
ajp_connection_tcp_send_message::jk_ajp_common.c (1145): sending to ajp13
pos=4 len=8192 max=16384
Here is some code from jk_ajp_common.c that sets the max size but appears to
ignore the max_packet_size...
Line 1708: if (ae-left_bytes_to_send 0) {
int len = AJP13_MAX_SEND_BODY_SZ;
if (ae-left_bytes_to_send
(jk_uint64_t)AJP13_MAX_SEND_BODY_SZ) {
len = (int)ae-left_bytes_to_send;
}
Any help would be much appreciated.
That's quite possible. The original problem that was solved with
max_packet_size was big headers, because the AJP protocol needs to fit
all headers into one AJP packet. The problem arose when people were
using https and client certs and wanted to forward client cert details
from Apache to Tomcat. Then often 8KB were not enough.
Are you sure, that bigger AJP packets will be a significant win in your
situation? Request and response bodies are streamed, so there is no
obvious limitation and packets will be relayed as data comes in.
I'm willing to look into how easy it would be to making max_packet_size
work even for other packets (in combination with packetSize on the
Tomcat side), but would like to first understand better why you think it
would make a noticeable difference for you.
Regards,
Rainer
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Ping_mode
On 26.12.2013 20:15, vicky007aggar...@yahoo.co.in wrote: Hi guys, Can you plz help me in understand that under what circumstances i should use ping_mode attribute in worker.properties. Actually i dont understand that how it will make the applications more reliable because as per my understanding if apache-tomcat connection results in failure then mod_jk will fallback try connecting with another tomcat jvm. when this is happening automatically then what is the point of configuring ping_mode Please enlighten me AJP ping/pong is a fast way of detecting low level connection problems and also some higher level problems. Reliable detection of problems must come before any retry or failover decision mod_jk can do. AJP ping/pong is especially interesting when a firewall sits between the web server and Tomcat. Depending on your connection_pool_timeout and firewall configuration, TCP idle connection drop could happen there which often results in problems if there is no good connection checking enabled - like AJP ping/pong. There are other - more rare cases - where AJP ping/pong is of help. I generally use it as a basic mechanism to make the web server to Tomcat integration more robust. For more details see http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Changelog entry missing for 7.0.47
On 01.11.2013 18:01, Timothy Astle wrote: I don't see an entry in the changelog for 7.0.47? Is it going to be updated? http://tomcat.apache.org/tomcat-7.0-doc/changelog.html I can see it right now. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache 7, Logrotate
On 25.10.2013 11:54, Konstantin Kolinko wrote: 2013/10/25 Web2 Solutions m...@web2-solutions.com: Hallo All, I've installed tomcat 7.0.42 and due heavy use my catalina.out and localhost_access-jjj-mm-dd.txt grows quit big. I've successfully configured logrotate to rotate both files. I've removed the date from the access log. So Tomcat now writes without rotating into localhost_access.txt Logrotate now create a new file (localhost_access-dd-mm-dd.txt) and makes localhost_access.txt empty. But tomcat now writes into the new localhost_access-dd-mm-dd.txt instead of the configured file (localhost_access.txt). FYI: You can configure the filename pattern so that it rotates more frequently, e.g. every hour or every ten minutes. What do I have todo so that tomcat continues to write into localhost_access.txt even after rotating? Renaming the file is futile, because Tomcat (for access logs) or the shell (for catalina.out) has the file open and continues to write to it, regardless of the file name. You can use copytruncate option of logrotate. FYI: catalina.out is not a proper log file, but a redirection of stdout (as managed in catalina.sh script that launches Tomcat java process). If a system is configured properly, this file is usually empty. Only for the access log: there's a property checkExists=true, that will close the file and reopen it if the access log valve detects the file has been moved/renamed. That option could be more expensive though then just using an appropriate value for fileDateFormat. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: libapr-1.so.0 not deployed for Tomcat 7
On 03.08.2013 06:06, srinivas yelamanchili wrote: Thanks Chris I installed the APR with --prefix=/apps/mstrat/apache/apr-1.4.8 The tomcat native configure had this parameter '--with-apr=/apps/mstrat/apache/apr-1.4.8' , so if it's building libtcnative i would expected it pick the libapr-1.so.0 from this apr folder just to be sure that the libtcnative and libapr in the deployed catalina lib folder are on the same page. I however noted your point (and valid) and will copy the libapr from APR lib manually to catalina lib In the world of native libraries --prefix=/apps/mstrat/apache/apr-1.4.8 isn't sufficient to install. The library needs to be found by the runtime linker. So you have several options: - install it into a system default library path - install it anywhere but configure the runtime linker to look for libs in that place as well - install it anywhere and set the appropriate environment variable to inform the runtime linker about this place (for your platform LD_LIBRARY_PATH). These are all general Unix/Linux things not really special to Tomcat Native. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache Tomcat Connector - configuration
On 29.07.2013 03:49, srinivas yelamanchili wrote: Hi, I am trying to configure Apache httpd talk to Tomcat From the documentation at http://tomcat.apache.org/connectors-doc/webserver_howto/apache.html 1. Under 'Using Tomcat auto-configure' we have this line creating your workers.properties file at $TOMCAT_HOME/conf/jk/workers.properties Under 'Simple configuration example' we have this line # Where to find workers.properties JkWorkersFile /etc/httpd/conf/workers.properties For the auto-configuration the worker properties file path is in TOMCAT_HOME path but for custom it's in HTTPd path? What's the recommended path? AFAIK auto configuration does not work any more and never did very well. 2. Under 'Using Tomcat auto-configure' a) The simplest way to configure Apache HTTP Server to use mod_jk is to turn on the Apache HTTP Server auto-configure setting in Tomcat How to turn on the auto-configure setting in Tomcat? b) Please note that this example is specific to Tomcat 5.x, unlike other sections of this document which also apply to previous Tomcat branches. This statement may need to be updated with respect to latest Tomcat 7.0? c) Listener className=org.apache.jk.config.ApacheConfig modJk=/path/to/mod_jk.so / Here '/path/to' is not actual folders so should be italicized to mean as a pseudo-code? AFAIK auto configuration does not work any more and never did very well. Start with the configuration examples distributed with the mod_jk source distribution (and don't take the minimal config file). 3. Under 'Simple configuration example' In which file (name) should the given configuration be saved to and under location HTTPd or TOMCAT_HOME path? Start with the configuration examples distributed with the mod_jk source distribution (and don't take the minimal config file). There's a httpd-jk.conf there which should be included in the httpd.conf and a workers.properties which is referenced from httpd-jk.conf. 4. Under 'Introduction' There is actually three versions of Apache HTTP Server, 1.3, 2.0 and 2.2 and all can be used with mod_jk, the Tomcat redirector module. comments: 2.4 is missing and 'three' is no longer accurate 'There is' should be 'There are' That was fixed recently in the docs sources and will be published with 1.2.38. Apache 2.4 is supported since some time now. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: The Apache Tomcat Connector - Webserver HowTo documentation for Apache Httpd 2.4.x and Tomcat 7
On 29.07.2013 17:26, Nicholas Williams wrote: On Mon, Jul 29, 2013 at 10:18 AM, Christopher Schultz ch...@christopherschultz.net wrote: If it's a resources limitation, I have a publicly-accessible TeamCity server with an unlimited OpenSource license hosting Windows 7, Mac OS X SLeopard/Lion/MLion, Debian, RedHat, and SuSE agents that I would be happy to donate some resources from. It's not very busy and would have plenty of time to run CI, SNAPSHOT, and RC builds for all of the platforms. No, the problem is that there are so many different combinations of platform, environment, etc. that it would represent an explosion of options that would never meet everyone's needs. Building mod_jk just isn't that difficult. The only legitimate complaint that I have heard is that most responsible admins don't have a build chain available on a production server. We solve that by building on a test server and pushing the binaries out to our production servers. Others may do other things. I do know that Debian-based distros of Linux can install the libapache2-mod-jk package, though it is often out-of-date with respect to the currently-available version. Inexplicably, Red Hat does not provide mod_jk binaries through their package manager. I'm not sure about Suse and others. Understood. I don't know about others, but SuSE DOES provide mod_jk in their package manager. That's how I installed it. Even so, it's rarely even close to up-to-date. Usually about a year or two behind. We had times where we did provide some Linux binaries. Development for mod_jk has slowed down, many distributions contain a reasonable version of it and no one in the team is really interested in providing binaries so it wasn't a priority. It surely is a mixture of time and technical resources but probably even more we don't feel it's an important itch to scratch. I wouldn't oppose if someone stood up and wanted to provide binaries, but it should be someone in our web of trust, because we can't validate binaries we get from outside. So we don't want to officially distribute contributed binaries. If there were such a person, she might happily comeback to your offer concerning build platforms though the ASF has quite a few build servers as well. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: The Apache Tomcat Connector - Webserver HowTo documentation for Apache Httpd 2.4.x and Tomcat 7
On 29.07.2013 22:36, Christopher Schultz wrote: Rainer, On 7/29/13 1:22 PM, Rainer Jung wrote: On 29.07.2013 17:26, Nicholas Williams wrote: On Mon, Jul 29, 2013 at 10:18 AM, Christopher Schultz ch...@christopherschultz.net wrote: If it's a resources limitation, I have a publicly-accessible TeamCity server with an unlimited OpenSource license hosting Windows 7, Mac OS X SLeopard/Lion/MLion, Debian, RedHat, and SuSE agents that I would be happy to donate some resources from. It's not very busy and would have plenty of time to run CI, SNAPSHOT, and RC builds for all of the platforms. No, the problem is that there are so many different combinations of platform, environment, etc. that it would represent an explosion of options that would never meet everyone's needs. Building mod_jk just isn't that difficult. The only legitimate complaint that I have heard is that most responsible admins don't have a build chain available on a production server. We solve that by building on a test server and pushing the binaries out to our production servers. Others may do other things. I do know that Debian-based distros of Linux can install the libapache2-mod-jk package, though it is often out-of-date with respect to the currently-available version. Inexplicably, Red Hat does not provide mod_jk binaries through their package manager. I'm not sure about Suse and others. Understood. I don't know about others, but SuSE DOES provide mod_jk in their package manager. That's how I installed it. Even so, it's rarely even close to up-to-date. Usually about a year or two behind. We had times where we did provide some Linux binaries. Development for mod_jk has slowed down, many distributions contain a reasonable version of it and no one in the team is really interested in providing binaries so it wasn't a priority. It surely is a mixture of time and technical resources but probably even more we don't feel it's an important itch to scratch. I wouldn't oppose if someone stood up and wanted to provide binaries, but it should be someone in our web of trust, because we can't validate binaries we get from outside. So we don't want to officially distribute contributed binaries. If there were such a person, she might happily comeback to your offer concerning build platforms though the ASF has quite a few build servers as well. Don't forget about the long internal argument(s) about building on untrusted machines, etc. How far does the ASF trust its contributors to produce binaries? That's why I wrote someone in our web of trust and So we don't want to officially distribute contributed binaries. ;) Typically our web of trust would mean committers, but in some situations it might be a slightly different group. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: The Apache Tomcat Connector - Webserver HowTo documentation for Apache Httpd 2.4.x and Tomcat 7
On 25.07.2013 17:14, srinivas yelamanchili wrote: Hi, I installed Apache Httpd 2.4.6 and Tomcat 7.0.42 from source code (.tar.gz) on Redhat Linux and looking for documentation to enable AJP and connect Tomcat with Httpd using AJP/APR I have the following questions (and please let me know where I can post the same if this email group is the not the correct place) : 1. The 'Tomcat Connectors JK 1.2' Binary Releases at http://tomcat.apache.org/download-connectors.cgi only seem to be available for Windows (and not for Redhat Linux). Aren't there any binary releases for unix based systems? They are very easy to build on Linux. Since there are so many distributions we currently do not provide official binaries. 2. The following documentation seem outdated, has references to Tomcat 3,4,5.5,6 but not 7. has references to Httpd 2.x in general and 2.2 specific but nothing specific to 2.4 The Apache Tomcat Connector - Webserver HowTo http://tomcat.apache.org/connectors-doc/webserver_howto/apache.html http://tomcat.apache.org/connectors-doc-archive/jk2/jk/aphowto.html 'Tomcat 7' doesn't appear under the 'Supported Configuration' section With Tomcat 7 could the means of using and configuring JK been updated or enhanced? I added TC 7 to the docs right now. That was an oversight. mod_jk and AJP was supported there from day 1 (as will TC 8 be once it gets released). Note that the jk2 docs are in the doc-archive. jk2 is a variant that is out of support since many years. Only use mod_jk, not mod_jk2. The Tomcat 7 doc mentions JK 1.2.x though http://tomcat.apache.org/tomcat-7.0-doc/config/ajp.html 3. Under 'Obtaining mod_jk' section of the 'Webserver HowTo' page If the binary is not available, follow the instructions for building mod_jk from source Where are the instructions? I added a hint to the docs. It's on the same page futher down in the seciotns starting with Building mod_jk. 4. Can the documentation pages have timestamp of 'last modified'? Not really. The docs published on the web site always belong to the latest mod_jk release, currently version 1.2.37. The starting page http://tomcat.apache.org/connectors-doc/ contains the date of the release prominently. The official repository for the docs sources is http://svn.apache.org/viewvc/tomcat/jk/trunk/xdocs/ or when using svn directly http://svn.apache.org/repos/asf/tomcat/jk/trunk/xdocs/ There you can always check the latest dates and the full change history. The changes I applied right now based on your advice are http://svn.apache.org/viewvc?view=revisionrevision=r1507026 Since they are not critical they will be pushed to the public web site once we release version 1.2.38. Thanks for haring your observations. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: JSVC - two processes running as root
On 18.07.2013 21:27, Shridev Makim wrote: Hello, We are trying to run tomcat7 as a daemon using JSVC in SunOS environment. The compiled jsvc executable is owned by root. We have modified daemon.sh that bundles with tomcat7 for our environment to run our tomcat instance. When we execute this shell script, it creates two processes and both the processes are owned by root as shown below (output of ps -ef | grep jsvc): root 9109 9108 2 15:15:03 ? 0:37 /doc/dmadmin6/product/tomcat7/bin/jsvc -java-home /doc/dmadmin6/product/jdk6/jd root 9108 1 0 15:15:03 ? 0:00 /doc/dmadmin6/product/tomcat7/bin/jsvc -java-home /doc/dmadmin6/product/jdk6/jd We are running the shell script as a non privileged user (dmadmin6) and we are even passing this user name with -user switch to jsvc. We were expecting the child process (pid 9109) to be run as a non privileged user (dmadmin6) in our case. Anyone else has experienced this? Currently in production we have tomcat 6 configured to run with jsvc and in that environment we see that the child process is running as a non privileged user. Thanks in advance for your help! Might not be related, but: Default shell on Solaris produces two processes when a command is called via sh -c somecommand arg1 arg2 One process is the shell, the second one the command after -c. This differs from Linux where you will only see the second one. /usr/bin/sh shows the behaviour, /usr/bin/ksh and /usr/xpg4/bin/sh do not. You could try to run everything with one of those other two shells to quickly check whether that fixes the problem. You'll have to experiment a bit, which shell(s) to switch, the one that executes daemon.sh, the login shell of user dmadmin6 etc. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: JSVC - two processes running as root
On 24.07.2013 17:34, Rainer Jung wrote: On 18.07.2013 21:27, Shridev Makim wrote: Hello, We are trying to run tomcat7 as a daemon using JSVC in SunOS environment. The compiled jsvc executable is owned by root. We have modified daemon.sh that bundles with tomcat7 for our environment to run our tomcat instance. When we execute this shell script, it creates two processes and both the processes are owned by root as shown below (output of ps -ef | grep jsvc): root 9109 9108 2 15:15:03 ? 0:37 /doc/dmadmin6/product/tomcat7/bin/jsvc -java-home /doc/dmadmin6/product/jdk6/jd root 9108 1 0 15:15:03 ? 0:00 /doc/dmadmin6/product/tomcat7/bin/jsvc -java-home /doc/dmadmin6/product/jdk6/jd We are running the shell script as a non privileged user (dmadmin6) and we are even passing this user name with -user switch to jsvc. We were expecting the child process (pid 9109) to be run as a non privileged user (dmadmin6) in our case. Anyone else has experienced this? Currently in production we have tomcat 6 configured to run with jsvc and in that environment we see that the child process is running as a non privileged user. Thanks in advance for your help! Might not be related, but: Default shell on Solaris produces two processes when a command is called via sh -c somecommand arg1 arg2 One process is the shell, the second one the command after -c. This differs from Linux where you will only see the second one. /usr/bin/sh shows the behaviour, /usr/bin/ksh and /usr/xpg4/bin/sh do not. You could try to run everything with one of those other two shells to quickly check whether that fixes the problem. You'll have to experiment a bit, which shell(s) to switch, the one that executes daemon.sh, the login shell of user dmadmin6 etc. Sorry, that has likely not to do with it. A more structured approach: Process 9108 is the parent of 9109 so it seems the 9108 should go away and not stay in the process table. Use pstack 9108 (or whatever the current parent process PID is) and post the results here. That'll show us where it hangs. And probably it would be nice to use the latest version of jsvc so we don't need to debug old problems. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Fwd: Error starting httpd-2.4.4 with mod_jk on osx
On 07.07.2013 14:27, Thiyagaraj Annamalai wrote: Hi, I'm on OS X 10.8.4. I'm trying to configure mod_jk with httpd-2.4.4 and getting the below error when starting apache. httpd: Syntax error on line 500 of /usr/local/apache/conf/httpd.conf: Syntax error on line 2 of /usr/local/apache/conf/other/httpd-mine.conf: Cannot load /usr/libexec/apache2/mod_jk.so into server: dlopen(/usr/libexec/apache2/mod_jk.so, 10): Symbol not found: _ap_log_error\n Referenced from: /usr/libexec/apache2/mod_jk.so Expected in: flat namespace\n in /usr/libexec/apache2/mod_jk.so How can I fix this? Sounds like an issue due to mixture between Apache 2.2 and 2.4 ... I compiled httpd and mod_jk from the source and both the build went fine. Here's how I built and the conf info.: # compile instructions for httpd # apr, apr-util in srclib/ httpd-2.4.4$ ./configure --prefix=/usr/local/apache --with-included-apr --with- pcre=/usr/local httpd-2.4.4$ make httpd-2.4.4$ sudo make install # compile instructions for mod_jk tomcat-connectors-1.2.37-src$ cd native native$ ./configure CFLAGS='-arch x86_64' APXSLDFLAGS='-arch x86_64' --with-apxs=/usr/sbin/apxs Here's likely the reason. /usr/sbin/apxs was likely installed with some Apache 2.2. The correct apxs for your 2.4.4 should be somewhere under /usr/local/apache (bin or build). native$ make native$ sudo make install #Content of /usr/local/apache/conf/other/httpd-mine.conf NameVirtualHost *:80 LoadModule jk_module /usr/libexec/apache2/mod_jk.so I tried googling and found the above build instructions but still I get the error. Thanks in advance for the help! Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Abandoned apache children with mod_jk
On 22.06.2013 23:18, Chris Boyce wrote: Thank you so much for the reply. Here are a couple of examples, as I'm not completely sure if my symptoms match, though the pstacks do look very similar to my untrained eye: Here is a two day-old child: 27743: /usr/local/apache2/bin/httpd -k start - lwp# 1 / thread# 1 ff00a42c lwp_wait (3, ffbff804) ff001e88 _thrp_join (3, 0, ffbff86c, 1, ff0b2780, ffbff804) + 38 ff214544 apr_thread_join (ffbff8ec, 32eea8, 7, 0, dc328, b15e0) + c 0008c43c join_workers (0, fe3aa8, 8bfcc, 32ec30, 0, 1) + ec 0008c790 child_main (2, 8b31c, 0, feee2a40, ff0b2840, ff0b2780) + 270 0008c970 make_child (c7800, 2, 0, c8800, c7000, c8400) + 128 0008d1b4 ap_mpm_run (fe4100f8, e, 0, 1, 27, 1) + 754 000343c0 main (d6218, d8190, ffbffc54, c7800, c7800, 0) + 79c 00033754 _start (0, 0, 0, 0, 0, 0) + 5c - lwp# 3 / thread# 3 ff0058d4 lwp_park (0, 0, 0) fefff6e8 cond_wait_queue (32ecc8, 32ec98, 0, 0, 0, 0) + 4c fefffd30 cond_wait (32ecc8, 32ec98, 0, 0, fe460a40, 0) + 10 fefffd6c pthread_cond_wait (32ecc8, 32ec98, 0, 0, 32ec98, 0) + 8 0008e674 ap_queue_pop (32ec78, fe30bf1c, fe30bf18, 4, 0, 32ee40) + 64 0008be1c worker_thread (32eea8, 2, fe460a40, c8400, c8400, 0) + 10c ff21440c dummy_worker (32eea8, 0, 0, fe460a40, ff214400, 1) + c ff005850 _lwp_start (0, 0, 0, 0, 0, 0) - lwp# 4 / thread# 4 ff0058d4 lwp_park (0, 0, 0) fefff6e8 cond_wait_queue (32ecc8, 32ec98, 0, 0, 0, 0) + 4c fefffd30 cond_wait (32ecc8, 32ec98, 0, 0, fe461240, 11692d8) + 10 fefffd6c pthread_cond_wait (32ecc8, 32ec98, 0, 0, 32ec98, 0) + 8 0008e674 ap_queue_pop (32ec78, fe20bf1c, fe20bf18, 0, 0, 32ee40) + 64 0008be1c worker_thread (32eec8, 2, fe461240, c8400, c8400, 4) + 10c ff21440c dummy_worker (32eec8, 0, 0, fe461240, ff214400, 1) + c ff005850 _lwp_start (0, 0, 0, 0, 0, 0) ...and several more in lwp_park. The abopve one could be related to the cited BZ issue. And here's another one that's a day old, but looks different (including lots of jk references): 7934: /usr/local/apache2/bin/httpd -k start - lwp# 1 / thread# 1 ff00a42c lwp_wait (6, ffbff80c) ff001e88 _thrp_join (6, 0, ffbff874, 1, ff0b2780, ffbff80c) + 38 ff214544 apr_thread_join (ffbff8f4, 28e228, 2, 0, 1, b1600) + c 0008c43c join_workers (c, 3c5f38, 8bfcc, 28df50, 0, 1) + ec 0008c790 child_main (0, 8b31c, 0, feee2a40, ff0b2840, ff0b2780) + 270 0008c970 make_child (c7800, 0, 0, c8800, c7000, c8400) + 128 0008d1b4 ap_mpm_run (fe4100f8, e, 0, 1, 26, 1) + 754 000343c0 main (d6218, d8190, ffbffc5c, c7800, c7800, 0) + 79c 00033754 _start (0, 0, 0, 0, 0, 0) + 5c - lwp# 6 / thread# 6 ff00a14c read (15, fe00a908, 4) fe4a87dc jk_tcp_socket_recvfull (15, fe00a908, 4, 2e4bf8, 510, 4ec) + 74 fe4c3088 ajp_connection_tcp_get_message (35f130, 35f168, 2e4bf8, 361188, 2000, 2064) + 44 fe4c5588 ajp_get_reply (361168, fe00bb50, 2e4bf8, 35f130, fe00aa70, 2028) + 9c fe4c9304 ajp_service (361168, fe00bb50, 2e4bf8, fe00ab38, 1, c00) + 22b8 fe4a1234 jk_handler (23c, 35e740, 3f4390, 1, 13, 3544c8) + 9e4 00047534 ap_run_handler (3f40a0, 0, 11, 3e7028, 3f5a08, 0) + 3c 000479c0 ap_invoke_handler (3f40a0, 9d000, 3f40a0, 0, fe410028, 0) + c0 00073aa4 ap_process_request (3f40a0, 3, 4, 3f40a0, c8420, 21d8d8) + 160 00070b34 ap_process_http_connection (3d52e8, 3d5038, 3d5038, 3, c8420, 211980) + 10c 0004dce8 ap_run_process_connection (3d52e8, 3d5038, 3d5038, 3, 3d52e0, 3d7068) + 3c 0008bf1c worker_thread (28e228, 0, fe462240, c8400, c8400, c) + 20c ff21440c dummy_worker (28e228, 0, 0, fe462240, ff214400, 1) + c ff005850 _lwp_start (0, 0, 0, 0, 0, 0) - lwp# 7 / thread# 7 ff214400 dummy_worker(), exit value = 0x ** zombie (exited, not detached, not yet joined) ** - lwp# 8 / thread# 8 ff214400 dummy_worker(), exit value = 0x ** zombie (exited, not detached, not yet joined) ** - lwp# 9 / thread# 9 ff214400 dummy_worker(), exit value = 0x ** zombie (exited, not detached, not yet joined) ** - lwp# 10 / thread# 10 ff214400 dummy_worker(), exit value = 0x ** zombie (exited, not detached, not yet joined) ** - lwp# 11 / thread# 11 ff214400 dummy_worker(), exit value = 0x ** zombie (exited, not detached, not yet joined) ** - lwp# 12 / thread# 12 ff214400 dummy_worker(), exit value = 0x ** zombie (exited, not detached, not yet joined) ** - lwp# 13 / thread# 13 ff214400 dummy_worker(), exit
Re: Abandoned apache children with mod_jk
On 21.06.2013 19:47, Chris Boyce wrote: Hello, I'm running apache 2.2.24 (worker MPM) with mod_jk 1.2.37 under Solaris 11, compiled as follows (from config.log): --with-included-apr --with-mpm=worker --enable-so --enable-rewrite --enable-headers --enable-proxy --enable-proxy-http --enable-expires --enable-nonportable-atomics=yes --disable-include --disable-autoindex --disable-imap --disable-userdir CC=/usr/sfw/bin/gcc We are running Tomcat 7.0.32. Since moving to Solaris 11 I'm noticing over time that apache children are getting left in an idle state (and usually not showing up on the scoreboard at all) when doing graceful restarts. If I do a hard restart, the error_log notes that the process had to be forcibly killed: [Wed May 15 11:41:24 2013] [warn] child process 10057 still did not exit, sending a SIGTERM [Wed May 15 11:41:26 2013] [error] child process 10057 still did not exit, sending a SIGKILL If I let apache go unchecked, it will eventually stop passing traffic completely and a hard restart is required. Example ps output looks like this: nobody 24429 20925 0 11:43:59 ? 0:02 /usr/local/apache2/bin/httpd -k start nobody 9750 20925 0 23:59:02 ? 0:00 /usr/local/apache2/bin/httpd -k start nobody 20925 2440 0 May 15 ? 3:07 /usr/local/apache2/bin/httpd -k start nobody 24689 20925 0 11:47:52 ? 0:00 /usr/local/apache2/bin/httpd -k start nobody 24628 20925 0 11:46:18 ? 0:01 /usr/local/apache2/bin/httpd -k start nobody 24428 20925 0 11:43:39 ? 0:02 /usr/local/apache2/bin/httpd -k start Note PID 9750 is lingering, doing nothing according to pfiles and truss, and its timestamp coincides with the last graceful restart (log rotation). Two main differences between this web server and ones that are working include: a) This is Solaris 11 (vs. Solaris 10) b) I have hardened apache by putting it in a Solaris 11 zone, and I'm starting apache as the nobody user with the net_privaddr privilege so it can function as the parent process. It talks to Tomcat on another zone and everything works great (other than the problem described here). Apache has permission to write to /logs, and /log/apache2 is where I set these: JkLogFile /logs/apache2/mod_jk.log JkShmFile /logs/apache2/jk-runtime-status And this. PidFile /logs/apache2/run/httpd.pid Can anyone think of a reason why children are not being recycled or getting stranded like this over successive graceful restarts? We do use multiple listeners, so I don't know if I'm dealing with a locking/mutex/serialization type of issue. I'm not a C programmer. There seems to be little info out there for Solaris platforms that's recent. I'd be happy to post more info if needed. I appreciate your time. What does pstack show for such an abandoned child? Maybe another occurance of https://issues.apache.org/bugzilla/show_bug.cgi?id=49504. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk worker activation not working anymore in 1.2.37?
On 10.06.2013 17:29, Konstantin Kolinko wrote: 2013/6/10 David Gubler d...@doodle.com: Hi list, We have recently upgraded our Apache servers from Debian Squeeze to Wheezy (from Apache 2.2.16 with mod_jk 1.2.30 to Apache 2.2.22 with mod_jk 1.2.37). The Tomcat version hasn't changed (7.0.37). We often do rolling releases by disabling (DIS) some worker in jkmanager, waiting for a few minutes for most sessions to go away (we use sticky sessions but not forced), upgrading it, and re-enabling it. This worked perfectly with mod_jk 1.2.30. The server is rather busy (order of 100-200 req/s going to tomcat). However, with mod_jk 1.2.37, the activation state behaves erratically. Say I disable worker1 on all apache servers via jkmanager. When I go back to the jkmanager overview screen, it still shows as active. I hit reload, now it shows as disabled. I can wait for a few seconds or minutes, reload, and suddenly it shows up as active again! It keeps switching back and forth between active and disabled if I reload often enough. Afterwards I usually have to set it to active a few times to make it stick there. This happens on all apache servers independently. And more worringly, the load on the worker does not decrease, not even after waiting for half an hour or longer (with 1.2.30, the load on a worker decreased to about 5% after 5-10 minutes). When I set a worker to stopped, the activation state also switches between active and stopped, the load on the worker goes down slowly, but the requests do not cease completely. With 1.2.30, I could set a worker to stopped and it instantaneously received no more requests. Other than that, mod_jk behaves as expected (e.g. if I shut down one of the tomcats, the requests go to the other; load balancing works fine in normal operation). I have stripped down our workers.properties to the bare minimum that we need, and the problem is still there: ps=/ worker.list=loadbalancer,jkstatus worker.jkstatus.type=status worker.loadbalancer.type=lb worker.loadbalancer.sticky_session=true worker.loadbalancer.balance_workers=worker1,worker2 worker.worker1.type=ajp13 worker.worker1.host=WW.XX.YY.ZZ worker.worker1.port=8009 worker.worker1.connect_timeout=7 worker.worker1.prepost_timeout=7 worker.worker1.socket_timeout=70 worker.worker1.connection_pool_timeout=70 worker.worker1.connection_pool_size=200 worker.worker1.retry_interval=1000 worker.worker1.lbfactor=1 [same for worker2, only difference is the IP address] Rest of the configuration is Debian standard. Apache uses JkAutoAlias, JkMount and a bunch of JkUnMounts, but nothing fancy. The changelog does not really give me any clues as to what change could cause this, and neither does the workers.properties documentation :( Does anyone have an idea what I could be doing wrong? Looking at the current changelog, section name=Changes between 1.2.37 and 1.2.38 ... fix Fix status worker not updating parameters for all members. (mturk) /fix That is http://svn.apache.org/viewvc?view=revisionrevision=1354021 Yes that should be it. If the OP compiles himself, just add the tiny patch http://svn.apache.org/viewvc/tomcat/jk/trunk/native/common/jk_status.c?r1=1354021r2=1354020pathrev=1354021 to your mod_jk source before compiling. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk worker activation not working anymore in 1.2.37?
On 11.06.2013 00:58, Martin Knoblauch wrote: Any plans when 1.2.38 will be released? Not really, but it is overdue. So IMO we should release it during the next few weeks. Regards, Rainer On Mon, Jun 10, 2013 at 10:20 PM, Rainer Jung rainer.j...@kippdata.dewrote: On 10.06.2013 17:29, Konstantin Kolinko wrote: 2013/6/10 David Gubler d...@doodle.com: Hi list, We have recently upgraded our Apache servers from Debian Squeeze to Wheezy (from Apache 2.2.16 with mod_jk 1.2.30 to Apache 2.2.22 with mod_jk 1.2.37). The Tomcat version hasn't changed (7.0.37). We often do rolling releases by disabling (DIS) some worker in jkmanager, waiting for a few minutes for most sessions to go away (we use sticky sessions but not forced), upgrading it, and re-enabling it. This worked perfectly with mod_jk 1.2.30. The server is rather busy (order of 100-200 req/s going to tomcat). However, with mod_jk 1.2.37, the activation state behaves erratically. Say I disable worker1 on all apache servers via jkmanager. When I go back to the jkmanager overview screen, it still shows as active. I hit reload, now it shows as disabled. I can wait for a few seconds or minutes, reload, and suddenly it shows up as active again! It keeps switching back and forth between active and disabled if I reload often enough. Afterwards I usually have to set it to active a few times to make it stick there. This happens on all apache servers independently. And more worringly, the load on the worker does not decrease, not even after waiting for half an hour or longer (with 1.2.30, the load on a worker decreased to about 5% after 5-10 minutes). When I set a worker to stopped, the activation state also switches between active and stopped, the load on the worker goes down slowly, but the requests do not cease completely. With 1.2.30, I could set a worker to stopped and it instantaneously received no more requests. Other than that, mod_jk behaves as expected (e.g. if I shut down one of the tomcats, the requests go to the other; load balancing works fine in normal operation). I have stripped down our workers.properties to the bare minimum that we need, and the problem is still there: ps=/ worker.list=loadbalancer,jkstatus worker.jkstatus.type=status worker.loadbalancer.type=lb worker.loadbalancer.sticky_session=true worker.loadbalancer.balance_workers=worker1,worker2 worker.worker1.type=ajp13 worker.worker1.host=WW.XX.YY.ZZ worker.worker1.port=8009 worker.worker1.connect_timeout=7 worker.worker1.prepost_timeout=7 worker.worker1.socket_timeout=70 worker.worker1.connection_pool_timeout=70 worker.worker1.connection_pool_size=200 worker.worker1.retry_interval=1000 worker.worker1.lbfactor=1 [same for worker2, only difference is the IP address] Rest of the configuration is Debian standard. Apache uses JkAutoAlias, JkMount and a bunch of JkUnMounts, but nothing fancy. The changelog does not really give me any clues as to what change could cause this, and neither does the workers.properties documentation :( Does anyone have an idea what I could be doing wrong? Looking at the current changelog, section name=Changes between 1.2.37 and 1.2.38 ... fix Fix status worker not updating parameters for all members. (mturk) /fix That is http://svn.apache.org/viewvc?view=revisionrevision=1354021 Yes that should be it. If the OP compiles himself, just add the tiny patch http://svn.apache.org/viewvc/tomcat/jk/trunk/native/common/jk_status.c?r1=1354021r2=1354020pathrev=1354021 to your mod_jk source before compiling. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: How to get SSL connection information from Apache HTTPD over AJP?
On 07.06.2013 02:50, Omari Stephens wrote: *phew* Got it figured out, and now everything is working (including the JkEnvVar thing that I had set up) For anyone who runs across this thread in the future, the problem was that I was using a Directory / stanza to require authentication. Because Jakarta queries don't actually hit the filesystem, though, they don't match that stanza. I ended up using the advice here: http://web.archiveorange.com/archive/v/JBjmW7BaH8HOOefUz8eK When I created a Location /nftest stanza that required authentication, the Jakarta queries started requiring authentication, and then all of the authentication stuff in Tomcat started working (with tomcatAuthentication=false, as described in my previous email). Now all is well. Thanks again, Rainer, Thanks for the feedback, helpful to others. Good to know it is working now. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: How to get SSL connection information from Apache HTTPD over AJP?
On 06.06.2013 07:23, Omari Stephens wrote: Howdy, y'all I'm working on porting a pure java CGI to a servlet. I'm using Tomcat 6 behind Apache HTTPD 2.2. At this point, I have everything talking to each other fine. When I hit the right URL on httpd, my servlet gets run. yay. My question: incoming connections to httpd are over SSL. For the CGI, apache sets user-identifying information in the environment, so that I can read a particular environment variable and uniquely identify the user making the request. So far, I can't figure out how to uniquely identify the user from the Tomcat side. All of the obvious methods (like #getRemoteUser()) from HttpServletRequest return null. I see JkEnvVar at http://tomcat.apache.org/connectors-doc/webserver_howto/apache.html, but either that only copies variables from Apache's environment (rather than ones that it sets for CGI), or I'm not using it correctly. Lastly, I'm not hitting Tomcat SSL directly because I depend on a module that only exists for Apache HTTPD. Set tomcatAuthentication=false in your ajp connector. See tomcatAuthentication on page http://tomcat.apache.org/tomcat-7.0-doc/config/ajp.html. Setting it to false means that Tomcat will not authenticate the user but instead fully trust the remoteUser send by Apache. default is true. Note that this is not really related to the subject of your mail (SSL connection information). Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Encrypting AJP13 Traffic With isapi_redirect
On 31.05.2013 01:38, Cochran, Jonathan - IS.CONTRACTOR wrote: Does the IIS isapi_redirect.dll support encrypting AJP13 traffic? We are setting up IIS 7.5 talking to GlassFish 3.1.2.2 using the 1.2.37 isapi_redirect.dll. We have everything working with HTTPS/SSL coming into IIS and passing through to GlassFish using unencrypted AJP13, but want to also encrypt the traffic between IIS and GlassFish. There is GlassFish documentation for enabling SSL between Apache and GlassFish using mod_jk, and it involves setting some mod_jk settings (in addition to some settings in GlassFish to enable SSL on that end). I’ve made the changes to GlassFish to enable SSL on the passthrough port, but can’t find any settings for isapi_redirect that would indicate using SSL. The GlassFish documentation for using SSL with mod_jk involved some settings like “JkExtractSSL On” and “JkHTTPSIndicator HTTPS”, but there is nothing like that available for the isapi_redirect configuration. I can access the site fine using the built-in GlassFish HTTPS! /SSL por t 8181, but I’m getting a 502 error when trying to do the IIS passthrough to the SSL-enabled AJP13 port in GlassFish. Following is what I’m seeing in the isapi_redirect log file: mod_jk and the isapi redirector both do not support encrypting the connection between web server and servlet engine. You could set up an encrypted tunnel. The SSL options for mod_jk are just to control what kind if information about the HTTPS connection betwen client and web server are forwarded from there to the servlet engine (like original ssl session id, crtificate details etc.). [Thu May 30 17:51:44.219 2013] [224:1172] [debug] jk_shutdown_socket::jk_connect.c (732): About to shutdown socket 1300 [127.0.0.1:61402 - 127.0.0.1:8009] [Thu May 30 17:51:44.219 2013] [224:1172] [debug] jk_shutdown_socket::jk_connect.c (803): shutting down the read side of socket 1300 [127.0.0.1:61402 - 127.0.0.1:8009] [Thu May 30 17:51:44.219 2013] [224:1172] [debug] jk_shutdown_socket::jk_connect.c (814): Shutdown socket 1300 [127.0.0.1:61402 - 127.0.0.1:8009] and read 0 lingering bytes in 0 sec. [Thu May 30 17:51:44.219 2013] [224:1172] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1259): (worker1) can't receive the response header message from tomcat, tomcat (127.0.0.1:8009) has forced a connection close for socket 1300 [Thu May 30 17:51:44.219 2013] [224:1172] [error] ajp_get_reply::jk_ajp_common.c (2126): (worker1) Tomcat is down or refused connection. No response has been sent to the client (yet) Is encrypting the AJP13 traffic possible with isapi_redirect.dll and I just don’t have something configured properly, or am I trying to do something that isn’t supported natively? I saw some old posts about needing to use other methods to encrypt the traffic, like VPNs or IPSEC, but they also indicated that something was in the works to support this natively. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
[OT] ab and load testing (was: Performance Issue while upgrading from Embedded Tomcat 6 to Tomcat 7)
On 24.05.2013 17:54, Christopher Schultz wrote: Top reported that Tomcat was taking somewhere between 550-600% CPU. (This is a 4-core hyperthreaded CPU so I have 8 logical cores. 'ab' was taking about 100% CPU so I think 600% CPU means it was roughly pegging 6 of my logical cores. Roughly 30% system, 60% user CPU usage.). The 100% CPU for ab indicates that ab was your bottleneck. AFAIK ab is single threaded async, so it can handle multiple parallel connections but will max out on one consumed CPU core. If you want to go further, you would have to start multiple ab in parallel. Since you are close to a CPU saturated system, in this specific case you might not get much further, but 17454 requests/sec isn't that bad either :) Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: CORS on Tomcat?
On 21.05.2013 18:20, Leo Donahue - RDSA IT wrote: -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Subject: Re: CORS on Tomcat? -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Leo, On 5/21/13 11:34 AM, Leo Donahue - RDSA IT wrote: Does Tomcat support setting this header on the server? Header set Access-Control-Allow-Origin * If yes, where do we set it? You should know how to do this by now: url-rewrite. Thanks Chris. But.. but.. Apache has it... I wanted to avoid using a proxy that turns lengthy GET requests into POST requests for one of our REST based web apps. I was reading online where Cross Origin Resource Sharing was possible on some servers. Specifically here: http://enable-cors.org/server.html I guess Chris meant the Tuckey urlrewrite filter (http://tuckey.org/urlrewrite/) which can be used in Tomcat. Look for response-header in http://urlrewritefilter.googlecode.com/svn/trunk/src/doc/manual/4.0/guide.html. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Monitoring Tomcat - Delta Values
On 03.05.2013 18:13, Caldarale, Charles R wrote: From: Christopher Schultz [mailto:ch...@christopherschultz.net] Subject: Monitoring Tomcat - Delta Values I was wondering if anyone could recommend an existing tool to capture that data, compute the deltas, etc. or if folks just roll their own? I believe moskito does this already. http://moskito.anotheria.net/ and there's Jolokia in combination with jmx4perl: http://www.jolokia.org/ http://search.cpan.org/~roland/jmx4perl/ http://search.cpan.org/~roland/jmx4perl/scripts/jmx4perl which fits nicely together with nagios. Although AFAIR last time it couldn't do my favorite quotient of deltas, but until now I haven't found a standard tool doing that. Average response time in last intervall = delta(cumulatedResponseTime)/ delta(RequestCount) Tomcat provides cumulatedResponseTime and RequestCount per Servlet, Webapp and globally, but unfortunately the tools lack for this one. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Monitoring Tomcat - Delta Values
On 03.05.2013 19:03, Jess Holle wrote: On 5/3/2013 11:58 AM, Rainer Jung wrote: Although AFAIR last time it couldn't do my favorite quotient of deltas, but until now I haven't found a standard tool doing that. Average response time in last intervall = delta(cumulatedResponseTime)/ delta(RequestCount) Tomcat provides cumulatedResponseTime and RequestCount per Servlet, Webapp and globally, but unfortunately the tools lack for this one. You can always add an MBean for such things -- and for cumulative CPU time per request, cumulative time spent blocked and waiting per request, etc, etc. We do have the above cumulative stuff for Tomcat as MBeans. But if you want the above statistics on a client controlled interval, the client needs to be able to do quotient of deltas. Haven't seen any standard client that can do that yet (they can do deltas and rates and some quotients, but not the needed combination). So that's something one might need to implement oneself. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Configuring IIS to use the JK ISAPI redirector plugin when URL paths are different
On 24.04.2013 09:02, Rainer Jung wrote: On 24.04.2013 06:53, Beavers, Melinda K (Kay) wrote: We have installed the IIS-Tomcat redirector (isapi_redirect.dll) on an IIS 6 server so that http://iis.company.com/website/myfile.jsp will correctly redirect according to our 'isapi_redirect.properties', 'workers.properties', and 'uriworkermap.properties ' and serve the JSP page from http://tomcat.company.com/website/myfile.jsp . That appears to be working just fine. But we actually need to have a different IIS URL. What we are trying to figure out is if we can configure it so that http://iis.company.com/apps/cepv/website/myfile.jsp will redirect and serve the JSP content at http://tomcat.company.com/website/myfile.jsp. The path on the IIS server is has two extra levels (/apps/cepv) in the URL path and does not match the path on the tomcat server where the JSP content is. We have to have those two extra levels in the IIS URL path for other technical reasons and we cannot match or include those two extra levels on the tomcat side. We have tried the following but cannot get it to work. website.worker=website_ajp13 /apps/cepv/website/*.jsp=$(website.worker) Is there anything we can do to map this correctly? Have a look at https://tomcat.apache.org/connectors-doc/generic_howto/proxy.html#URL%20Rewriting starting from If you are using Microsoft IIS as a web server The OP reported via PM that it now works after upgrading from an outdated version to a recent one. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Configuring IIS to use the JK ISAPI redirector plugin when URL paths are different
On 24.04.2013 06:53, Beavers, Melinda K (Kay) wrote: We have installed the IIS-Tomcat redirector (isapi_redirect.dll) on an IIS 6 server so that http://iis.company.com/website/myfile.jsp will correctly redirect according to our 'isapi_redirect.properties', 'workers.properties', and 'uriworkermap.properties ' and serve the JSP page from http://tomcat.company.com/website/myfile.jsp . That appears to be working just fine. But we actually need to have a different IIS URL. What we are trying to figure out is if we can configure it so that http://iis.company.com/apps/cepv/website/myfile.jsp will redirect and serve the JSP content at http://tomcat.company.com/website/myfile.jsp. The path on the IIS server is has two extra levels (/apps/cepv) in the URL path and does not match the path on the tomcat server where the JSP content is. We have to have those two extra levels in the IIS URL path for other technical reasons and we cannot match or include those two extra levels on the tomcat side. We have tried the following but cannot get it to work. website.worker=website_ajp13 /apps/cepv/website/*.jsp=$(website.worker) Is there anything we can do to map this correctly? Have a look at https://tomcat.apache.org/connectors-doc/generic_howto/proxy.html#URL%20Rewriting starting from If you are using Microsoft IIS as a web server Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
