Re: Filtering HTTP OPTIONS request method from logs?
On 17/09/2013 5:05 PM, André Warnier wrote: Jim Barber wrote: All, On 16/09/2013 10:52 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Felix, On 9/16/13 10:25 AM, Felix Schumacher wrote: Am Montag, den 16.09.2013, 10:02 -0400 schrieb Christopher Schultz: Jim, On 9/16/13 3:42 AM, Jim Barber wrote: I'm hoping someone on this list can help me since I've been reading docs, mailing lists, FAQs, and so on for hours now, and I'm not having much luck finding an answer to my question. I am using Tomcat version 7.0.42 as packaged in Debian Linux. In front of my Tomcat servers, I am using haproxy for load balancing. The haproxy load balancers are using the HTTP OPTIONS request method to check if the Tomcat servers are alive and healthy. This results in log entries like the following in the Tomcat accesslog file: 10.122.32.4 - - [16/Sep/2013:17:12:49 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:12:51 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:12:53 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:12:55 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:12:57 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:12:59 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:13:01 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:13:03 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:13:05 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:13:07 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:13:09 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:13:11 +1000] "OPTIONS / HTTP/1.0" 200 - At the moment I'm getting one of these every 2seconds, but I haven't enabled the second load balancer for HA purposes yet. When I do that, I'll be getting twice as many hits of this type. This is going to result in rather large log files full of noise that I'm not interested in. Playing the devil's advocate here a bit... Why wouldn't you be interested in getting these logs? They are requests being handled by your web server. They require (a small amount of) time and resources to process, and indicate that your lb is still reaching-out to determine the status of the app server. My recommendation would be to leave those logs in there (they accurately describe a real request) and filter them out if you want to do some kind of analytics against your log files and consider those OPTIONS requests to be noise. I have had one case where I wanted to get rid of those requests too, so I can understand the OP. But I have to admint I had a scary feeling about it. Specifically adding the condition="" attribute, but I have no idea what to set to. It's not that simple: if you want to use "condition", then you have a write a Valve (can't be a Filter, since it must run *before* the AccessLogValve) that tests the request and sets a request attribute that will then trigger this condition. That is not true, you can use a filter, since the logging will happen *after* the request and can and will check the request attribute then. Thanks for pointing that out. After I had sent my message, I realized that and decided not to post a followup after reading yours. - -chris Thank you all for your responses. It looks like I'll just have to put up with these messages. There seems to be no easy way to filter them out. Actually, based on earlier responses and on the URLrewrite filter (http://http://tuckey.org/urlrewrite/), there may be a solution after all, which does not involve additional Java programming, as long as you are willing to do some research by yourself. (Short intro : the URLrewrite filter is a bit of a workhorse, simioar to mod_rewrite for Apache httpd, and which can do a multitude of things when it comes to filter/modify HTTP requests in Tomcat) First, get the URLrewrite User's Manual at http://urlrewritefilter.googlecode.com/svn/trunk/src/doc/manual/4.0/index.html and then search for : element (see "method" and "remote-addr") and element (see "request (default)") : The same as request.setAttribute([name], [value]) (note, name must be set). So, the requests that you want to "not log" look like this : >>>>> 10.122.32.4 - - [16/Sep/2013:17:12:49 +1000] "OPTIONS / >>>>> HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:12:51 +1000] and thus they address the "/" (default) webapp, originate from the client IP 10.122.32.4, and have the method "OPTIONS". And on the other hand, the AccessLogValve has an attribute which allows you to specify that if the request has an attribute (e.g.) &qu
Re: Filtering HTTP OPTIONS request method from logs?
All, On 16/09/2013 10:52 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Felix, On 9/16/13 10:25 AM, Felix Schumacher wrote: Am Montag, den 16.09.2013, 10:02 -0400 schrieb Christopher Schultz: Jim, On 9/16/13 3:42 AM, Jim Barber wrote: I'm hoping someone on this list can help me since I've been reading docs, mailing lists, FAQs, and so on for hours now, and I'm not having much luck finding an answer to my question. I am using Tomcat version 7.0.42 as packaged in Debian Linux. In front of my Tomcat servers, I am using haproxy for load balancing. The haproxy load balancers are using the HTTP OPTIONS request method to check if the Tomcat servers are alive and healthy. This results in log entries like the following in the Tomcat accesslog file: 10.122.32.4 - - [16/Sep/2013:17:12:49 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:12:51 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:12:53 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:12:55 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:12:57 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:12:59 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:13:01 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:13:03 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:13:05 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:13:07 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:13:09 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:13:11 +1000] "OPTIONS / HTTP/1.0" 200 - At the moment I'm getting one of these every 2seconds, but I haven't enabled the second load balancer for HA purposes yet. When I do that, I'll be getting twice as many hits of this type. This is going to result in rather large log files full of noise that I'm not interested in. Playing the devil's advocate here a bit... Why wouldn't you be interested in getting these logs? They are requests being handled by your web server. They require (a small amount of) time and resources to process, and indicate that your lb is still reaching-out to determine the status of the app server. My recommendation would be to leave those logs in there (they accurately describe a real request) and filter them out if you want to do some kind of analytics against your log files and consider those OPTIONS requests to be noise. I have had one case where I wanted to get rid of those requests too, so I can understand the OP. But I have to admint I had a scary feeling about it. Specifically adding the condition="" attribute, but I have no idea what to set to. It's not that simple: if you want to use "condition", then you have a write a Valve (can't be a Filter, since it must run *before* the AccessLogValve) that tests the request and sets a request attribute that will then trigger this condition. That is not true, you can use a filter, since the logging will happen *after* the request and can and will check the request attribute then. Thanks for pointing that out. After I had sent my message, I realized that and decided not to post a followup after reading yours. - -chris Thank you all for your responses. It looks like I'll just have to put up with these messages. There seems to be no easy way to filter them out. Regards, - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Filtering HTTP OPTIONS request method from logs?
On 16/09/2013 4:46 PM, André Warnier wrote:
Apologies for top posting, just following the trend.
OPTIONS are used quite a bit by e.g. DAV clients.
Won't you want also to add an IP filter then, to be able to block selectively
only the requests from the proxies themselves ?
Cédric Couralet wrote:
Hi,
I'm also interested in a method to filter those OPTIONS.
With the same setup, I basically created my own AccessLogValve wich
does the filtering, something like :
/**
* Don't log request when HTTP Method is one of the exclude List
*/
@Override
public void log(Request request, Response response, long time) {
if (Arrays.asList(exclude.split(",")).contains(request.getMethod())) {
return;
}
super.log(request, response, time);
}
But there must be something better.
2013/9/16 Jim Barber :
Hi all.
I'm hoping someone on this list can help me since I've been reading docs,
mailing lists, FAQs, and so on for hours now, and I'm not having much luck
finding an answer to my question.
I am using Tomcat version 7.0.42 as packaged in Debian Linux.
In front of my Tomcat servers, I am using haproxy for load balancing.
The haproxy load balancers are using the HTTP OPTIONS request method to
check
if the Tomcat servers are alive and healthy.
This results in log entries like the following in the Tomcat accesslog file:
10.122.32.4 - - [16/Sep/2013:17:12:49 +1000] "OPTIONS / HTTP/1.0" 200 -
10.122.32.4 - - [16/Sep/2013:17:12:51 +1000] "OPTIONS / HTTP/1.0" 200 -
10.122.32.4 - - [16/Sep/2013:17:12:53 +1000] "OPTIONS / HTTP/1.0" 200 -
10.122.32.4 - - [16/Sep/2013:17:12:55 +1000] "OPTIONS / HTTP/1.0" 200 -
10.122.32.4 - - [16/Sep/2013:17:12:57 +1000] "OPTIONS / HTTP/1.0" 200 -
10.122.32.4 - - [16/Sep/2013:17:12:59 +1000] "OPTIONS / HTTP/1.0" 200 -
10.122.32.4 - - [16/Sep/2013:17:13:01 +1000] "OPTIONS / HTTP/1.0" 200 -
10.122.32.4 - - [16/Sep/2013:17:13:03 +1000] "OPTIONS / HTTP/1.0" 200 -
10.122.32.4 - - [16/Sep/2013:17:13:05 +1000] "OPTIONS / HTTP/1.0" 200 -
10.122.32.4 - - [16/Sep/2013:17:13:07 +1000] "OPTIONS / HTTP/1.0" 200 -
10.122.32.4 - - [16/Sep/2013:17:13:09 +1000] "OPTIONS / HTTP/1.0" 200 -
10.122.32.4 - - [16/Sep/2013:17:13:11 +1000] "OPTIONS / HTTP/1.0" 200 -
At the moment I'm getting one of these every 2seconds, but I haven't enabled
the second load balancer for HA purposes yet.
When I do that, I'll be getting twice as many hits of this type.
This is going to result in rather large log files full of noise that I'm not
interested in.
I've been trying to work out how to filter these out.
Basically I don't want to log anything that is using the HTTP OPTIONS
Request
Method, but still want to log anything else that Tomcat usually logs.
I have a feeling it will come down to modifying the following entry in the
/etc/tomcat7/server.xml file:
Specifically adding the condition="" attribute, but I have no idea
what to set
to.
The docs say that if ServletRequest.getAttribute() returns null for
the
attribute defined in condition, then the item will be logged.
Is there an ServletRequest attribute that is null when the http request
method
is not using "OPTIONS"?
Or am I completely off track and there is a different way to filter these
access log messages?
Regards,
--
Jim Barber
Hi André.
I'm not sure I follow what you're saying.
I don't want an IP filter, since I need the HTTP OPTIONS check from the load
balancers to reach the Tomcat servers and a response to come back, or else the
load balancers will think the tomcat instance is unhealthy.
I just don't want that check to be logged at all.
Although there are other things that use the HTTP OPTIONS check, these load
balancers are only exposed to internal traffic requesting specific servlets
from the Tomcat servers, and so there won't be anything else of interest using
the HTTP OPTIONS request methods to the Tomcat servers.
Hi Cédric.
What you posted is some Java code that needs to be compiled and then the
resulting class file put somewhere where Tomcat can find it right?
Is it only partial code where 'exclude' was some sort of pre-populated
comma separated string?
Just checking as it doesn't look like anything that you can put direct into a
Tomcat configuration file to me.
Or is it?
Regards,
Jim Barber
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Filtering HTTP OPTIONS request method from logs?
Hi all. I'm hoping someone on this list can help me since I've been reading docs, mailing lists, FAQs, and so on for hours now, and I'm not having much luck finding an answer to my question. I am using Tomcat version 7.0.42 as packaged in Debian Linux. In front of my Tomcat servers, I am using haproxy for load balancing. The haproxy load balancers are using the HTTP OPTIONS request method to check if the Tomcat servers are alive and healthy. This results in log entries like the following in the Tomcat accesslog file: 10.122.32.4 - - [16/Sep/2013:17:12:49 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:12:51 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:12:53 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:12:55 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:12:57 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:12:59 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:13:01 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:13:03 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:13:05 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:13:07 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:13:09 +1000] "OPTIONS / HTTP/1.0" 200 - 10.122.32.4 - - [16/Sep/2013:17:13:11 +1000] "OPTIONS / HTTP/1.0" 200 - At the moment I'm getting one of these every 2seconds, but I haven't enabled the second load balancer for HA purposes yet. When I do that, I'll be getting twice as many hits of this type. This is going to result in rather large log files full of noise that I'm not interested in. I've been trying to work out how to filter these out. Basically I don't want to log anything that is using the HTTP OPTIONS Request Method, but still want to log anything else that Tomcat usually logs. I have a feeling it will come down to modifying the following entry in the /etc/tomcat7/server.xml file: Specifically adding the condition="" attribute, but I have no idea what to set to. The docs say that if ServletRequest.getAttribute() returns null for the attribute defined in condition, then the item will be logged. Is there an ServletRequest attribute that is null when the http request method is not using "OPTIONS"? Or am I completely off track and there is a different way to filter these access log messages? Regards, -- Jim Barber - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
