On 12/20/06, Aengus <[EMAIL PROTECTED]> wrote:
On Tuesday, December 19, 2006 11:01 PM [EDT],
howard chen <[EMAIL PROTECTED]> wrote:
> Anyone have been thinking of using analog to detect abnormal requests?
When I worked on a system that generated dynamic price lists, we
occassionally noticed that it was being spidered (by a competitor, we
presumed). I would do a quick log analysis with Analog, with all reports off
except the Host report, sorted by Page Requests, and showing the number of
Requests and the number of Page Requests. Any spider would stick out like a
sore thumb (spiders usually don't request images, so the number of page
requests would match the number of Requests). We could then do some urther
investigation and decide whether or not to block that IP address. (A caching
proxy server might also be only requesting Pages, as it might have cached
most of the common images, so it would be important to check the request
pattern to see if it indicated ordinary use, or more methodical spidering).
> For example, a DOS at track from a remote IP, it would be not
> efficient to implement as a real time system using such as PHP.
>
> For example, If we analyze the log using analog every 30 min, and find
> out those abnormal request, it would be quite interesting. And most
> importantly, analog is fast and will not hurt your system,
We never automated this process, we just used it as an investigative
technique when other monitors indicated a problem, but you could certainly
use it as a primary monitor to look for specific troublesome patterns.
Aengus
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
Well, this is interesting...
I think Yahoo also have this kind of functions, sometimes ago, I have
a spider to check for the stock's price, and Yahoo blocked me for
spiding one hour later...
funny?
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
