A few weeks ago I asked about LEAF-based solutions for monitoring Internet (HTTP) usage. I had received a request from a client to do just that, and I was looking for suggestions. I wanted to outline what I ended up doing. Maybe it will be useful for someone else.
My first thought was something like Squid. However, this presented several challenges. The client wanted zero dependence on the client: it had to be 100% transparent. A transparent Squid server is not perfect, and it requires a decent amount of reconfiguration of the firewall. Also, it would have needed another computer to run on: you don't stick something like that on your firewall!
The client has another Linux box, and it's already working as a circuit proxy with Delegate. But it's in the DMZ: it proxies all incoming TCP requests for the few services hosted on the Internal LAN. The internal firewall is doing NAT, and using that proxy server would involve reconfiguring the internal firewall not to do NAT (and all the other changes necessary now that the internal IP's are no longer hidden from the DMZ...). It's also another service to run on what needs to be a secure box. So, if possible, I wanted to keep the logging within the internal network. But there was no available Linux box there, and if possible I wanted to avoid adding another server-type box. So Squid was eliminated because of the fairly extensive firewall reconfiguration and lack of suitable box to put it on.
Fortunately, Eric Wolzak had an excellent alternative: urlsnarf. It's part of a package called dsniff that is designed to analyze a variety of Internet traffic and report all kinds of useful (and valuable) data. Urlsnarf in particular scans a stream of IP traffic and logs all HTTP requests in an NCSA-compatible log format. There's even a win32 version (http://www.datanerds.net/~mike/dsniff.html).
This solved most of my problems: I could throw the software on a lightly loaded Windows server (one of their applications requires a "dedicated indexer" that does very, very little) and voila! Instant logging!
There was a couple of smaller problem with this, though. First, there is a bug in the win32 version (at least that I can tell) that does not put the date in the log file! So, the logs must be parsed daily to insert the date into each record. Also, of course, urlsnarf only captures IP information. The client wants to see names. I'm using Bering's DHCPD for DHCP, and, sadly, it doesn't communicate with *anything* as far as I can tell, so I had to reconfigure the server to give out static IP addresses based on MAC address. Now that the IP's are static (though still assigned by DHCP), I can simply replace the IP address with the hostname (or most any other string, really). I then use Analog to run a variety of reports: daily data, weekly data, and all data for each user, as well as the office as a whole.
To do all of this, I wrote a REXX script that is called from a cron daemon at midnight. It kills urlsnarf, copies the log, restarts urlsnarf, and then parses the copy. It adds the date, turns known IP's into readable names and runs the necessary Analog reports. It also generates a simple HTML interface to all of the reports for each day. All-in-all, it works very well.
If anyone would like a copy of the script, just drop me an e-mail: I'm making it available under the GPL. All of the other items that it uses (the cron daemon, process killer, Analog, etc.) are also freeware or GPL as well. The script is designed for both Windows 9x and 2000/XP, and could trivally be adapted to Linux; of course, it could also be trivally replaced by a Perl script, though why you'd use line noise--err, Perl over a beautiful REXX script is beyond me... :)
Thanks to everyone who helped me with this solution, particularly Eric Wolzak for mentioning urlsnarf, and Victor McAllister for mentioning how to get DHCPD to give out static IP's in a nearby thread. If you have any thoughts or suggestions on improvements--or even completely different solutions--I'd love to hear them.
Tim Massey
------------------------------------------------------- The SF.Net email is sponsored by EclipseCon 2004 Premiere Conference on Open Tools Development and Integration See the breadth of Eclipse activity. February 3-5 in Anaheim, CA. http://www.eclipsecon.org/osdn ------------------------------------------------------------------------ leaf-user mailing list: [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/leaf-user SR FAQ: http://leaf-project.org/pub/doc/docmanager/docid_1891.html
