Hello!
I have a bit of a problem with a script and am hoping someone will have an idea as to how I can mitigate the problem. Basically I have a syslog server that receives messages from various hosts and logs them to /var/log/ files. One of these remote hosts is a firewall which is sending all of it's access information and logging it to /var/log/loghosts/firewall.log. We need a web gui for this host that allows management to see recently downloaded files, denied URLs, etc. So I wrote a perl script which uses the Tail module to constantly tail this file, parsing each line and inserting into a mysql database using the DBI module. My problem here is that the logs are coming in extremely fast and while the script does work, it progressively gets further and further behind. It's currently Sep 15, 10AM yet the script is still dealing with lines from Sep 14 at 8am. This is obviously due to the fact that the Tail module, and more likely the DBI module simply can't INSERT/UPDATE the
database as fast as the logs are coming in.
Has anyone ever dealt with an issue like this and have any idea how I might
rework the script and/or application to avoid this delay, which will obviously
get worse and worse as time goes on?
Thanks,
Brandon
