Hi Guys,
I'm just throwing this out there looking for some suggestions really...
we currently use PHP-Syslog-NG at work, and our central logging host
aggregates logs from about 110-125 physical boxes plus another sizable
whack of networking gear (switches, routers, terminal servers).
It looks like we're generating about 5500000 rows of logged data per day
which equals about 1.25GB of actual data; some days this may burst up to
around 1.5GB - 2GB of data depending on whether anything went bang ;)
We store data for thirty days at the moment. In actuality we have 182m
rows of data in the database at the moment :)
Searches are *really* slow over this data - I'm looking for suggestions
on how to improve the standard schema or application to speed this up?
Typical return time from a search over all the hosts is 75 - 90 seconds.
In future the following will happen:
* Number of servers will increase to perhaps two hundred.
We don't currently log anywhere near all our hardware to it.
* Networking devices sending logging data will increase.
* Logging data will need to be stored for longer (six months).
* More data will be logged on servers.
I estimate we'll be looking at up to 5GB/day once all that happens :(
Our current hardware is:
Web Node: virtual machine running on a low-contention Xen box.
Dual Opteron 275, 4096MB RAM, 2 x 320GB SATA
(the WWW node has 512MB allocated to it...)
DB Node: Dual Opteron 285, 4096MB RAM, 2 x 150GB Raptors
physical hardware, shared MySQL environment.
The rest of the databases hosted on here aren't
particularly chunky, low query rate.
I'd rather not pay $$$ for Splunk if at all possible! Any ideas?
Cheerio,
Alex Howells
Bytemark Hosting
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Php-syslog-ng-support mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/php-syslog-ng-support
