(cc'ing the google group, the sf list is being deprecated)
Splunk costs money? I thought it was free :-)
Anyhoo...
What version are you running? I applied a change to version j (I think,
maybe it was k) that speeds up the display time considerably for
regularresult and tailresult.
Also, how often are you rotating logs?
If you're that serious about it, you may also want to look into the
possibility of purchasing Mysql Max (although I've not tested this with it,
so it may need some tweaks).
On 9/14/07, Alex Howells <[EMAIL PROTECTED]> wrote:
>
> Hi Guys,
>
> I'm just throwing this out there looking for some suggestions really...
> we currently use PHP-Syslog-NG at work, and our central logging host
> aggregates logs from about 110-125 physical boxes plus another sizable
> whack of networking gear (switches, routers, terminal servers).
>
> It looks like we're generating about 5500000 rows of logged data per day
> which equals about 1.25GB of actual data; some days this may burst up to
> around 1.5GB - 2GB of data depending on whether anything went bang ;)
>
> We store data for thirty days at the moment. In actuality we have 182m
> rows of data in the database at the moment :)
>
> Searches are *really* slow over this data - I'm looking for suggestions
> on how to improve the standard schema or application to speed this up?
> Typical return time from a search over all the hosts is 75 - 90 seconds.
>
> In future the following will happen:
>
> * Number of servers will increase to perhaps two hundred.
> We don't currently log anywhere near all our hardware to it.
>
> * Networking devices sending logging data will increase.
>
> * Logging data will need to be stored for longer (six months).
>
> * More data will be logged on servers.
>
> I estimate we'll be looking at up to 5GB/day once all that happens :(
>
> Our current hardware is:
>
> Web Node: virtual machine running on a low-contention Xen box.
> Dual Opteron 275, 4096MB RAM, 2 x 320GB SATA
> (the WWW node has 512MB allocated to it...)
>
> DB Node: Dual Opteron 285, 4096MB RAM, 2 x 150GB Raptors
> physical hardware, shared MySQL environment.
> The rest of the databases hosted on here aren't
> particularly chunky, low query rate.
>
> I'd rather not pay $$$ for Splunk if at all possible! Any ideas?
>
> Cheerio,
>
> Alex Howells
> Bytemark Hosting
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2005.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> Php-syslog-ng-support mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/php-syslog-ng-support
>
--
______________________________________________________________
Clayton Dukes
______________________________________________________________
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Php-syslog-ng-support mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/php-syslog-ng-support
