I forgot to mention that I also did a paper for Usenix that I presented in
December at the LISA conference, the video, paper, and deck are available for
free at:
https://www.usenix.org/conference/lisa12/building-100k-logsec-logging-infrastructure
David Lang
On Sat, 31 Aug 2013, David Lang wrote:
In this month's issue of ;login magazine, I have an article published on
exactly this subject. It's available at
https://www.usenix.org/publications/login/august-2013-volume-38-number-4/enterprise-logging
In a couple of months I'll be willing to send out a copy of the article
directly, but since this is the month of publication, it seems only fair to
direct you to the publishers site.
As it happens, I got caught between the gears on a company divestiture, and
received a layoff notice, so I'm available for consulting in the short term
;-)
David Lang
On Sat, 31 Aug 2013, Radu Gheorghe wrote:
Date: Sat, 31 Aug 2013 21:11:42 +0300
From: Radu Gheorghe <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Large Scale Logging Design
Hi Lee,
I think the answer for your first question is "yes", because rsyslog is
especially good for processing lots of logs.
I think a design would depend a lot on what you need to do with those logs.
Archive, daily reports, alerts, search...? Where do your logs come from
(can all apps log to syslog, or you need to slurp files as well)? Do you
need to parse them, and if yes, how?
Like Rainer said, a consulting job would probably give you all the answers
you need. But if you want some good pointers and do the research yourself,
I think some more details on the requirements are needed.
Anyway, if you're looking for large-scale log transportation, and you need
search and statistics, you might want to check out the
rsyslog+elasticsearch combination. Here are two blog posts I've done in
this area, if that sounds interesting to you:
http://blog.sematext.com/2013/05/28/structured-logging-with-rsyslog-and-elasticsearch/
http://blog.sematext.com/2013/07/01/recipe-rsyslog-elasticsearch-kibana/
Best regards,
Radu
2013/8/31 Lee Eric <[email protected]>
Hi all,
We manage 4 regions DCs with 3000+ servers and all systems are running
Linux so we would like to design a or a bunch of rsyslog server(s). So
obviously we need a large scale logging system.
Can we use rsyslog as this role? If yes, is there any draft plan or
design for this?
Thanks.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
