Hi all,
I'm embarking on a project to centralise system and application logs from
our Linux and Windows (and other) machines. We've shortlisted nxlog as
being a strong contender for doing the actual centralisation, and we'll use
something else (perhaps logstash) to further analyse and process some
subset of the logs.
I want to make it easy to use, particularly around data retention. There
will be a lot of data coming into this, and I want to make a tiered
retention service.
I'm thinking that I could have a selection of different ports that you
could send to; each being processed in a different nxlog route, which would
go through a different log rotation block. The client system would be
responsible for sending their log data to the appropriate retention tier.
Besides tiered retention, I also want to have tiered storage (local disk to
write incoming logs to, and then rotate to a remote file-system)
I'd also like to have machines in different parts of the network log to
different (closest ideally, with a fail-over) nxlog servers, but that
doesn't seem to concern me too much with regard to this configuration,
other than how files are rotated.
For argument's sake, the retention tiers would be 1 week, 3 months, and 7
years.
To avoid vendor lock-in, I'd prefer to normalise logs to JSON, or keep them
as raw (which has some advantages, I suppose)... but I'd like to hear what
people's thoughts are on that.
I'd also like to be able for a client system to give some sort of
indication as to what type of component it came from... eg. webserver,
applicationserver, syslog, .... which ideally could be used in later
processing. Not sure what really works well for that, so would love some
feedback on that.
Here is my sketched configuration thus far... I'd appreciate some guidance
as to whether this seems like a good idea, and how it could be improved
etc. I'm laying them down on disk without reference to $Hostname, as I
don't want to create a DNS storm, as I have recently experienced with a
similar configuration in rsyslogd.
# I think Macros can be used to reduce duplication of these definitions
<Input incoming_syslog_with_1wk_lifetime>
Module im_tcp
Port 5141
Host 0.0.0.0
Exec parse_syslog_bsd();
</Input>
<Input incoming_syslog_with_3mnth_lifetime>
Module im_tcp
Port 5143
Host 0.0.0.0
Exec parse_syslog_bsd();
</Input>
<Input incoming_syslog_with_7yr_lifetime>
Module im_tcp
Port 5147
Host 0.0.0.0
Exec parse_syslog_bsd();
</Input>
# Conceivably another set of ports could be used to request
deliverance to something like Logstash.
# Eg. 5147 is 7yr streamed to disk, while 5157 is streamed to disk AND
loaded into Logstash...
# I'm ambivalent as to whether this is a good idea... so its perhaps
best avoided for now.
<Output output_to_disk_with_1wk_lifetime>
Module om_file
File "/logs/current/" + $SourceIPAddress + "_1wk.log"
Exec if out->file_size() > 15M \
{ \
$newfile = "/logs/archive/" + $Hostname + "/" +
strftime(now(), "%Y%m%d%H%M%S") + "_recv_by_THISNXLOGSERVER_1wk.log";
\
out->rotate_to($newfile); \
exec_async("/bin/bzip2", $newfile); \
}
</Output>
<Output output_to_disk_with_3mnth_lifetime>
Module om_file
File "/logs/current/" + $SourceIPAddress + "_3mnth.log"
Exec if out->file_size() > 15M \
{ \
$newfile = "/logs/archive/" + $Hostname + "/" +
strftime(now(), "%Y%m%d%H%M%S") +
"_recv_by_THISNXLOGSERVER_3mnth.log"; \
out->rotate_to($newfile); \
exec_async("/bin/bzip2", $newfile); \
}
</Output>
<Output output_to_disk_with_7yr_lifetime>
Module om_file
File "/logs/current/" + $SourceIPAddress + "_7yr"
Exec if out->file_size() > 15M \
{ \
$newfile = "/logs/archive/" + $Hostname + "/" +
strftime(now(), "%Y%m%d%H%M%S") + "_recv_by_THISNXLOGSERVER_7yr.log";
\
out->rotate_to($newfile); \
exec_async("/bin/bzip2", $newfile); \
}
</Output>
<Route 1wk>
Path incoming_syslog_with_1wk_lifetime =>
output_to_disk_with_1wk_lifetime
</Route>
<Route 3mnth>
Path incoming_syslog_with_3mnth_lifetime =>
output_to_disk_with_3mnth_lifetime
</Route>
<Route 7yr>
Path incoming_syslog_with_7yr_lifetime =>
output_to_disk_with_7yr_lifetime
</Route>
Thanks very much,
Cameron
--
Cameron Kerr <cameron.kerr...@gmail.com>
See my blog at http://distracted-it.blogspot.co.nz/ (previously
http://humbledown.org/)
Skype me on cameron.kerr.nz
------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck®
Code Sight™ - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
nxlog-ce-users mailing list
nxlog-ce-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
