Hi Cameron,
Doing this sort of source routing can be done using separate ports as you
have described. This has the benefit of zero processing overhead.
Otherwise you might want to consider using a single port and do the
filtering in nxlog based on the fields present in the log. This would
allow some simplification and possibly a lot more flexibility. If you use
JSON or the nxlog binary format to send from the clients you can also tag
the data on the client side.
Regards,
Botond
On Mon, 14 Jul 2014 15:41:51 +1200
Cameron Kerr <cameron.kerr...@gmail.com> wrote:
> Hi all,
>
> I'm embarking on a project to centralise system and application logs from
> our Linux and Windows (and other) machines. We've shortlisted nxlog as
> being a strong contender for doing the actual centralisation, and we'll use
> something else (perhaps logstash) to further analyse and process some
> subset of the logs.
>
> I want to make it easy to use, particularly around data retention. There
> will be a lot of data coming into this, and I want to make a tiered
> retention service.
>
> I'm thinking that I could have a selection of different ports that you
> could send to; each being processed in a different nxlog route, which would
> go through a different log rotation block. The client system would be
> responsible for sending their log data to the appropriate retention tier.
>
> Besides tiered retention, I also want to have tiered storage (local disk to
> write incoming logs to, and then rotate to a remote file-system)
>
> I'd also like to have machines in different parts of the network log to
> different (closest ideally, with a fail-over) nxlog servers, but that
> doesn't seem to concern me too much with regard to this configuration,
> other than how files are rotated.
>
> For argument's sake, the retention tiers would be 1 week, 3 months, and 7
> years.
>
> To avoid vendor lock-in, I'd prefer to normalise logs to JSON, or keep them
> as raw (which has some advantages, I suppose)... but I'd like to hear what
> people's thoughts are on that.
>
> I'd also like to be able for a client system to give some sort of
> indication as to what type of component it came from... eg. webserver,
> applicationserver, syslog, .... which ideally could be used in later
> processing. Not sure what really works well for that, so would love some
> feedback on that.
>
> Here is my sketched configuration thus far... I'd appreciate some guidance
> as to whether this seems like a good idea, and how it could be improved
> etc. I'm laying them down on disk without reference to $Hostname, as I
> don't want to create a DNS storm, as I have recently experienced with a
> similar configuration in rsyslogd.
>
> # I think Macros can be used to reduce duplication of these definitions
>
> <Input incoming_syslog_with_1wk_lifetime>
> Module im_tcp
> Port 5141
> Host 0.0.0.0
> Exec parse_syslog_bsd();
> </Input>
>
> <Input incoming_syslog_with_3mnth_lifetime>
> Module im_tcp
> Port 5143
> Host 0.0.0.0
> Exec parse_syslog_bsd();
> </Input>
>
> <Input incoming_syslog_with_7yr_lifetime>
> Module im_tcp
> Port 5147
> Host 0.0.0.0
> Exec parse_syslog_bsd();
> </Input>
>
> # Conceivably another set of ports could be used to request
> deliverance to something like Logstash.
> # Eg. 5147 is 7yr streamed to disk, while 5157 is streamed to disk AND
> loaded into Logstash...
> # I'm ambivalent as to whether this is a good idea... so its perhaps
> best avoided for now.
>
> <Output output_to_disk_with_1wk_lifetime>
> Module om_file
> File "/logs/current/" + $SourceIPAddress + "_1wk.log"
> Exec if out->file_size() > 15M \
> { \
> $newfile = "/logs/archive/" + $Hostname + "/" +
> strftime(now(), "%Y%m%d%H%M%S") + "_recv_by_THISNXLOGSERVER_1wk.log";
> \
> out->rotate_to($newfile); \
> exec_async("/bin/bzip2", $newfile); \
> }
> </Output>
>
> <Output output_to_disk_with_3mnth_lifetime>
> Module om_file
> File "/logs/current/" + $SourceIPAddress + "_3mnth.log"
> Exec if out->file_size() > 15M \
> { \
> $newfile = "/logs/archive/" + $Hostname + "/" +
> strftime(now(), "%Y%m%d%H%M%S") +
> "_recv_by_THISNXLOGSERVER_3mnth.log"; \
> out->rotate_to($newfile); \
> exec_async("/bin/bzip2", $newfile); \
> }
> </Output>
>
> <Output output_to_disk_with_7yr_lifetime>
> Module om_file
> File "/logs/current/" + $SourceIPAddress + "_7yr"
> Exec if out->file_size() > 15M \
> { \
> $newfile = "/logs/archive/" + $Hostname + "/" +
> strftime(now(), "%Y%m%d%H%M%S") + "_recv_by_THISNXLOGSERVER_7yr.log";
> \
> out->rotate_to($newfile); \
> exec_async("/bin/bzip2", $newfile); \
> }
> </Output>
>
> <Route 1wk>
> Path incoming_syslog_with_1wk_lifetime =>
> output_to_disk_with_1wk_lifetime
> </Route>
>
> <Route 3mnth>
> Path incoming_syslog_with_3mnth_lifetime =>
> output_to_disk_with_3mnth_lifetime
> </Route>
>
> <Route 7yr>
> Path incoming_syslog_with_7yr_lifetime =>
> output_to_disk_with_7yr_lifetime
> </Route>
>
>
>
> Thanks very much,
> Cameron
>
> --
> Cameron Kerr <cameron.kerr...@gmail.com>
> See my blog at http://distracted-it.blogspot.co.nz/ (previously
> http://humbledown.org/)
> Skype me on cameron.kerr.nz
------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
nxlog-ce-users mailing list
nxlog-ce-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
