Thank you for you response. Could you give any example of how you might add
tags in the incoming JSON from the client-side nxlog?
Cheers,
Cameron
--
Cameron Kerr <cameron.kerr...@gmail.com>
See my blog at http://distracted-it.blogspot.co.nz/ (previously
http://humbledown.org/)
Skype me on cameron.kerr.nz
On 17 July 2014 20:57, Botond Botyanszki <b...@nxlog.org> wrote:
> Hi Cameron,
>
> Doing this sort of source routing can be done using separate ports as you
> have described. This has the benefit of zero processing overhead.
>
> Otherwise you might want to consider using a single port and do the
> filtering in nxlog based on the fields present in the log. This would
> allow some simplification and possibly a lot more flexibility. If you use
> JSON or the nxlog binary format to send from the clients you can also tag
> the data on the client side.
>
> Regards,
> Botond
>
> On Mon, 14 Jul 2014 15:41:51 +1200
> Cameron Kerr <cameron.kerr...@gmail.com> wrote:
>
> > Hi all,
> >
> > I'm embarking on a project to centralise system and application logs from
> > our Linux and Windows (and other) machines. We've shortlisted nxlog as
> > being a strong contender for doing the actual centralisation, and we'll
> use
> > something else (perhaps logstash) to further analyse and process some
> > subset of the logs.
> >
> > I want to make it easy to use, particularly around data retention. There
> > will be a lot of data coming into this, and I want to make a tiered
> > retention service.
> >
> > I'm thinking that I could have a selection of different ports that you
> > could send to; each being processed in a different nxlog route, which
> would
> > go through a different log rotation block. The client system would be
> > responsible for sending their log data to the appropriate retention tier.
> >
> > Besides tiered retention, I also want to have tiered storage (local disk
> to
> > write incoming logs to, and then rotate to a remote file-system)
> >
> > I'd also like to have machines in different parts of the network log to
> > different (closest ideally, with a fail-over) nxlog servers, but that
> > doesn't seem to concern me too much with regard to this configuration,
> > other than how files are rotated.
> >
> > For argument's sake, the retention tiers would be 1 week, 3 months, and 7
> > years.
> >
> > To avoid vendor lock-in, I'd prefer to normalise logs to JSON, or keep
> them
> > as raw (which has some advantages, I suppose)... but I'd like to hear
> what
> > people's thoughts are on that.
> >
> > I'd also like to be able for a client system to give some sort of
> > indication as to what type of component it came from... eg. webserver,
> > applicationserver, syslog, .... which ideally could be used in later
> > processing. Not sure what really works well for that, so would love some
> > feedback on that.
> >
> > Here is my sketched configuration thus far... I'd appreciate some
> guidance
> > as to whether this seems like a good idea, and how it could be improved
> > etc. I'm laying them down on disk without reference to $Hostname, as I
> > don't want to create a DNS storm, as I have recently experienced with a
> > similar configuration in rsyslogd.
> >
> > # I think Macros can be used to reduce duplication of these definitions
> >
> > <Input incoming_syslog_with_1wk_lifetime>
> > Module im_tcp
> > Port 5141
> > Host 0.0.0.0
> > Exec parse_syslog_bsd();
> > </Input>
> >
> > <Input incoming_syslog_with_3mnth_lifetime>
> > Module im_tcp
> > Port 5143
> > Host 0.0.0.0
> > Exec parse_syslog_bsd();
> > </Input>
> >
> > <Input incoming_syslog_with_7yr_lifetime>
> > Module im_tcp
> > Port 5147
> > Host 0.0.0.0
> > Exec parse_syslog_bsd();
> > </Input>
> >
> > # Conceivably another set of ports could be used to request
> > deliverance to something like Logstash.
> > # Eg. 5147 is 7yr streamed to disk, while 5157 is streamed to disk AND
> > loaded into Logstash...
> > # I'm ambivalent as to whether this is a good idea... so its perhaps
> > best avoided for now.
> >
> > <Output output_to_disk_with_1wk_lifetime>
> > Module om_file
> > File "/logs/current/" + $SourceIPAddress + "_1wk.log"
> > Exec if out->file_size() > 15M \
> > { \
> > $newfile = "/logs/archive/" + $Hostname + "/" +
> > strftime(now(), "%Y%m%d%H%M%S") + "_recv_by_THISNXLOGSERVER_1wk.log";
> > \
> > out->rotate_to($newfile); \
> > exec_async("/bin/bzip2", $newfile); \
> > }
> > </Output>
> >
> > <Output output_to_disk_with_3mnth_lifetime>
> > Module om_file
> > File "/logs/current/" + $SourceIPAddress + "_3mnth.log"
> > Exec if out->file_size() > 15M \
> > { \
> > $newfile = "/logs/archive/" + $Hostname + "/" +
> > strftime(now(), "%Y%m%d%H%M%S") +
> > "_recv_by_THISNXLOGSERVER_3mnth.log"; \
> > out->rotate_to($newfile); \
> > exec_async("/bin/bzip2", $newfile); \
> > }
> > </Output>
> >
> > <Output output_to_disk_with_7yr_lifetime>
> > Module om_file
> > File "/logs/current/" + $SourceIPAddress + "_7yr"
> > Exec if out->file_size() > 15M \
> > { \
> > $newfile = "/logs/archive/" + $Hostname + "/" +
> > strftime(now(), "%Y%m%d%H%M%S") + "_recv_by_THISNXLOGSERVER_7yr.log";
> > \
> > out->rotate_to($newfile); \
> > exec_async("/bin/bzip2", $newfile); \
> > }
> > </Output>
> >
> > <Route 1wk>
> > Path incoming_syslog_with_1wk_lifetime =>
> output_to_disk_with_1wk_lifetime
> > </Route>
> >
> > <Route 3mnth>
> > Path incoming_syslog_with_3mnth_lifetime =>
> > output_to_disk_with_3mnth_lifetime
> > </Route>
> >
> > <Route 7yr>
> > Path incoming_syslog_with_7yr_lifetime =>
> output_to_disk_with_7yr_lifetime
> > </Route>
> >
> >
> >
> > Thanks very much,
> > Cameron
> >
> > --
> > Cameron Kerr <cameron.kerr...@gmail.com>
> > See my blog at http://distracted-it.blogspot.co.nz/ (previously
> > http://humbledown.org/)
> > Skype me on cameron.kerr.nz
>
>
> ------------------------------------------------------------------------------
> Want fast and easy access to all the code in your enterprise? Index and
> search up to 200,000 lines of code with a free copy of Black Duck
> Code Sight - the same software that powers the world's largest code
> search on Ohloh, the Black Duck Open Hub! Try it now.
> http://p.sf.net/sfu/bds
> _______________________________________________
> nxlog-ce-users mailing list
> nxlog-ce-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
>
------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
nxlog-ce-users mailing list
nxlog-ce-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
