Hi, Example:
<Extension json> Module xm_json </Extension> <Output o> Module om_tcp Host 1.2.3.4 Port 1234 Exec $Source = "myapp"; to_json(); </Output> Note that the above is not syslog. If you want to wrap it into syslog, that's possible too: Exec $Source = "myapp"; $Message = to_json(); to_syslog_bsd(); HTH, Botond On Fri, 18 Jul 2014 19:48:32 +1200 Cameron Kerr <cameron.kerr...@gmail.com> wrote: > Thank you for you response. Could you give any example of how you might add > tags in the incoming JSON from the client-side nxlog? > > Cheers, > Cameron > > -- > Cameron Kerr <cameron.kerr...@gmail.com> > See my blog at http://distracted-it.blogspot.co.nz/ (previously > http://humbledown.org/) > Skype me on cameron.kerr.nz > > > > On 17 July 2014 20:57, Botond Botyanszki <b...@nxlog.org> wrote: > > > Hi Cameron, > > > > Doing this sort of source routing can be done using separate ports as you > > have described. This has the benefit of zero processing overhead. > > > > Otherwise you might want to consider using a single port and do the > > filtering in nxlog based on the fields present in the log. This would > > allow some simplification and possibly a lot more flexibility. If you use > > JSON or the nxlog binary format to send from the clients you can also tag > > the data on the client side. > > > > Regards, > > Botond > > > > On Mon, 14 Jul 2014 15:41:51 +1200 > > Cameron Kerr <cameron.kerr...@gmail.com> wrote: > > > > > Hi all, > > > > > > I'm embarking on a project to centralise system and application logs from > > > our Linux and Windows (and other) machines. We've shortlisted nxlog as > > > being a strong contender for doing the actual centralisation, and we'll > > use > > > something else (perhaps logstash) to further analyse and process some > > > subset of the logs. > > > > > > I want to make it easy to use, particularly around data retention. There > > > will be a lot of data coming into this, and I want to make a tiered > > > retention service. > > > > > > I'm thinking that I could have a selection of different ports that you > > > could send to; each being processed in a different nxlog route, which > > would > > > go through a different log rotation block. The client system would be > > > responsible for sending their log data to the appropriate retention tier. > > > > > > Besides tiered retention, I also want to have tiered storage (local disk > > to > > > write incoming logs to, and then rotate to a remote file-system) > > > > > > I'd also like to have machines in different parts of the network log to > > > different (closest ideally, with a fail-over) nxlog servers, but that > > > doesn't seem to concern me too much with regard to this configuration, > > > other than how files are rotated. > > > > > > For argument's sake, the retention tiers would be 1 week, 3 months, and 7 > > > years. > > > > > > To avoid vendor lock-in, I'd prefer to normalise logs to JSON, or keep > > them > > > as raw (which has some advantages, I suppose)... but I'd like to hear > > what > > > people's thoughts are on that. > > > > > > I'd also like to be able for a client system to give some sort of > > > indication as to what type of component it came from... eg. webserver, > > > applicationserver, syslog, .... which ideally could be used in later > > > processing. Not sure what really works well for that, so would love some > > > feedback on that. > > > > > > Here is my sketched configuration thus far... I'd appreciate some > > guidance > > > as to whether this seems like a good idea, and how it could be improved > > > etc. I'm laying them down on disk without reference to $Hostname, as I > > > don't want to create a DNS storm, as I have recently experienced with a > > > similar configuration in rsyslogd. > > > > > > # I think Macros can be used to reduce duplication of these definitions > > > > > > <Input incoming_syslog_with_1wk_lifetime> > > > Module im_tcp > > > Port 5141 > > > Host 0.0.0.0 > > > Exec parse_syslog_bsd(); > > > </Input> > > > > > > <Input incoming_syslog_with_3mnth_lifetime> > > > Module im_tcp > > > Port 5143 > > > Host 0.0.0.0 > > > Exec parse_syslog_bsd(); > > > </Input> > > > > > > <Input incoming_syslog_with_7yr_lifetime> > > > Module im_tcp > > > Port 5147 > > > Host 0.0.0.0 > > > Exec parse_syslog_bsd(); > > > </Input> > > > > > > # Conceivably another set of ports could be used to request > > > deliverance to something like Logstash. > > > # Eg. 5147 is 7yr streamed to disk, while 5157 is streamed to disk AND > > > loaded into Logstash... > > > # I'm ambivalent as to whether this is a good idea... so its perhaps > > > best avoided for now. > > > > > > <Output output_to_disk_with_1wk_lifetime> > > > Module om_file > > > File "/logs/current/" + $SourceIPAddress + "_1wk.log" > > > Exec if out->file_size() > 15M \ > > > { \ > > > $newfile = "/logs/archive/" + $Hostname + "/" + > > > strftime(now(), "%Y%m%d%H%M%S") + "_recv_by_THISNXLOGSERVER_1wk.log"; > > > \ > > > out->rotate_to($newfile); \ > > > exec_async("/bin/bzip2", $newfile); \ > > > } > > > </Output> > > > > > > <Output output_to_disk_with_3mnth_lifetime> > > > Module om_file > > > File "/logs/current/" + $SourceIPAddress + "_3mnth.log" > > > Exec if out->file_size() > 15M \ > > > { \ > > > $newfile = "/logs/archive/" + $Hostname + "/" + > > > strftime(now(), "%Y%m%d%H%M%S") + > > > "_recv_by_THISNXLOGSERVER_3mnth.log"; \ > > > out->rotate_to($newfile); \ > > > exec_async("/bin/bzip2", $newfile); \ > > > } > > > </Output> > > > > > > <Output output_to_disk_with_7yr_lifetime> > > > Module om_file > > > File "/logs/current/" + $SourceIPAddress + "_7yr" > > > Exec if out->file_size() > 15M \ > > > { \ > > > $newfile = "/logs/archive/" + $Hostname + "/" + > > > strftime(now(), "%Y%m%d%H%M%S") + "_recv_by_THISNXLOGSERVER_7yr.log"; > > > \ > > > out->rotate_to($newfile); \ > > > exec_async("/bin/bzip2", $newfile); \ > > > } > > > </Output> > > > > > > <Route 1wk> > > > Path incoming_syslog_with_1wk_lifetime => > > output_to_disk_with_1wk_lifetime > > > </Route> > > > > > > <Route 3mnth> > > > Path incoming_syslog_with_3mnth_lifetime => > > > output_to_disk_with_3mnth_lifetime > > > </Route> > > > > > > <Route 7yr> > > > Path incoming_syslog_with_7yr_lifetime => > > output_to_disk_with_7yr_lifetime > > > </Route> > > > > > > > > > > > > Thanks very much, > > > Cameron > > > > > > -- > > > Cameron Kerr <cameron.kerr...@gmail.com> > > > See my blog at http://distracted-it.blogspot.co.nz/ (previously > > > http://humbledown.org/) > > > Skype me on cameron.kerr.nz > > > > > > ------------------------------------------------------------------------------ > > Want fast and easy access to all the code in your enterprise? Index and > > search up to 200,000 lines of code with a free copy of Black Duck > > Code Sight - the same software that powers the world's largest code > > search on Ohloh, the Black Duck Open Hub! Try it now. > > http://p.sf.net/sfu/bds > > _______________________________________________ > > nxlog-ce-users mailing list > > nxlog-ce-users@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users > > ------------------------------------------------------------------------------ Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds _______________________________________________ nxlog-ce-users mailing list nxlog-ce-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
