That looks very useful. I look forward to trying it on Monday.
Thanks!
On Saturday, 19 July 2014, Botond Botyanszki <b...@nxlog.org> wrote:
> Hi,
>
> Example:
>
> <Extension json>
> Module xm_json
> </Extension>
>
> <Output o>
> Module om_tcp
> Host 1.2.3.4
> Port 1234
> Exec $Source = "myapp"; to_json();
> </Output>
>
> Note that the above is not syslog. If you want to wrap it into syslog,
> that's possible too:
> Exec $Source = "myapp"; $Message = to_json(); to_syslog_bsd();
>
> HTH,
> Botond
>
> On Fri, 18 Jul 2014 19:48:32 +1200
> Cameron Kerr <cameron.kerr...@gmail.com <javascript:;>> wrote:
>
> > Thank you for you response. Could you give any example of how you might
> add
> > tags in the incoming JSON from the client-side nxlog?
> >
> > Cheers,
> > Cameron
> >
> > --
> > Cameron Kerr <cameron.kerr...@gmail.com <javascript:;>>
> > See my blog at http://distracted-it.blogspot.co.nz/ (previously
> > http://humbledown.org/)
> > Skype me on cameron.kerr.nz
> >
> >
> >
> > On 17 July 2014 20:57, Botond Botyanszki <b...@nxlog.org <javascript:;>>
> wrote:
> >
> > > Hi Cameron,
> > >
> > > Doing this sort of source routing can be done using separate ports as
> you
> > > have described. This has the benefit of zero processing overhead.
> > >
> > > Otherwise you might want to consider using a single port and do the
> > > filtering in nxlog based on the fields present in the log. This would
> > > allow some simplification and possibly a lot more flexibility. If you
> use
> > > JSON or the nxlog binary format to send from the clients you can also
> tag
> > > the data on the client side.
> > >
> > > Regards,
> > > Botond
> > >
> > > On Mon, 14 Jul 2014 15:41:51 +1200
> > > Cameron Kerr <cameron.kerr...@gmail.com <javascript:;>> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'm embarking on a project to centralise system and application logs
> from
> > > > our Linux and Windows (and other) machines. We've shortlisted nxlog
> as
> > > > being a strong contender for doing the actual centralisation, and
> we'll
> > > use
> > > > something else (perhaps logstash) to further analyse and process some
> > > > subset of the logs.
> > > >
> > > > I want to make it easy to use, particularly around data retention.
> There
> > > > will be a lot of data coming into this, and I want to make a tiered
> > > > retention service.
> > > >
> > > > I'm thinking that I could have a selection of different ports that
> you
> > > > could send to; each being processed in a different nxlog route, which
> > > would
> > > > go through a different log rotation block. The client system would be
> > > > responsible for sending their log data to the appropriate retention
> tier.
> > > >
> > > > Besides tiered retention, I also want to have tiered storage (local
> disk
> > > to
> > > > write incoming logs to, and then rotate to a remote file-system)
> > > >
> > > > I'd also like to have machines in different parts of the network log
> to
> > > > different (closest ideally, with a fail-over) nxlog servers, but that
> > > > doesn't seem to concern me too much with regard to this
> configuration,
> > > > other than how files are rotated.
> > > >
> > > > For argument's sake, the retention tiers would be 1 week, 3 months,
> and 7
> > > > years.
> > > >
> > > > To avoid vendor lock-in, I'd prefer to normalise logs to JSON, or
> keep
> > > them
> > > > as raw (which has some advantages, I suppose)... but I'd like to hear
> > > what
> > > > people's thoughts are on that.
> > > >
> > > > I'd also like to be able for a client system to give some sort of
> > > > indication as to what type of component it came from... eg.
> webserver,
> > > > applicationserver, syslog, .... which ideally could be used in later
> > > > processing. Not sure what really works well for that, so would love
> some
> > > > feedback on that.
> > > >
> > > > Here is my sketched configuration thus far... I'd appreciate some
> > > guidance
> > > > as to whether this seems like a good idea, and how it could be
> improved
> > > > etc. I'm laying them down on disk without reference to $Hostname, as
> I
> > > > don't want to create a DNS storm, as I have recently experienced
> with a
> > > > similar configuration in rsyslogd.
> > > >
> > > > # I think Macros can be used to reduce duplication of these
> definitions
> > > >
> > > > <Input incoming_syslog_with_1wk_lifetime>
> > > > Module im_tcp
> > > > Port 5141
> > > > Host 0.0.0.0
> > > > Exec parse_syslog_bsd();
> > > > </Input>
> > > >
> > > > <Input incoming_syslog_with_3mnth_lifetime>
> > > > Module im_tcp
> > > > Port 5143
> > > > Host 0.0.0.0
> > > > Exec parse_syslog_bsd();
> > > > </Input>
> > > >
> > > > <Input incoming_syslog_with_7yr_lifetime>
> > > > Module im_tcp
> > > > Port 5147
> > > > Host 0.0.0.0
> > > > Exec parse_syslog_bsd();
> > > > </Input>
> > > >
> > > > # Conceivably another set of ports could be used to request
> > > > deliverance to something like Logstash.
> > > > # Eg. 5147 is 7yr streamed to disk, while 5157 is streamed to disk
> AND
> > > > loaded into Logstash...
> > > > # I'm ambivalent as to whether this is a good idea... so its perhaps
> > > > best avoided for now.
> > > >
> > > > <Output output_to_disk_with_1wk_lifetime>
> > > > Module om_file
> > > > File "/logs/current/" + $SourceIPAddress + "_1wk.log"
> > > > Exec if out->file_size() > 15M \
> > > > { \
> > > > $newfile = "/logs/archive/" + $Hostname + "/" +
> > > > strftime(now(), "%Y%m%d%H%M%S") + "_recv_by_THISNXLOGSERVER_1wk.log";
> > > > \
> > > > out->rotate_to($newfile); \
> > > > exec_async("/bin/bzip2", $newfile); \
> > > > }
> > > > </Output>
> > > >
> > > > <Output output_to_disk_with_3mnth_lifetime>
> > > > Module om_file
> > > > File "/logs/current/" + $SourceIPAddress + "_3mnth.log"
> > > > Exec if out->file_size() > 15M \
> > > > { \
> > > > $newfile = "/logs/archive/" + $Hostname + "/" +
> > > > strftime(now(), "%Y%m%d%H%M%S") +
> > > > "_recv_by_THISNXLOGSERVER_3mnth.log"; \
> > > > out->rotate_to($newfile); \
> > > > exec_async("/bin/bzip2", $newfile); \
> > > > }
> > > > </Output>
> > > >
> > > > <Output output_to_disk_with_7yr_lifetime>
> > > > Module om_file
> > > > File "/logs/current/" + $SourceIPAddress + "_7yr"
> > > > Exec if out->file_size() > 15M \
> > > > { \
> > > > $newfile = "/logs/archive/" + $Hostname + "/" +
> > > > strftime(now(), "%Y%m%d%H%M%S") + "_recv_by_THISNXLOGSERVER_7yr.log";
> > > > \
> > > > out->rotate_to($newfile); \
> > > > exec_async("/bin/bzip2", $newfile); \
> > > > }
> > > > </Output>
> > > >
> > > > <Route 1wk>
> > > > Path incoming_syslog_with_1wk_lifetime =>
> > > output_to_disk_with_1wk_lifetime
> > > > </Route>
> > > >
> > > > <Route 3mnth>
> > > > Path incoming_syslog_with_3mnth_lifetime =>
> > > > output_to_disk_with_3mnth_lifetime
> > > > </Route>
> > > >
> > > > <Route 7yr>
> > > > Path incoming_syslog_with_7yr_lifetime =>
> > > output_to_disk_with_7yr_lifetime
> > > > </Route>
> > > >
> > > >
> > > >
> > > > Thanks very much,
> > > > Cameron
> > > >
> > > > --
> > > > Cameron Kerr <cameron.kerr...@gmail.com <javascript:;>>
> > > > See my blog at http://distracted-it.blogspot.co.nz/ (previously
> > > > http://humbledown.org/)
> > > > Skype me on cameron.kerr.nz
> > >
> > >
> > >
> ------------------------------------------------------------------------------
> > > Want fast and easy access to all the code in your enterprise? Index and
> > > search up to 200,000 lines of code with a free copy of Black Duck
> > > Code Sight - the same software that powers the world's largest code
> > > search on Ohloh, the Black Duck Open Hub! Try it now.
> > > http://p.sf.net/sfu/bds
> > > _______________________________________________
> > > nxlog-ce-users mailing list
> > > nxlog-ce-users@lists.sourceforge.net <javascript:;>
> > > https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
> > >
>
>
> ------------------------------------------------------------------------------
> Want fast and easy access to all the code in your enterprise? Index and
> search up to 200,000 lines of code with a free copy of Black Duck
> Code Sight - the same software that powers the world's largest code
> search on Ohloh, the Black Duck Open Hub! Try it now.
> http://p.sf.net/sfu/bds
> _______________________________________________
> nxlog-ce-users mailing list
> nxlog-ce-users@lists.sourceforge.net <javascript:;>
> https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
>
--
--
Cameron Kerr <cameron.kerr...@gmail.com>
See my blog at http://distracted-it.blogspot.co.nz/ (previously
http://humbledown.org/)
Skype me on cameron.kerr.nz
------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
nxlog-ce-users mailing list
nxlog-ce-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
