Hi, Ok, I think some misconceptions needs to be cleared up because a lot of people tend to think nxlog will drop messages, probably due to experience with other log collector software that behaves this way.
See the note at http://nxlog-ce.sourceforge.net/nxlog-docs/en/nxlog-reference-manual.html#pm_buffer Also see the FlowControl directive: http://nxlog-ce.sourceforge.net/nxlog-docs/en/nxlog-reference-manual.html#config_module_flowcontrol In short: if you are using im_msvistalog => om_tcp you will not need to use pm_buffer or any other external buffering solution such as redis. When om_tcp detects that it cannot send, the input module (im_msvistalog) will stop reading logs. When the tcp connection is resumed, the input module will resume reading from the same position and will properly collect all messages that were stored in the windows eventlog during that time. This works the same way with im_file. This should also work if you restart nxlog during this time as it will remember the last position read in the input. The only situation this can go wrong is when there is log rotation set up for the windows eventlog, or the log file read by im_file. When nxlog tries to resume, the input source is no longer there. Normally this shouldn't be a problem. Redis and AMPQ support would be nice to have, there are plans to have support for these , unfortunately there are always higher priority issues/features. If someone would volunteer, I'd be happy to help. Interestingly the exact same issue has been discussed on #nxlog yesterday: (04:52:10 PM) nemish: b0ti: does nxlog support redis output? (04:53:39 PM) b0ti: nemish: not directly (04:54:58 PM) b0ti: there are different options: use some script with om_exec (04:55:01 PM) b0ti: https://github.com/blakeblackshear/nxlog-to-redis-proxy (04:55:24 PM) b0ti: nxlog => logstash => redis (04:57:10 PM) b0ti: another option might be om_http + webdis 04:57:02 PM) nemish: b0ti: yeah i was trying to do nxlog -> redis -> logstash or some intermediary queue in case LS is down or being recycled (04:58:12 PM) b0ti: so you only want to use redis as a buffer? Regards, Botond On Fri, 2 Aug 2013 00:30:23 -0700 DMF Lists <dmfli...@gmail.com> wrote: > I was expecting that even without a buffer that nxlog would reconnect to > it's outputs once the target started accepting connections again. > > Botond, do you have any plans to support Redis or RabbitMQ(AMPQ) as an > output? > > > On Thu, Aug 1, 2013 at 6:12 PM, Mark D. Nagel <mna...@willingminds.com>wrote: > > > On 8/1/2013 5:49 PM, DMF Lists wrote: > > > I'm setup to ship windows logs to a central linux host running nxlog via > > native > > > transport which then ships it off to logstash via generic TCP. If > > logstash backs up or > > > dies and it's caught immediately and restarted, the nxlog transport > > chain seems to > > > survive. However, if logstash is stopped for a long period of time ie a > > few hours, not > > > all windows nxlog clients will reconnect properly. The nxlog logs just > > show the > > > following mutiple times: > > > > > > 2013-03-27 14:49:59 ERROR couldn't connect to tcp socket on logger1:514 > > No connection > > > could be made because the target machine actively refused it. > > > 2013-03-27 14:50:00 INFO reconnecting in 10 seconds > > > > > > Once I restart the nxlog client on the windows side, it starts logging > > again. Any thoughts? > > > > I believe this is a use case for pm_buffer - > > > > http://nxlog-ce.sourceforge.net/nxlog-docs/en/nxlog-reference-manual.html#pm_buffer. > > You > > just need to figure out how much disk storage to reserve for your worst > > case. > > > > Regards, > > Mark > > > > -- > > Mark D. Nagel, CCIE #3177 <mna...@willingminds.com> > > Principal Consultant, Willing Minds LLC (http://www.willingminds.com) > > cell: 949-279-5817, desk: 714-495-4001, fax: 714-646-8277 > > > > ** For faster support response time, please > > ** email supp...@willingminds.com or call 714-495-4000 > > > > > > > > ------------------------------------------------------------------------------ > > Get your SQL database under version control now! > > Version control is standard for application code, but databases havent > > caught up. So what steps can you take to put your SQL databases under > > version control? Why should you start doing it? Read more to find out. > > http://pubads.g.doubleclick.net/gampad/clk?id=49501711&iu=/4140/ostg.clktrk > > _______________________________________________ > > nxlog-ce-users mailing list > > nxlog-ce-users@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users > > ------------------------------------------------------------------------------ Get your SQL database under version control now! Version control is standard for application code, but databases havent caught up. So what steps can you take to put your SQL databases under version control? Why should you start doing it? Read more to find out. http://pubads.g.doubleclick.net/gampad/clk?id=49501711&iu=/4140/ostg.clktrk _______________________________________________ nxlog-ce-users mailing list nxlog-ce-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
