Hi, There is no maximum limit. The number in the DEBUG log below is only to limit files which can be open at the same time. Even if there are more files than this which were appended to, all should be detected and read. There is some work in progress which should address situations where the current im_file implementation seems to have problems.
Regards, Botond On Tue, 1 Oct 2013 20:34:14 -0700 DMF Lists <dmfli...@gmail.com> wrote: > Hi Botond, > > I finally had some time to try and dig into this. I was able to trace it to > the client nxlog process not sending the logs. I set the log level to debug > and noticed several lines with: > > 2013-10-01 20:09:57 DEBUG opening > C:\Inetpub\logs\W3SVC123456789\ex131002.log > 2013-10-01 20:09:57 DEBUG maximum number (21) of files open, closing current > > Is there a maximum limit of log files that can be opened by nxlog? I'm > renaming my logs to miss the wildcard path but I still have roughly 50 iis > files that nxlog would have to keep open at any given time. > > Thanks. > > > > > On Fri, Aug 2, 2013 at 10:28 AM, DMF Lists <dmfli...@gmail.com> wrote: > > > Right, I am renaming the files to avoid the wildcard masking every couple > > of days. I'll try and reproduce the scenario and submit a bug. > > > > > > On Fri, Aug 2, 2013 at 8:56 AM, Botond Botyanszki <b...@nxlog.org> wrote: > > > >> Hi, > >> > >> Rotated log files (including IIS) should work ok using a wildcarded File > >> directive. Actually by default IIS always starts a new file daily and > >> there is no rotation, but it might be configured differently in your > >> setup. If you have rotated files make sure to have them rotated to > >> another directory or at least in such a way that the wildcarded name > >> doesn't match those. > >> > >> Regards, > >> Botond > >> > >> On Fri, 2 Aug 2013 01:20:25 -0700 > >> DMF Lists <dmfli...@gmail.com> wrote: > >> > >> > Thanks Botond, > >> > > >> > That's why I haven't been using the pm_buffer module. However, now that > >> I > >> > think of it, it's generally the iis logs that seem to stop being > >> forwarded. > >> > Is it possible that if the flow control pauses the clients and the file > >> > gets rotated while it's paused, nxlog doesn't know how to continue? What > >> > would you suggest for iis logs that are rolled daily? > >> > > >> > I was asking about redis/ampq support for the same reason, use it as a > >> > queue. I'll take a look at those options you mentioned. Thanks. > >> > > >> > > >> > On Fri, Aug 2, 2013 at 1:02 AM, Botond Botyanszki <b...@nxlog.org> > >> wrote: > >> > > >> > > Hi, > >> > > > >> > > Ok, I think some misconceptions needs to be cleared up because a lot > >> > > of people tend to think nxlog will drop messages, probably due to > >> > > experience with other log collector software that behaves this way. > >> > > > >> > > See the note at > >> > > > >> > > > >> http://nxlog-ce.sourceforge.net/nxlog-docs/en/nxlog-reference-manual.html#pm_buffer > >> > > Also see the FlowControl directive: > >> > > > >> > > > >> http://nxlog-ce.sourceforge.net/nxlog-docs/en/nxlog-reference-manual.html#config_module_flowcontrol > >> > > > >> > > In short: > >> > > if you are using im_msvistalog => om_tcp you will not need to use > >> > > pm_buffer or any other external buffering solution such as redis. When > >> > > om_tcp detects that it cannot send, the input module (im_msvistalog) > >> will > >> > > stop reading logs. When the tcp connection is resumed, the input > >> module > >> > > will resume reading from the same position and will properly collect > >> all > >> > > messages that were stored in the windows eventlog during that time. > >> This > >> > > works the same way with im_file. This should also work if you restart > >> > > nxlog during this time as it will remember the last position read in > >> the > >> > > input. > >> > > > >> > > The only situation this can go wrong is when there is log rotation > >> set up > >> > > for the windows eventlog, or the log file read by im_file. When nxlog > >> > > tries to resume, the input source is no longer there. Normally this > >> > > shouldn't be a problem. > >> > > > >> > > Redis and AMPQ support would be nice to have, there are plans to have > >> > > support for these , unfortunately there are always higher priority > >> > > issues/features. If someone would volunteer, I'd be happy to help. > >> > > > >> > > Interestingly the exact same issue has been discussed on #nxlog > >> yesterday: > >> > > > >> > > (04:52:10 PM) nemish: b0ti: does nxlog support redis output? > >> > > (04:53:39 PM) b0ti: nemish: not directly > >> > > (04:54:58 PM) b0ti: there are different options: use some script with > >> > > om_exec > >> > > (04:55:01 PM) b0ti: > >> > > https://github.com/blakeblackshear/nxlog-to-redis-proxy > >> > > (04:55:24 PM) b0ti: nxlog => logstash => redis > >> > > (04:57:10 PM) b0ti: another option might be om_http + webdis > >> > > 04:57:02 PM) nemish: b0ti: yeah i was trying to do nxlog -> redis -> > >> > > logstash or some intermediary queue in case LS is down or being > >> recycled > >> > > (04:58:12 PM) b0ti: so you only want to use redis as a buffer? > >> > > > >> > > Regards, > >> > > Botond > >> > > > >> > > > >> > > On Fri, 2 Aug 2013 00:30:23 -0700 > >> > > DMF Lists <dmfli...@gmail.com> wrote: > >> > > > >> > > > I was expecting that even without a buffer that nxlog would > >> reconnect to > >> > > > it's outputs once the target started accepting connections again. > >> > > > > >> > > > Botond, do you have any plans to support Redis or RabbitMQ(AMPQ) as > >> an > >> > > > output? > >> > > > > >> > > > > >> > > > On Thu, Aug 1, 2013 at 6:12 PM, Mark D. Nagel < > >> mna...@willingminds.com > >> > > >wrote: > >> > > > > >> > > > > On 8/1/2013 5:49 PM, DMF Lists wrote: > >> > > > > > I'm setup to ship windows logs to a central linux host running > >> nxlog > >> > > via > >> > > > > native > >> > > > > > transport which then ships it off to logstash via generic TCP. > >> If > >> > > > > logstash backs up or > >> > > > > > dies and it's caught immediately and restarted, the nxlog > >> transport > >> > > > > chain seems to > >> > > > > > survive. However, if logstash is stopped for a long period of > >> time > >> > > ie a > >> > > > > few hours, not > >> > > > > > all windows nxlog clients will reconnect properly. The nxlog > >> logs > >> > > just > >> > > > > show the > >> > > > > > following mutiple times: > >> > > > > > > >> > > > > > 2013-03-27 14:49:59 ERROR couldn't connect to tcp socket on > >> > > logger1:514 > >> > > > > No connection > >> > > > > > could be made because the target machine actively refused it. > >> > > > > > 2013-03-27 14:50:00 INFO reconnecting in 10 seconds > >> > > > > > > >> > > > > > Once I restart the nxlog client on the windows side, it starts > >> > > logging > >> > > > > again. Any thoughts? > >> > > > > > >> > > > > I believe this is a use case for pm_buffer - > >> > > > > > >> > > > > > >> > > > >> http://nxlog-ce.sourceforge.net/nxlog-docs/en/nxlog-reference-manual.html#pm_buffer > >> > > . > >> > > > > You > >> > > > > just need to figure out how much disk storage to reserve for your > >> worst > >> > > > > case. > >> > > > > > >> > > > > Regards, > >> > > > > Mark > >> > > > > > >> > > > > -- > >> > > > > Mark D. Nagel, CCIE #3177 <mna...@willingminds.com> > >> > > > > Principal Consultant, Willing Minds LLC ( > >> http://www.willingminds.com) > >> > > > > cell: 949-279-5817, desk: 714-495-4001, fax: 714-646-8277 > >> > > > > > >> > > > > ** For faster support response time, please > >> > > > > ** email supp...@willingminds.com or call 714-495-4000 > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > >> ------------------------------------------------------------------------------ > >> > > > > Get your SQL database under version control now! > >> > > > > Version control is standard for application code, but databases > >> havent > >> > > > > caught up. So what steps can you take to put your SQL databases > >> under > >> > > > > version control? Why should you start doing it? Read more to find > >> out. > >> > > > > > >> > > > >> http://pubads.g.doubleclick.net/gampad/clk?id=49501711&iu=/4140/ostg.clktrk > >> > > > > _______________________________________________ > >> > > > > nxlog-ce-users mailing list > >> > > > > nxlog-ce-users@lists.sourceforge.net > >> > > > > https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users > >> > > > > > >> > > > >> > > > >> > > > >> ------------------------------------------------------------------------------ > >> > > Get your SQL database under version control now! > >> > > Version control is standard for application code, but databases havent > >> > > caught up. So what steps can you take to put your SQL databases under > >> > > version control? Why should you start doing it? Read more to find out. > >> > > > >> http://pubads.g.doubleclick.net/gampad/clk?id=49501711&iu=/4140/ostg.clktrk > >> > > _______________________________________________ > >> > > nxlog-ce-users mailing list > >> > > nxlog-ce-users@lists.sourceforge.net > >> > > https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users > >> > > > >> > > > > ------------------------------------------------------------------------------ October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register > http://pubads.g.doubleclick.net/gampad/clk?id=60134791&iu=/4140/ostg.clktrk _______________________________________________ nxlog-ce-users mailing list nxlog-ce-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
