Hi Jeff, The Recursive directive for im_file defaults to TRUE, as such your containers_in instance reads the files that nginx_in and nsq_in also read. The purpose of Recursive is to make it simpler to collect logs such as this case so that you don't need to define a lot of additional im_file instances.
Regards, Botond On Fri, 25 Jul 2014 15:37:33 -0400 Jeffrey Valeo <jva...@mac.com> wrote: > I’ve noticed all of my logs are being duplicated as they’re sent over UDP to > my logstash stack. I bypassed logstash by just listening on UDP and I can see > each log is coming in 2 times, a tcpdump confirms the same. It’s happening > will all of my logs, including ones that log rotate has never touched. Here > is my nxlog config. Is there something I’m missing? > > > ######################################## > # Global directives # > ######################################## > User nxlog > Group root > > LogFile /var/log/nxlog/nxlog.log > LogLevel INFO > > ######################################## > # Modules # > ######################################## > <Extension json> > Module xm_json > </Extension> > > <Input nginx_in> > Module im_file > File '/var/log/containers/nginx/*.access.log' > Exec $FileName = file_name(); > Exec $raw_event = $FileName + ' ' + $raw_event ; > Exec $Message = $raw_event ; > </Input> > <Input containers_in> > Module im_file > File '/var/log/containers/*.log' > Exec $FileName = file_name(); > Exec $raw_event = $FileName + ' ' + $raw_event ; > Exec $Message = $raw_event ; > </Input> > <Input upstart_in> > Module im_file > File '/var/log/upstart/*.log' > Exec $FileName = file_name(); > Exec $raw_event = $FileName + ' ' + $raw_event ; > Exec $Message = $raw_event ; > </Input> > <Input nsq_in> > Module im_file > File '/var/log/containers/nsq_to_file/*' > Exec $FileName = file_name(); > Exec $raw_event = 'nsq_to_file' + ' ' + $FileName + ' ' + $raw_event ; > Exec $Message = $raw_event ; > </Input> > > #Buffer to UDP > <Processor buffer_udp> > Module pm_buffer > MaxSize 2044 > Type Mem > WarnLimit 1512 > </Processor> > > #Output for Production Logstash > <Output logstash_prod> > Module om_udp > Port 5140 > Host logstash__DOMAIN > </Output> > > #Output > <Route logstash_out> > Path nginx_in, containers_in, upstart_in, nsq_in => buffer_udp => > logstash_prod > </Route> > ------------------------------------------------------------------------------ > Want fast and easy access to all the code in your enterprise? Index and > search up to 200,000 lines of code with a free copy of Black Duck > Code Sight - the same software that powers the world's largest code > search on Ohloh, the Black Duck Open Hub! Try it now. > http://p.sf.net/sfu/bds > _______________________________________________ > nxlog-ce-users mailing list > nxlog-ce-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users ------------------------------------------------------------------------------ Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now. http://p.sf.net/sfu/bds _______________________________________________ nxlog-ce-users mailing list nxlog-ce-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
