Hi Alec, To get around some of the issues with imfile and the default inotify mode, we changed to module(load="imfile" mode="polling"). We don't see any issues with rotated logs.
Ciprian -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Sun, Dec 13, 2015 at 7:49 PM, Alec Swan <[email protected]> wrote: > David, cassandra.log file exists when I start rsyslog. However, I noticed > that Cassandra recreates the log file every time it restarts. So, that > explains why I need to restart rsyslog (so that it re-reads the new file > recreated by Cassandra) after Cassandra restarts. > Is there a way to configure rsyslog to read files which get periodically > recreated? > > Thanks, > > Alec > > On Sun, Dec 13, 2015 at 1:07 AM, David Lang <[email protected]> wrote: > > > I suspect that no logs are being read from the file before you restart. > > > > does the logfile exist when rsyslog starts? > > > > if you do a HUP of rsyslog instead of a restart, does it start reading > > logs from the file? > > > > impstats would show how many logs have been processed from that input. > > > > David Lang > > > > On Sun, 13 Dec 2015, Alec Swan wrote: > > > > Date: Sun, 13 Dec 2015 00:26:40 -0700 > >> From: Alec Swan <[email protected]> > >> Reply-To: rsyslog-users <[email protected]> > >> To: rsyslog-users <[email protected]> > >> Subject: Re: [rsyslog] Delayed batch processing > >> > >> > >> Ciprian, that's correct, I am using imfile to tail cassandra.log. Sorry > I > >> wasn't clear about that. I am not using startmsg.regex setting. My > >> configuration for imfile, template and liblognorm rule are shown below. > >> I'd > >> like to make clear that it's dozens of log lines, not just the last one, > >> that are not being sent. Note that Cassandra log lines start with a > space, > >> which I handle in liblognorm rule (see below). > >> > >> input( > >> type = "imfile" > >> File = "/var/log/cassandra/cassandra.log" > >> Tag = "cassandra" > >> readMode = "0" > >> Facility = "local2" > >> Ruleset = "cassandra.log" > >> ) > >> > >> ruleset(name = "cassandra.log") { > >> # Properties prefixed with $! will be included in json by > mmnormalize > >> #set $!@timestamp = $timereported; > >> set $!@timestamp = exec_template("timereportedrfc3339"); > >> set $!host = $hostname; > >> set $!severity = $syslogseverity-text; > >> set $!facility = $syslogfacility-text; > >> set $!syslogtag = $syslogtag; > >> set $!filename = "cassandra.log"; > >> > >> # Parse logfile using liblognorm > >> > >> action(type = "mmnormalize" rulebase = > >> "/etc/rsyslog.d/rules/cassandra.log.rb") > >> > >> > >> # Output parsed logs > >> action(type = "omfile" template="es-payload" > >> file="/var/log/rsyslog/rsyslog-cassandra.log" FileCreateMode="0644") > >> action( > >> type = "omelasticsearch" > >> template = "es-payload" > >> dynSearchIndex = "on" > >> searchIndex = "logstash-index" > >> searchType = "cassandra" > >> server = "myhost" > >> serverport = "9200" > >> errorFile = "/var/log/rsyslog/ES-error.log" > >> bulkmode = "on" > >> action.resumeretrycount="5" # retry if ES is unreachable (-1 > for > >> infinite retries) > >> action.resumeInterval="60" > >> queue.dequeuebatchsize="1000" # ES bulk size > >> queue.type="FixedArray" > >> queue.size="100000" > >> queue.workerthreads="5" > >> queue.spoolDirectory="/var/spool/rsyslog" > >> queue.filename="plain" > >> queue.maxfilesize="100m" > >> queue.maxdiskspace="1g" > >> # queue.highwatermark="50000" > >> # queue.lowwatermark="20000" > >> queue.saveonshutdown="on" > >> ) > >> stop > >> } > >> > >> > >> Liblognorm configuration in /etc/rsyslog.d/rules/cassandra.log.rb: > >> > >> > >> # The rule to parse each line in the log file. IMPORTANT: The leading > >> space > >> is important (see > >> http://www.rsyslog.com/log-normalization-and-the-leading-space/) > >> rule=: %log_level:word% %log_time:word% %message:rest% > >> > >> On Sun, Dec 13, 2015 at 12:01 AM, Ciprian Hacman < > >> [email protected]> wrote: > >> > >> I see Alec is using imfile and omelasticsearch so I suppose he is > tailing > >>> a > >>> log file. > >>> By any chance, are you using Cassandra to capture multiline logs (using > >>> the > >>> startmsg.regex setting)? That would explain why the last log line is > not > >>> sent until restart. > >>> > >>> Ciprian > >>> > >>> -- > >>> Performance Monitoring * Log Analytics * Search Analytics > >>> Solr & Elasticsearch Support * http://sematext.com/ > >>> > >>> On Sun, Dec 13, 2015 at 6:45 AM, David Lang <[email protected]> wrote: > >>> > >>> On Sat, 12 Dec 2015, Alec Swan wrote: > >>>> > >>>> David, I don't have name= attribute in the action statement. However, > >>>> > >>> there > >>> > >>>> are log messages from omelasticsearch module after rsyslog restart > (see > >>>>> > >>>> my > >>> > >>>> previous email). > >>>>> > >>>>> Cassandra does not generate a log of logs and I noticed that the few > >>>>> > >>>> that > >>> > >>>> get generated are not sent to elasticsearch until rsyslog is > restarted. > >>>>> > >>>> It > >>> > >>>> feels like logs are being buffered and then sent to ES on subsequent > >>>>> startup. > >>>>> > >>>>> So, here is what I see happening: > >>>>> 1. I start Rsyslog > >>>>> 2. I start Cassandra and it generates several dozens of log lines > >>>>> 3. No logs show up in ES (even if I restart Cassandra several times) > >>>>> 4. I restart Rsyslog > >>>>> 5. The logs generated in step 2 show up in ES > >>>>> > >>>>> > >>>> This does not sound like rsyslog is buffering the logs, but that there > >>>> is > >>>> some other problem. > >>>> > >>>> > >>>> how are the logs getting from Cassandra to rsyslog? > >>>> > >>>> can you enable impstats so that you can see how many logs are > received, > >>>> and how many are processed by the various actions? > >>>> > >>>> it would help to name new style actions so that it's easier to track > >>>> > >>> them. > >>> > >>>> > >>>> in your debug log (the output of -dn), track a known message that > >>>> Cassandra generates and see exactly what happens to it. > >>>> > >>>> > >>>> David Lang > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com/professional-services/ > >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>>> DON'T LIKE THAT. > >>>> > >>>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com/professional-services/ > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>> DON'T LIKE THAT. > >>> > >>> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > >> > >> _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
