David, cassandra.log file exists when I start rsyslog. However, I noticed that Cassandra recreates the log file every time it restarts. So, that explains why I need to restart rsyslog (so that it re-reads the new file recreated by Cassandra) after Cassandra restarts. Is there a way to configure rsyslog to read files which get periodically recreated?
Thanks, Alec On Sun, Dec 13, 2015 at 1:07 AM, David Lang <[email protected]> wrote: > I suspect that no logs are being read from the file before you restart. > > does the logfile exist when rsyslog starts? > > if you do a HUP of rsyslog instead of a restart, does it start reading > logs from the file? > > impstats would show how many logs have been processed from that input. > > David Lang > > On Sun, 13 Dec 2015, Alec Swan wrote: > > Date: Sun, 13 Dec 2015 00:26:40 -0700 >> From: Alec Swan <[email protected]> >> Reply-To: rsyslog-users <[email protected]> >> To: rsyslog-users <[email protected]> >> Subject: Re: [rsyslog] Delayed batch processing >> >> >> Ciprian, that's correct, I am using imfile to tail cassandra.log. Sorry I >> wasn't clear about that. I am not using startmsg.regex setting. My >> configuration for imfile, template and liblognorm rule are shown below. >> I'd >> like to make clear that it's dozens of log lines, not just the last one, >> that are not being sent. Note that Cassandra log lines start with a space, >> which I handle in liblognorm rule (see below). >> >> input( >> type = "imfile" >> File = "/var/log/cassandra/cassandra.log" >> Tag = "cassandra" >> readMode = "0" >> Facility = "local2" >> Ruleset = "cassandra.log" >> ) >> >> ruleset(name = "cassandra.log") { >> # Properties prefixed with $! will be included in json by mmnormalize >> #set $!@timestamp = $timereported; >> set $!@timestamp = exec_template("timereportedrfc3339"); >> set $!host = $hostname; >> set $!severity = $syslogseverity-text; >> set $!facility = $syslogfacility-text; >> set $!syslogtag = $syslogtag; >> set $!filename = "cassandra.log"; >> >> # Parse logfile using liblognorm >> >> action(type = "mmnormalize" rulebase = >> "/etc/rsyslog.d/rules/cassandra.log.rb") >> >> >> # Output parsed logs >> action(type = "omfile" template="es-payload" >> file="/var/log/rsyslog/rsyslog-cassandra.log" FileCreateMode="0644") >> action( >> type = "omelasticsearch" >> template = "es-payload" >> dynSearchIndex = "on" >> searchIndex = "logstash-index" >> searchType = "cassandra" >> server = "myhost" >> serverport = "9200" >> errorFile = "/var/log/rsyslog/ES-error.log" >> bulkmode = "on" >> action.resumeretrycount="5" # retry if ES is unreachable (-1 for >> infinite retries) >> action.resumeInterval="60" >> queue.dequeuebatchsize="1000" # ES bulk size >> queue.type="FixedArray" >> queue.size="100000" >> queue.workerthreads="5" >> queue.spoolDirectory="/var/spool/rsyslog" >> queue.filename="plain" >> queue.maxfilesize="100m" >> queue.maxdiskspace="1g" >> # queue.highwatermark="50000" >> # queue.lowwatermark="20000" >> queue.saveonshutdown="on" >> ) >> stop >> } >> >> >> Liblognorm configuration in /etc/rsyslog.d/rules/cassandra.log.rb: >> >> >> # The rule to parse each line in the log file. IMPORTANT: The leading >> space >> is important (see >> http://www.rsyslog.com/log-normalization-and-the-leading-space/) >> rule=: %log_level:word% %log_time:word% %message:rest% >> >> On Sun, Dec 13, 2015 at 12:01 AM, Ciprian Hacman < >> [email protected]> wrote: >> >> I see Alec is using imfile and omelasticsearch so I suppose he is tailing >>> a >>> log file. >>> By any chance, are you using Cassandra to capture multiline logs (using >>> the >>> startmsg.regex setting)? That would explain why the last log line is not >>> sent until restart. >>> >>> Ciprian >>> >>> -- >>> Performance Monitoring * Log Analytics * Search Analytics >>> Solr & Elasticsearch Support * http://sematext.com/ >>> >>> On Sun, Dec 13, 2015 at 6:45 AM, David Lang <[email protected]> wrote: >>> >>> On Sat, 12 Dec 2015, Alec Swan wrote: >>>> >>>> David, I don't have name= attribute in the action statement. However, >>>> >>> there >>> >>>> are log messages from omelasticsearch module after rsyslog restart (see >>>>> >>>> my >>> >>>> previous email). >>>>> >>>>> Cassandra does not generate a log of logs and I noticed that the few >>>>> >>>> that >>> >>>> get generated are not sent to elasticsearch until rsyslog is restarted. >>>>> >>>> It >>> >>>> feels like logs are being buffered and then sent to ES on subsequent >>>>> startup. >>>>> >>>>> So, here is what I see happening: >>>>> 1. I start Rsyslog >>>>> 2. I start Cassandra and it generates several dozens of log lines >>>>> 3. No logs show up in ES (even if I restart Cassandra several times) >>>>> 4. I restart Rsyslog >>>>> 5. The logs generated in step 2 show up in ES >>>>> >>>>> >>>> This does not sound like rsyslog is buffering the logs, but that there >>>> is >>>> some other problem. >>>> >>>> >>>> how are the logs getting from Cassandra to rsyslog? >>>> >>>> can you enable impstats so that you can see how many logs are received, >>>> and how many are processed by the various actions? >>>> >>>> it would help to name new style actions so that it's easier to track >>>> >>> them. >>> >>>> >>>> in your debug log (the output of -dn), track a known message that >>>> Cassandra generates and see exactly what happens to it. >>>> >>>> >>>> David Lang >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> DON'T LIKE THAT. >>>> >>>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
