I suspect that no logs are being read from the file before you restart.
does the logfile exist when rsyslog starts?
if you do a HUP of rsyslog instead of a restart, does it start reading logs from
the file?
impstats would show how many logs have been processed from that input.
David Lang
On Sun, 13 Dec 2015, Alec Swan wrote:
Date: Sun, 13 Dec 2015 00:26:40 -0700
From: Alec Swan <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Delayed batch processing
Ciprian, that's correct, I am using imfile to tail cassandra.log. Sorry I
wasn't clear about that. I am not using startmsg.regex setting. My
configuration for imfile, template and liblognorm rule are shown below. I'd
like to make clear that it's dozens of log lines, not just the last one,
that are not being sent. Note that Cassandra log lines start with a space,
which I handle in liblognorm rule (see below).
input(
type = "imfile"
File = "/var/log/cassandra/cassandra.log"
Tag = "cassandra"
readMode = "0"
Facility = "local2"
Ruleset = "cassandra.log"
)
ruleset(name = "cassandra.log") {
# Properties prefixed with $! will be included in json by mmnormalize
#set $!@timestamp = $timereported;
set $!@timestamp = exec_template("timereportedrfc3339");
set $!host = $hostname;
set $!severity = $syslogseverity-text;
set $!facility = $syslogfacility-text;
set $!syslogtag = $syslogtag;
set $!filename = "cassandra.log";
# Parse logfile using liblognorm
action(type = "mmnormalize" rulebase =
"/etc/rsyslog.d/rules/cassandra.log.rb")
# Output parsed logs
action(type = "omfile" template="es-payload"
file="/var/log/rsyslog/rsyslog-cassandra.log" FileCreateMode="0644")
action(
type = "omelasticsearch"
template = "es-payload"
dynSearchIndex = "on"
searchIndex = "logstash-index"
searchType = "cassandra"
server = "myhost"
serverport = "9200"
errorFile = "/var/log/rsyslog/ES-error.log"
bulkmode = "on"
action.resumeretrycount="5" # retry if ES is unreachable (-1 for
infinite retries)
action.resumeInterval="60"
queue.dequeuebatchsize="1000" # ES bulk size
queue.type="FixedArray"
queue.size="100000"
queue.workerthreads="5"
queue.spoolDirectory="/var/spool/rsyslog"
queue.filename="plain"
queue.maxfilesize="100m"
queue.maxdiskspace="1g"
# queue.highwatermark="50000"
# queue.lowwatermark="20000"
queue.saveonshutdown="on"
)
stop
}
Liblognorm configuration in /etc/rsyslog.d/rules/cassandra.log.rb:
# The rule to parse each line in the log file. IMPORTANT: The leading space
is important (see
http://www.rsyslog.com/log-normalization-and-the-leading-space/)
rule=: %log_level:word% %log_time:word% %message:rest%
On Sun, Dec 13, 2015 at 12:01 AM, Ciprian Hacman <
[email protected]> wrote:
I see Alec is using imfile and omelasticsearch so I suppose he is tailing a
log file.
By any chance, are you using Cassandra to capture multiline logs (using the
startmsg.regex setting)? That would explain why the last log line is not
sent until restart.
Ciprian
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Sun, Dec 13, 2015 at 6:45 AM, David Lang <[email protected]> wrote:
On Sat, 12 Dec 2015, Alec Swan wrote:
David, I don't have name= attribute in the action statement. However,
there
are log messages from omelasticsearch module after rsyslog restart (see
my
previous email).
Cassandra does not generate a log of logs and I noticed that the few
that
get generated are not sent to elasticsearch until rsyslog is restarted.
It
feels like logs are being buffered and then sent to ES on subsequent
startup.
So, here is what I see happening:
1. I start Rsyslog
2. I start Cassandra and it generates several dozens of log lines
3. No logs show up in ES (even if I restart Cassandra several times)
4. I restart Rsyslog
5. The logs generated in step 2 show up in ES
This does not sound like rsyslog is buffering the logs, but that there is
some other problem.
how are the logs getting from Cassandra to rsyslog?
can you enable impstats so that you can see how many logs are received,
and how many are processed by the various actions?
it would help to name new style actions so that it's easier to track
them.
in your debug log (the output of -dn), track a known message that
Cassandra generates and see exactly what happens to it.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
