Hello,
I'm trying to get rsyslog parsing out my Apache access logs and store them into
ElasticSearch in a usable format that can work with logstash as well.
First, here's a code snipped of my recipient rsyslog configuration which
details out how it's handling apache access logs already:
rsyslog.conf snippet:
template(name="RemoteHost" type="string"
string="/srv/log/%HOSTNAME%/%$YEAR%/%$MONTH%/syslog-%$DAY%.log")
ruleset(name="remote") {
$IncludeConfig /etc/rsyslog.d/*.remote
action(type="omfile" DynaFile="RemoteHost")
}
input(type="imudp" port="514" ruleset="remote")
input(type="imtcp" port="514" ruleset="remote")
webservers.template: (imported globally):
template(name="httpd-access" type="list") {
property(name="msg" position.from="2" spifno1stsp="off")
property(name="msg" droplastlf="on")
constant(value="\n")
}
template(name="httpd-error" type="list") {
constant(value="<")
property(name="syslogpriority-text")
constant(value="> ")
property(name="timestamp" dateFormat="rfc3339")
constant(value=" ")
property(name="hostname")
constant(value=" ")
property(name="syslogtag" position.from="1" position.to="32")
property(name="msg" spifno1stsp="off")
property(name="msg" droplastlf="on")
constant(value="\n")
}
template(name="WebErrFiles" type="string"
string="/srv/log/WEB/%$YEAR%/%$MONTH%/httpd_error-%$DAY%.log")
template(name="PhpErrFiles" type="string"
string="/srv/log/WEB/%$YEAR%/%$MONTH%/php_error-%$DAY%.log")
template(name="WebFiles" type="string"
string="/srv/log/WEB/%$YEAR%/%$MONTH%/access_%$!vhost%-%$DAY%.log")
logstash.template: (also imported globally):
template(name="logstash"
type="list"
option.json="on") {
constant(value="{")
constant(value="\"@timestamp\":\"")
property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"message\":\"") property(name="msg")
constant(value="\",\"host\":\"") property(name="hostname")
constant(value="\",\"severity\":\"")
property(name="syslogseverity-text")
constant(value="\",\"facility\":\"")
property(name="syslogfacility-text")
constant(value="\",\"syslogtag\":\"") property(name="syslogtag")
constant(value="\"}")
}
template(name="httpd-access-logstash"
type="list"
option.json="on") {
constant(value="{")
#constant(value="\"@timestamp\":\"")
property(name="timereported" dateFormat="rfc3339")
constant(value="\"@timestamp\":\"")
property(name="timereported" dateFormat="rfc3339")
constant(value="\"@fields.bytes\":\"")
property(name="$!bytesend")
constant(value="\"@fields.clientip\":\"") property(name="$!ip")
#constant(value="\"@fields.duration\":\"") property(name="$!...")
constant(value="\"@fields.method\":\"")
property(name="$!method")
constant(value="\"@fields.request\":\"") property(name="$!url")
#constant(value="\"@fields.urlpath\":\"") property(name="$!url")
#constant(value="\"@fields.urlquery\":\"") property(name="...")
constant(value="\"@message\":\"") property(name="msg"
position.from="2" spifno1stsp="off")
constant(value="\"@source_host\":\"")
property(name="hostname")
constant(value="\"@vhost\":\"")
property(name="$!vhost")
constant(value="\"}")
}
apacheaccess.rule: (used in following configuration):
rule=: %vhost:word% %ip:word% %rlogname:word% %ruser:word% [%date:word%
%heure:word% %tz:char-to:]%] "%method:word% %url:word% %pver:char-to:"%"
%status:word% %bytesend:word% %referer:word% %useragent:quoted-string%
%ssl:word% %sslport:word% %sslproto:word%
10-apacheaccess.remote: (as imported from the rsyslog.conf):
if $programname == 'httpd' or $programname == 'php' then {
#if $syslogseverity-text == 'error' then {
if $programname == 'php' then {
action(type="omfile" DynaFile="PhpErrFiles" template="httpd-error")
} else if $syslogfacility-text == 'local1' then {
action(type="omfile" DynaFile="WebErrFiles" template="httpd-error")
#action(type="omfwd" Target="172.17.51.4" Port="514" Protocol="tcp")
} else {
action(type="mmnormalize" userawmsg="off"
rulebase="/etc/rsyslog.d/apacheaccess.rule")
action(type="omfile" DynaFile="WebFiles" template="httpd-access")
action(type="omelasticsearch"
server="localhost"
serverport="9200"
template="httpd-access-logstash"
searchIndex="system"
searchType="apache-access"
bulkmode="on"
queue.type="linkedlist"
queue.size="5000"
queue.dequeuebatchsize="300"
action.resumeretrycount="-1")
#action(type="omfwd" Target="172.17.51.4" Port="514" Protocol="tcp")
}
stop
}
90-logstash.remote:
*.* action(type="omelasticsearch"
server="localhost"
serverport="9200"
template="logstash"
searchIndex="system"
searchType="syslog"
bulkmode="on"
queue.type="linkedlist"
queue.size="5000"
queue.dequeuebatchsize="300"
action.resumeretrycount="-1")
What is working with all of this so far is the fact the rsyslog server receives
Apache access logs from two different webservers in a load-balanced cluster,
and it stores the logs into /srv/log/WEB/$year/$month/access_$!vhost.log
successfully. Also, the catch-all rule in 90 is sending all local and remote
syslog events to elasticsearch and that data is quite usable as-is so far.
The problem is with configuration 10 and its template. I'm trying to extract
the portions of the apache access log and store them directly into the
elasticsearch accordingly. So far, my problem is, I'm not getting /anything/ in
elasticsearch for this right now, and I'm wondering why. I looked all over the
rsyslog website to try to figure out how to use mmnormalize variables in
template property() sections, but I see nothing so I can only guess, and I'm
probably guessing wrong.
Secondly... Logstash uses ISO-8601 date format, which is very specific. I see
on rsyslog's features that ISO-9601 is supported, but I see now way to actually
have it utilize that at all.. My example case is in the following three date
variations, 2 of which I've seen coming into rsyslog which oddly differs
whether it's forwarded by tcp or local:
2013-11-23T13:14:00.818Z ISO-8601 -- desired format
2013-11-23T08:19:01-05:00 RFC-3339 -- from forward
2013-11-23T08:20:01.392547-05:00 RFC-3339 also? -- from log
It is notibly odd that tcp-forwarded syslog events only have date, time,
timezone offset, no microseconds, and no Z. It is also notible that none of
them have the Z, and they all have the timezone offset, which means that are
not ISO-8601, but RFC-3339 specifically, which is a variation of ISO-8601 with
localization to the native timezone, basically.
So, how can I fix all these issues? I would appreciate the help in this.
Eric Renfro
Linux-Help.org System Engineer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
