Hmm, I hadn't thought to try !variablename in template properties.. Like this?
template(name="logstash-accesslog"
type="list"
option.json="on") {
constant(value="{")
constant(value="\"@timestamp\":\"")
property(name="timereported" dateFormat="rfc3339")
constant(value="\"@fields.bytes\":\"")
property(name="!bytesend")
constant(value="\"@fields.clientip\":\"") property(name="!ip")
constant(value="\"@fields.method\":\"")
property(name="!method")
constant(value="\"@fields.request\":\"") property(name="!url")
constant(value="\"@message\":\"") property(name="msg"
position.from="2" spifno1stsp="off")
constant(value="\"@source_host\":\"")
property(name="hostname")
constant(value="\"@vhost\":\"") property(name="!vhost")
constant(value="\"host\":\"")
property(name="fromhost-ip")
constant(value="\"}")
}
But, still, so far this doesn't seem to be working at all. I get nothing in my
elasticsearch under the expected document type.
--
Eric Renfro (Psi-Jack)
Linux-Help.org System Engineer
On Sunday, November 24, 2013 01:33:20 PM David Lang wrote:
> I can't help you with the date format, but I think there were some changes in
> a recent 7.5 release related to supporting timezone conversions.
>
> logs received from remote machines have the timestamp provided by the remote
> machine, but the template setting should let you change the format.
>
> as far as accessing the mmnormalize variables, they are $!name, so with your
> rule:
>
> > rule=: %vhost:word% %ip:word% %rlogname:word% %ruser:word% [%date:word%
> > %heure:word% %tz:char-to:]%] "%method:word% %url:word% %pver:char-to:"%"
> > %status:word% %bytesend:word% %referer:word% %useragent:quoted-string%
> > %ssl:word% %sslport:word% %sslproto:word%
>
> you would use $!vhost $!ip $!rlogname etc in a string template, or in a
> template
> where you are using properties explicitly (like http-error) you would use
> !vhost
> !ip !rlogname
>
> David Lang
>
> On Sun, 24 Nov 2013, Eric Renfro wrote:
>
> > Hello,
> >
> > I'm trying to get rsyslog parsing out my Apache access logs and store them
> > into ElasticSearch in a usable format that can work with logstash as well.
> >
> > First, here's a code snipped of my recipient rsyslog configuration which
> > details out how it's handling apache access logs already:
> >
> > rsyslog.conf snippet:
> >
> > template(name="RemoteHost" type="string"
> > string="/srv/log/%HOSTNAME%/%$YEAR%/%$MONTH%/syslog-%$DAY%.log")
> >
> > ruleset(name="remote") {
> > $IncludeConfig /etc/rsyslog.d/*.remote
> >
> > action(type="omfile" DynaFile="RemoteHost")
> > }
> >
> > input(type="imudp" port="514" ruleset="remote")
> > input(type="imtcp" port="514" ruleset="remote")
> >
> >
> >
> > webservers.template: (imported globally):
> >
> > template(name="httpd-access" type="list") {
> > property(name="msg" position.from="2" spifno1stsp="off")
> > property(name="msg" droplastlf="on")
> > constant(value="\n")
> > }
> > template(name="httpd-error" type="list") {
> > constant(value="<")
> > property(name="syslogpriority-text")
> > constant(value="> ")
> > property(name="timestamp" dateFormat="rfc3339")
> > constant(value=" ")
> > property(name="hostname")
> > constant(value=" ")
> > property(name="syslogtag" position.from="1" position.to="32")
> > property(name="msg" spifno1stsp="off")
> > property(name="msg" droplastlf="on")
> > constant(value="\n")
> > }
> >
> > template(name="WebErrFiles" type="string"
> > string="/srv/log/WEB/%$YEAR%/%$MONTH%/httpd_error-%$DAY%.log")
> > template(name="PhpErrFiles" type="string"
> > string="/srv/log/WEB/%$YEAR%/%$MONTH%/php_error-%$DAY%.log")
> > template(name="WebFiles" type="string"
> > string="/srv/log/WEB/%$YEAR%/%$MONTH%/access_%$!vhost%-%$DAY%.log")
> >
> >
> >
> > logstash.template: (also imported globally):
> >
> > template(name="logstash"
> > type="list"
> > option.json="on") {
> > constant(value="{")
> > constant(value="\"@timestamp\":\"")
> > property(name="timereported" dateFormat="rfc3339")
> > constant(value="\",\"message\":\"") property(name="msg")
> > constant(value="\",\"host\":\"")
> > property(name="hostname")
> > constant(value="\",\"severity\":\"")
> > property(name="syslogseverity-text")
> > constant(value="\",\"facility\":\"")
> > property(name="syslogfacility-text")
> > constant(value="\",\"syslogtag\":\"")
> > property(name="syslogtag")
> > constant(value="\"}")
> > }
> >
> > template(name="httpd-access-logstash"
> > type="list"
> > option.json="on") {
> > constant(value="{")
> > #constant(value="\"@timestamp\":\"")
> > property(name="timereported" dateFormat="rfc3339")
> > constant(value="\"@timestamp\":\"")
> > property(name="timereported" dateFormat="rfc3339")
> > constant(value="\"@fields.bytes\":\"")
> > property(name="$!bytesend")
> > constant(value="\"@fields.clientip\":\"")
> > property(name="$!ip")
> > #constant(value="\"@fields.duration\":\"")
> > property(name="$!...")
> > constant(value="\"@fields.method\":\"")
> > property(name="$!method")
> > constant(value="\"@fields.request\":\"")
> > property(name="$!url")
> > #constant(value="\"@fields.urlpath\":\"")
> > property(name="$!url")
> > #constant(value="\"@fields.urlquery\":\"")
> > property(name="...")
> > constant(value="\"@message\":\"") property(name="msg"
> > position.from="2" spifno1stsp="off")
> > constant(value="\"@source_host\":\"")
> > property(name="hostname")
> > constant(value="\"@vhost\":\"")
> > property(name="$!vhost")
> > constant(value="\"}")
> > }
> >
> >
> >
> > apacheaccess.rule: (used in following configuration):
> >
> > rule=: %vhost:word% %ip:word% %rlogname:word% %ruser:word% [%date:word%
> > %heure:word% %tz:char-to:]%] "%method:word% %url:word% %pver:char-to:"%"
> > %status:word% %bytesend:word% %referer:word% %useragent:quoted-string%
> > %ssl:word% %sslport:word% %sslproto:word%
> >
> >
> >
> > 10-apacheaccess.remote: (as imported from the rsyslog.conf):
> >
> > if $programname == 'httpd' or $programname == 'php' then {
> > #if $syslogseverity-text == 'error' then {
> > if $programname == 'php' then {
> > action(type="omfile" DynaFile="PhpErrFiles" template="httpd-error")
> > } else if $syslogfacility-text == 'local1' then {
> > action(type="omfile" DynaFile="WebErrFiles" template="httpd-error")
> > #action(type="omfwd" Target="172.17.51.4" Port="514" Protocol="tcp")
> > } else {
> > action(type="mmnormalize" userawmsg="off"
> > rulebase="/etc/rsyslog.d/apacheaccess.rule")
> > action(type="omfile" DynaFile="WebFiles" template="httpd-access")
> > action(type="omelasticsearch"
> > server="localhost"
> > serverport="9200"
> > template="httpd-access-logstash"
> > searchIndex="system"
> > searchType="apache-access"
> > bulkmode="on"
> > queue.type="linkedlist"
> > queue.size="5000"
> > queue.dequeuebatchsize="300"
> > action.resumeretrycount="-1")
> > #action(type="omfwd" Target="172.17.51.4" Port="514" Protocol="tcp")
> > }
> > stop
> > }
> >
> >
> >
> > 90-logstash.remote:
> > *.* action(type="omelasticsearch"
> > server="localhost"
> > serverport="9200"
> > template="logstash"
> > searchIndex="system"
> > searchType="syslog"
> > bulkmode="on"
> > queue.type="linkedlist"
> > queue.size="5000"
> > queue.dequeuebatchsize="300"
> > action.resumeretrycount="-1")
> >
> >
> >
> > What is working with all of this so far is the fact the rsyslog server
> > receives Apache access logs from two different webservers in a
> > load-balanced cluster, and it stores the logs into
> > /srv/log/WEB/$year/$month/access_$!vhost.log successfully. Also, the
> > catch-all rule in 90 is sending all local and remote syslog events to
> > elasticsearch and that data is quite usable as-is so far.
> >
> > The problem is with configuration 10 and its template. I'm trying to
> > extract the portions of the apache access log and store them directly into
> > the elasticsearch accordingly. So far, my problem is, I'm not getting
> > /anything/ in elasticsearch for this right now, and I'm wondering why. I
> > looked all over the rsyslog website to try to figure out how to use
> > mmnormalize variables in template property() sections, but I see nothing so
> > I can only guess, and I'm probably guessing wrong.
> >
> >
> > Secondly... Logstash uses ISO-8601 date format, which is very specific. I
> > see on rsyslog's features that ISO-9601 is supported, but I see now way to
> > actually have it utilize that at all.. My example case is in the following
> > three date variations, 2 of which I've seen coming into rsyslog which oddly
> > differs whether it's forwarded by tcp or local:
> >
> > 2013-11-23T13:14:00.818Z ISO-8601 -- desired format
> > 2013-11-23T08:19:01-05:00 RFC-3339 -- from forward
> > 2013-11-23T08:20:01.392547-05:00 RFC-3339 also? -- from log
> >
> > It is notibly odd that tcp-forwarded syslog events only have date, time,
> > timezone offset, no microseconds, and no Z. It is also notible that none of
> > them have the Z, and they all have the timezone offset, which means that
> > are not ISO-8601, but RFC-3339 specifically, which is a variation of
> > ISO-8601 with localization to the native timezone, basically.
> >
> >
> > So, how can I fix all these issues? I would appreciate the help in this.
> >
> > Eric Renfro
> > Linux-Help.org System Engineer
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> > LIKE THAT.
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
