Hi Eric, Just some comments regarding timestamp: RFC 3339 and ISO 8601 are pretty much the same thing: http://stackoverflow.com/questions/522251/whats-the-difference-between-iso-8601-and-rfc-3339-date-formats
The idea is that with both formats, you can have a timezone information (-05:00, for example), or it can be UTC by specifying that "Z". Sub-second information may or may not be available, as well as the whole time part. So all the examples you wrote in your mail are actually compatible with both formats. In Elasticsearch, timestamps are stored as "long", which records the number of miliseconds since Epoch (and Epoch is UTC). It does the conversion from your date string to this long value automatically. So, as long as your timestamp is correct, when you sort your logs by timestamp, they should be sorted correctly. For example, 2013-11-23T13:14:00Z and 2013-11-23T13:19:00-05:00 are equivalent. In production, I'm also indexing logs coming from different sources. Each provides TZ info in its own way (rsyslog even does something like 2013-11-23T13:14:00+00:00 instead of 2013-11-23T13:14:00Z), and I just don't care. When I search, I provide a correct timestamp interval and ES does the timezone conversion for me. I hope this helps solving the timestamp issues (basically, I don't see any problem). As for logs not ending up in ES, it might also help by looking at the ES logs. Does it give any error? Best regards, Radu 2013/11/24 Eric Renfro <[email protected]>: > Hello, > > I'm trying to get rsyslog parsing out my Apache access logs and store them > into ElasticSearch in a usable format that can work with logstash as well. > > First, here's a code snipped of my recipient rsyslog configuration which > details out how it's handling apache access logs already: > > rsyslog.conf snippet: > > template(name="RemoteHost" type="string" > string="/srv/log/%HOSTNAME%/%$YEAR%/%$MONTH%/syslog-%$DAY%.log") > > ruleset(name="remote") { > $IncludeConfig /etc/rsyslog.d/*.remote > > action(type="omfile" DynaFile="RemoteHost") > } > > input(type="imudp" port="514" ruleset="remote") > input(type="imtcp" port="514" ruleset="remote") > > > > webservers.template: (imported globally): > > template(name="httpd-access" type="list") { > property(name="msg" position.from="2" spifno1stsp="off") > property(name="msg" droplastlf="on") > constant(value="\n") > } > template(name="httpd-error" type="list") { > constant(value="<") > property(name="syslogpriority-text") > constant(value="> ") > property(name="timestamp" dateFormat="rfc3339") > constant(value=" ") > property(name="hostname") > constant(value=" ") > property(name="syslogtag" position.from="1" position.to="32") > property(name="msg" spifno1stsp="off") > property(name="msg" droplastlf="on") > constant(value="\n") > } > > template(name="WebErrFiles" type="string" > string="/srv/log/WEB/%$YEAR%/%$MONTH%/httpd_error-%$DAY%.log") > template(name="PhpErrFiles" type="string" > string="/srv/log/WEB/%$YEAR%/%$MONTH%/php_error-%$DAY%.log") > template(name="WebFiles" type="string" > string="/srv/log/WEB/%$YEAR%/%$MONTH%/access_%$!vhost%-%$DAY%.log") > > > > logstash.template: (also imported globally): > > template(name="logstash" > type="list" > option.json="on") { > constant(value="{") > constant(value="\"@timestamp\":\"") > property(name="timereported" dateFormat="rfc3339") > constant(value="\",\"message\":\"") property(name="msg") > constant(value="\",\"host\":\"") property(name="hostname") > constant(value="\",\"severity\":\"") > property(name="syslogseverity-text") > constant(value="\",\"facility\":\"") > property(name="syslogfacility-text") > constant(value="\",\"syslogtag\":\"") > property(name="syslogtag") > constant(value="\"}") > } > > template(name="httpd-access-logstash" > type="list" > option.json="on") { > constant(value="{") > #constant(value="\"@timestamp\":\"") > property(name="timereported" dateFormat="rfc3339") > constant(value="\"@timestamp\":\"") > property(name="timereported" dateFormat="rfc3339") > constant(value="\"@fields.bytes\":\"") > property(name="$!bytesend") > constant(value="\"@fields.clientip\":\"") property(name="$!ip") > #constant(value="\"@fields.duration\":\"") > property(name="$!...") > constant(value="\"@fields.method\":\"") > property(name="$!method") > constant(value="\"@fields.request\":\"") > property(name="$!url") > #constant(value="\"@fields.urlpath\":\"") > property(name="$!url") > #constant(value="\"@fields.urlquery\":\"") property(name="...") > constant(value="\"@message\":\"") property(name="msg" > position.from="2" spifno1stsp="off") > constant(value="\"@source_host\":\"") > property(name="hostname") > constant(value="\"@vhost\":\"") > property(name="$!vhost") > constant(value="\"}") > } > > > > apacheaccess.rule: (used in following configuration): > > rule=: %vhost:word% %ip:word% %rlogname:word% %ruser:word% [%date:word% > %heure:word% %tz:char-to:]%] "%method:word% %url:word% %pver:char-to:"%" > %status:word% %bytesend:word% %referer:word% %useragent:quoted-string% > %ssl:word% %sslport:word% %sslproto:word% > > > > 10-apacheaccess.remote: (as imported from the rsyslog.conf): > > if $programname == 'httpd' or $programname == 'php' then { > #if $syslogseverity-text == 'error' then { > if $programname == 'php' then { > action(type="omfile" DynaFile="PhpErrFiles" template="httpd-error") > } else if $syslogfacility-text == 'local1' then { > action(type="omfile" DynaFile="WebErrFiles" template="httpd-error") > #action(type="omfwd" Target="172.17.51.4" Port="514" Protocol="tcp") > } else { > action(type="mmnormalize" userawmsg="off" > rulebase="/etc/rsyslog.d/apacheaccess.rule") > action(type="omfile" DynaFile="WebFiles" template="httpd-access") > action(type="omelasticsearch" > server="localhost" > serverport="9200" > template="httpd-access-logstash" > searchIndex="system" > searchType="apache-access" > bulkmode="on" > queue.type="linkedlist" > queue.size="5000" > queue.dequeuebatchsize="300" > action.resumeretrycount="-1") > #action(type="omfwd" Target="172.17.51.4" Port="514" Protocol="tcp") > } > stop > } > > > > 90-logstash.remote: > *.* action(type="omelasticsearch" > server="localhost" > serverport="9200" > template="logstash" > searchIndex="system" > searchType="syslog" > bulkmode="on" > queue.type="linkedlist" > queue.size="5000" > queue.dequeuebatchsize="300" > action.resumeretrycount="-1") > > > > What is working with all of this so far is the fact the rsyslog server > receives Apache access logs from two different webservers in a load-balanced > cluster, and it stores the logs into > /srv/log/WEB/$year/$month/access_$!vhost.log successfully. Also, the > catch-all rule in 90 is sending all local and remote syslog events to > elasticsearch and that data is quite usable as-is so far. > > The problem is with configuration 10 and its template. I'm trying to extract > the portions of the apache access log and store them directly into the > elasticsearch accordingly. So far, my problem is, I'm not getting /anything/ > in elasticsearch for this right now, and I'm wondering why. I looked all over > the rsyslog website to try to figure out how to use mmnormalize variables in > template property() sections, but I see nothing so I can only guess, and I'm > probably guessing wrong. > > > Secondly... Logstash uses ISO-8601 date format, which is very specific. I see > on rsyslog's features that ISO-9601 is supported, but I see now way to > actually have it utilize that at all.. My example case is in the following > three date variations, 2 of which I've seen coming into rsyslog which oddly > differs whether it's forwarded by tcp or local: > > 2013-11-23T13:14:00.818Z ISO-8601 -- desired format > 2013-11-23T08:19:01-05:00 RFC-3339 -- from forward > 2013-11-23T08:20:01.392547-05:00 RFC-3339 also? -- from log > > It is notibly odd that tcp-forwarded syslog events only have date, time, > timezone offset, no microseconds, and no Z. It is also notible that none of > them have the Z, and they all have the timezone offset, which means that are > not ISO-8601, but RFC-3339 specifically, which is a variation of ISO-8601 > with localization to the native timezone, basically. > > > So, how can I fix all these issues? I would appreciate the help in this. > > Eric Renfro > Linux-Help.org System Engineer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
