you aren't showing us the entire config, so some of this is guesswork.
try logging the data with the format RSYSLOG_DebugFormat to see what values are
in each variable, the most common problem for things going to the wrong place is
that the variables you are testing don't have the value you expect.
I think you need to test $programname not $Tag (Tag is what you set in the
infile input, but it gets put in the $programname variable)
Beyond that, I would look at the eleasticsearch logs to see if it's complaining
when you try to deliver the log.
You can also start rsyslog in debug mode (-dn) to see all the gory details of
what it's doing, you should see it attempting to deliver the log and any error
it gets back.
David Lang
On Tue, 7 Oct 2014, SjirBagmeijer wrote:
Hello,
I have a small question I am trying to make rsyslog to sent log files that are
already formatted in json directly into my elasticsearch but I have some
trouble to get this to work is there someone that could perhaps see where I am
going wrong with my config?
Example of a log file content:
{ "@timestamp": "2014-10-02T13:55:31+02:00", "message": "127.0.0.1 - - [02/Oct/2014:13:55:31 +0200] \"GET /_status HTTP/1.1\" 401 38 \"-\" \"curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18 libssh2/1.4.2\"", "tags": ["nginx"],
"clientip": "127.0.0.1", "remote_user": "-", "contenttype": "text/html; charset=utf-8", "bytes": 38, "duration": "0.012", "status": "401", "request": "GET /_status HTTP/1.1", "method": "GET",
"referrer": "-", "useragent": "curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18 libssh2/1.4.2" }
Here below some examples that I tried to make by combining examples from you
website:
http://www.rsyslog.com/tag/guides-for-rsyslog/
http://www.rsyslog.com/tag/elasticsearch/
http://www.rsyslog.com/doc/rsyslog_conf_filter.html
I tried the following:
module(load="imfile" PollingInterval="10")
module(load="omelasticsearch")
template(name="logstash-index"
type="list") {
constant(value="logstash-")
property(name="timereported" dateFormat="rfc3339" position.from="1"
position.to="4")
constant(value=".")
property(name="timereported" dateFormat="rfc3339" position.from="6"
position.to="7")
constant(value=".")
property(name="timereported" dateFormat="rfc3339" position.from="9"
position.to="10")
}
template(name="jsonULY" type="list") {
property(name="$!all-json")
}
ruleset(name="logstash"){
action(type="omelasticsearch"
server="loghost.ulyaoth.net"
serverport="9200"
searchIndex="logstash-index"
dynSearchIndex="on"
template="jsonULY")
stop
}
input(type="imfile"
File="/var/log/nginx/access.json"
Tag="accessnginx"
StateFile="/var/spool/rsyslog/accessnginx"
ruleset="logstash")
This seems to do nothing at all somehow, then I also tried the following:
template(name="jsonULY" type="list") {
property(name="$!all-json")
}
input(type="imfile"
File="/var/log/nginx/access.json"
Tag="nginxaccess"
StateFile="/var/spool/rsyslog/nginxaccess")
if $Tag == 'nginxaccess' then {
action(type="omelasticsearch"
server="logstash.ulyaoth.net"
serverport="9200"
searchIndex="logstash-index"
dynSearchIndex="on"
template="jsonULY")
stop
}
And multiple other ways, it seems the input works but most of the time it is
going directly to my /var/log/messages instead of going to my ES. I also double
checked that my ports are open.
Is there someone that or can see what I do wrong and give me a hint, or perhaps
a link to some example where people sent already existing json files directly
to ES with rsyslog?
Thank you so much in advanced.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
