Thank you for all the help I got much further now, it seems it was indeed that I was supposed to use $programname instead of the $Tag, now it looks like it at-least connects to my elasticsearch according to the debug information: 7794.801098530:7f38231c4700: omelasticsearch: beginTransaction 7794.801103695:7f38231c4700: Action 0x7f382d70c650 transitioned to state: itx 7794.801108636:7f38231c4700: entering actionCalldoAction(), state: itx 7794.801171315:7f38231c4700: omelasticsearch: using REST URL: 'http://loghost.ulyaoth.net:9200/logstash-2014.10.07/events?' 7794.801772371:7f38231c4700: omelasticsearch: pData replyLen = '108' 7794.801782872:7f38231c4700: omelasticsearch: pData reply: '{"_index":"logstash-2014.10.07","_type":"events","_id":"BHzZDcXTTjqim0mLsglobA","_version":1,"created":true}' 7794.801792573:7f38231c4700: omelasticsearch: no local error logger defined - ignoring ES error information 7794.801797633:7f38231c4700: omelasticsearch: result doAction: 0 (bulkmode 0) 7794.801801664:7f38231c4700: Action 0x7f382d70c650 transitioned to state: rdy 7794.801805504:7f38231c4700: action 0x7f382d70c650 call returned 0 I cannot find it yet in Kibana somehow but I think I will be able to figure that out :). -----Original Message----- From: "Radu Gheorghe"<[email protected]> To: "rsyslog-users"<[email protected]>; Cc: Sent: 2014-10-07 (Tue) 22:15:31 Subject: Re: [rsyslog] json files directly to ES
Two more points from me that will hopefully help: - if you're not sure where something breaks, try to isolate the problem by reducing the config to the bare minimum and building up on it once it works. For example, I wouldn't bother with rulesets if no logs can get to ES in the first place. Just make sure you get your messages in and through to ES. If a minimal config fails, usually running rsyslog -dn like David advised should reveal the issue (for example, give you any exceptions ES generates or libcurl errors) - if you're sure logs are already JSON, I wouldn't bother parsing them. I would just use templates to use the JSON as it is and eventually enrich it with new properties. Take a look here (scroll down to the last section) for an example: http://wiki.rsyslog.com/index.php/Queues_on_v6_with_omelasticsearch It's outdated (uses old config format for most snippets) but it should give you some clues. Best regards, Radu -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Tue, Oct 7, 2014 at 3:33 PM, David Lang <david>@lang.hm> wrote: > you aren't showing us the entire config, so some of this is guesswork. > > try logging the data with the format RSYSLOG_DebugFormat to see what > values are in each variable, the most common problem for things going to > the wrong place is that the variables you are testing don't have the value > you expect. > > I think you need to test $programname not $Tag (Tag is what you set in the > infile input, but it gets put in the $programname variable) > > Beyond that, I would look at the eleasticsearch logs to see if it's > complaining when you try to deliver the log. > > You can also start rsyslog in debug mode (-dn) to see all the gory details > of what it's doing, you should see it attempting to deliver the log and any > error it gets back. > > David Lang > > > On Tue, 7 Oct 2014, SjirBagmeijer wrote: > > Hello, >> >> I have a small question I am trying to make rsyslog to sent log files >> that are already formatted in json directly into my elasticsearch but I >> have some trouble to get this to work is there someone that could perhaps >> see where I am going wrong with my config? >> >> Example of a log file content: >> { "@timestamp": "2014-10-02T13:55:31+02:00", "message": "127.0.0.1 - - >> [02/Oct/2014:13:55:31 +0200] \"GET /_status HTTP/1.1\" 401 38 \"-\" >> \"curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 >> zlib/1.2.3 libidn/1.18 libssh2/1.4.2\"", "tags": ["nginx"], "clientip": >> "127.0.0.1", "remote_user": "-", "contenttype": "text/html; charset=utf-8", >> "bytes": 38, "duration": "0.012", "status": "401", "request": "GET /_status >> HTTP/1.1", "method": "GET", "referrer": "-", "useragent": "curl/7.19.7 >> (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18 >> libssh2/1.4.2" } >> >> Here below some examples that I tried to make by combining examples from >> you website: >> http://www.rsyslog.com/tag/guides-for-rsyslog/ >> http://www.rsyslog.com/tag/elasticsearch/ >> http://www.rsyslog.com/doc/rsyslog_conf_filter.html >> >> I tried the following: >> module(load="imfile" PollingInterval="10") >> module(load="omelasticsearch") >> >> template(name="logstash-index" >> type="list") { >> constant(value="logstash-") >> property(name="timereported" dateFormat="rfc3339" position.from="1" >> position.to="4") >> constant(value=".") >> property(name="timereported" dateFormat="rfc3339" position.from="6" >> position.to="7") >> constant(value=".") >> property(name="timereported" dateFormat="rfc3339" position.from="9" >> position.to="10") >> } >> >> template(name="jsonULY" type="list") { >> property(name="$!all-json") >> } >> >> ruleset(name="logstash"){ >> action(type="omelasticsearch" >> server="loghost.ulyaoth.net" >> serverport="9200" >> searchIndex="logstash-index" >> dynSearchIndex="on" >> template="jsonULY") >> stop >> } >> >> >> input(type="imfile" >> File="/var/log/nginx/access.json" >> Tag="accessnginx" >> StateFile="/var/spool/rsyslog/accessnginx" >> ruleset="logstash") >> >> This seems to do nothing at all somehow, then I also tried the following: >> template(name="jsonULY" type="list") { >> property(name="$!all-json") >> } >> >> input(type="imfile" >> File="/var/log/nginx/access.json" >> Tag="nginxaccess" >> StateFile="/var/spool/rsyslog/nginxaccess") >> >> if $Tag == 'nginxaccess' then { >> >> action(type="omelasticsearch" >> server="logstash.ulyaoth.net" >> serverport="9200" >> searchIndex="logstash-index" >> dynSearchIndex="on" >> template="jsonULY") >> stop >> } >> >> And multiple other ways, it seems the input works but most of the time it >> is going directly to my /var/log/messages instead of going to my ES. I also >> double checked that my ports are open. >> >> Is there someone that or can see what I do wrong and give me a hint, or >> perhaps a link to some example where people sent already existing json >> files directly to ES with rsyslog? >> >> Thank you so much in advanced. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
