I use rsyslog to logstash which backs off to ES and kibana on top. Lovely setup.
Sent from my iPhone > On 22 Feb 2014, at 18:26, Kendall Green <[email protected]> wrote: > > I'm interested but is there java ruby dependencies,? sorry I'm not yet > experienced with elasticsearch. >> On Feb 21, 2014 7:09 PM, "Eugene Istomin" <[email protected]> wrote: >> >> Thanks, >> >> became working after a few changes + CEE-like JSONs. >> >> Right now we are testing RSysLog + Elasticsearch + Kibana3, looking >> awesome and pretty flexible. >> Does anybody using such configuration? >> /---/ >> */Best regards,/* >> /Eugene Istomin/ >> >> >> >>> two quick pointers as I head to dinner: >>> >>> a) RELP requieres a proper syslog header. you can't just simply send the >>> user data to it >>> b) you need to use mmjsonparse on the receiver to re-populate the JSON >>> structure >>> >>> Note that b) requires a) + a proper CEE header for mmjsonparse to >> work. >>> >>> HTH at least a little bit;) >>> >>> Rainer >>> >>> On Fri, Feb 21, 2014 at 6:47 PM, Eugene Istomin <[email protected]> >> wrote: >>>> Another run for our app logging: >>>> >>>> >>>> client have rule for normalizing and action rule for server sending via >>>> relp: >>>> >>>> template(name="NginxFileFormat" type="list") { >>>> >>>> constant(value="{") >>>> property(name="$!date" format="jsonf") >>>> constant(value=", ") >>>> property(name="$!x_client" format="jsonf") >>>> constant(value=", ") >>>> property(name="$!request_method" format="jsonf") >>>> constant(value=", ") >>>> property(name="$!status" format="jsonf") >>>> constant(value=", ") >>>> property(name="$!request_uri" format="jsonf") >>>> constant(value=", ") >>>> property(name="$!gzip_ratio" format="jsonf") >>>> constant(value=", ") >>>> property(name="$!request_time" format="jsonf") >>>> constant(value=", ") >>>> property(name="$!body_bytes_sent" format="jsonf") >>>> constant(value=", ") >>>> property(name="$!user_agent" format="jsonf" position.To="8") >>>> constant(value="}\n") >>>> >>>> } >>>> >>>> template(name="NginxFileFormatSaver" type="string" >>>> string="/web/_deploy/%$!folder%/logs/nginx/%$month%. >> %$day%/%$!http_host%") >>>> >>>> if $programname == 'nginx' then >>>> >>>> action(type="mmnormalize" useRawMsg="on" >>>> >>>> ruleBase="/etc/rsyslog.d/rules/nginx") >>>> >>>> & action(type="omrelp" target="10.100.101.250" port="20514" >>>> >>>> template="NginxFileFormat") >>>> >>>> & action(type="omfile" Template="NginxFileFormat" >>>> >>>> DynaFile="NginxFileFormatSaver") >>>> >>>> & stop >>>> >>>> Local messaging works ok: >>>> >>>> tailf /web/_deploy/eds.work/logs/nginx/02.21/XXXX >>>> >>>> {"date":"Feb 21 19:37:52", "x_client":"XXX", "request_method":"GET", >>>> "status":"200", "request_uri":"XXXX", "gzip_ratio":"2.23", >>>> "request_time":"0.000", "body_bytes_sent":"292", >> "user_agent":"Mozilla/"} >>>> >>>> >>>> >>>> On the server side we have following conf: >>>> >>>> ...... >>>> module(load="imrelp" Ruleset="nginx") >>>> input(type="imrelp" Port="20514") >>>> >>>> template (name="nginx" type="list" sql.option="on") { >>>> constant(value="INSERT INTO nginx (http_host, ........... >>>> user_agent)") >>>> constant(value=" VALUES ('") >>>> property(name="$!http_host") >>>> constant(value=",") >>>> property(name="$!var_x_forwarded_for") >>>> >>>> ....... >>>> >>>> property(name="$!var_user_agent") >>>> constant(value="')") >>>> } >>>> >>>> ruleset(name="nginx") { >>>> >>>> action(type="mmjsonparse") >>>> >>>> set $!var_http_host = $http_host; >>>> >>>> ...... >>>> >>>> set $!var_user_agent = $user_agent; >>>> >>>> action(type="ommysql" server="localhost" serverport="3306" >>>> >>>> db="rsyslog" uid="rsyslog" pwd="rsyslog" template="nginx") >>>> } >>>> >>>> >>>> >>>> The question is why rsyslog doesn't work like expected (inserting data >>>> into DB), the errror is *INVALID PROPERTY NAME* >>>> >>>> >>>> 4001.894777365:7f578a2b7700: relp engine is dispatching frame with >> command >>>> 'syslog' >>>> 4001.894782968:7f578a2b7700: in 'syslog' command handler >>>> 4001.894803276:7f578a2b7700: main Q: qqueueAdd: entry added, >> size now log >>>> 1, phys 1 entries >>>> 4001.894819735:7f578a2b7700: main Q: EnqueueMsg advised worker >> start >>>> 4001.894842354:7f578a2b7700: in destructor: sendbuf >> 0x7f577c0293a0 >>>> 4001.894872669:7f578a2b7700: librelp: epoll_set_events sock 11, >> target >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
