two quick pointers as I head to dinner: a) RELP requieres a proper syslog header. you can't just simply send the user data to it b) you need to use mmjsonparse on the receiver to re-populate the JSON structure
Note that b) requires a) + a proper CEE header for mmjsonparse to work. HTH at least a little bit;) Rainer On Fri, Feb 21, 2014 at 6:47 PM, Eugene Istomin <[email protected]> wrote: > Another run for our app logging: > > > client have rule for normalizing and action rule for server sending via > relp: > > template(name="NginxFileFormat" type="list") { > constant(value="{") > property(name="$!date" format="jsonf") > constant(value=", ") > property(name="$!x_client" format="jsonf") > constant(value=", ") > property(name="$!request_method" format="jsonf") > constant(value=", ") > property(name="$!status" format="jsonf") > constant(value=", ") > property(name="$!request_uri" format="jsonf") > constant(value=", ") > property(name="$!gzip_ratio" format="jsonf") > constant(value=", ") > property(name="$!request_time" format="jsonf") > constant(value=", ") > property(name="$!body_bytes_sent" format="jsonf") > constant(value=", ") > property(name="$!user_agent" format="jsonf" position.To="8") > constant(value="}\n") > } > > template(name="NginxFileFormatSaver" type="string" > string="/web/_deploy/%$!folder%/logs/nginx/%$month%.%$day%/%$!http_host%") > > if $programname == 'nginx' then > action(type="mmnormalize" useRawMsg="on" > ruleBase="/etc/rsyslog.d/rules/nginx") > & action(type="omrelp" target="10.100.101.250" port="20514" > template="NginxFileFormat") > & action(type="omfile" Template="NginxFileFormat" > DynaFile="NginxFileFormatSaver") > & stop > > > > > Local messaging works ok: > > tailf /web/_deploy/eds.work/logs/nginx/02.21/XXXX > > {"date":"Feb 21 19:37:52", "x_client":"XXX", "request_method":"GET", > "status":"200", "request_uri":"XXXX", "gzip_ratio":"2.23", > "request_time":"0.000", "body_bytes_sent":"292", "user_agent":"Mozilla/"} > > > > On the server side we have following conf: > > ...... > module(load="imrelp" Ruleset="nginx") > input(type="imrelp" Port="20514") > > template (name="nginx" type="list" sql.option="on") { > constant(value="INSERT INTO nginx (http_host, ........... user_agent)") > constant(value=" VALUES ('") > property(name="$!http_host") > constant(value=",") > property(name="$!var_x_forwarded_for") > ....... > property(name="$!var_user_agent") > constant(value="')") > } > > > ruleset(name="nginx") { > action(type="mmjsonparse") > set $!var_http_host = $http_host; > ...... > set $!var_user_agent = $user_agent; > > action(type="ommysql" server="localhost" serverport="3306" > db="rsyslog" uid="rsyslog" pwd="rsyslog" template="nginx") > } > > > > The question is why rsyslog doesn't work like expected (inserting data > into DB), the errror is *INVALID PROPERTY NAME* > > > 4001.894777365:7f578a2b7700: relp engine is dispatching frame with command > 'syslog' > 4001.894782968:7f578a2b7700: in 'syslog' command handler > 4001.894803276:7f578a2b7700: main Q: qqueueAdd: entry added, size now log > 1, phys 1 entries > 4001.894819735:7f578a2b7700: main Q: EnqueueMsg advised worker start > 4001.894842354:7f578a2b7700: in destructor: sendbuf 0x7f577c0293a0 > 4001.894872669:7f578a2b7700: librelp: epoll_set_events sock 11, target > bits 01, current 01 > 4001.894902997:7f5789ab6700: wti 0x1c92ef0: worker awoke from idle > processing > 4001.894929072:7f5789ab6700: DeleteProcessedBatch: we deleted 0 objects > and enqueued 0 objects > 4001.894936009:7f5789ab6700: doDeleteBatch: delete batch from store, new > sizes: log 1, phys 1 > 4001.894949456:7f5789ab6700: msg parser: flags 30, from '10.100.101.97', > msg '{"date":"Feb 21 19:33:21", "x_client":"194.204.32.11", "requ' > 4001.894959428:7f5789ab6700: parse using parser list 0x1c7d790 (the > default list). > 4001.894965890:7f5789ab6700: dropped LF at very end of message > (DropTrailingLF is set) > 4001.894972794:7f578a2b7700: librelp: doing epoll_wait > 4001.894998278:7f5789ab6700: Parser 'rsyslog.rfc5424' returned -2160 > 4001.895012989:7f5789ab6700: Message will now be parsed by the legacy > syslog parser (one size fits all... ;)). > 4001.895022981:7f5789ab6700: Parser 'rsyslog.rfc3164' returned 0 > 4001.895031313:7f5789ab6700: processBATCH: batch of 1 elements must be > processed > 4001.895040008:7f5789ab6700: processBATCH: next msg 0: {"date":"Feb 21 > 19:33:21", "x_client":"194.204.32.11", "request_method":"GET", > "status":"200", "request_uri":"/session/14/52/4c7 > 4001.895052532:7f5789ab6700: ACTION 0 > [mmjsonparse:action(type="mmjsonparse" ...)] > 4001.895066603:7f5789ab6700: executing action 0 > 4001.895071930:7f5789ab6700: Called action, logging to mmjsonparse > 4001.895079697:7f5789ab6700: Action 0 transitioned to state: itx > 4001.895087519:7f5789ab6700: entering actionCalldoAction(), state: itx, > actionNbr 0 > > 4001.895098118:7f5789ab6700: mmjsonparse: no JSON cookie: '"Feb 21 > 19:33:21", "x_client":"XXXXXX", "request_method":"GET", "status":"200", > "request_uri":"XXXXX", "gzip_ratio":"4.55", "request_time":"0.001", > "body_bytes_sent":"495", "user_agent":"Mozilla/"}' > 4001.895115327:7f5789ab6700: Action 0 transitioned to state: rdy > 4001.895120802:7f5789ab6700: SET !var_http_host = > 4001.895132716:7f5789ab6700: var 'http_host' > 4001.895146722:7f5789ab6700: END SET > 4001.895160129:7f5789ab6700: eval expr 0x1c9a200, type 'V[86]' > 4001.895165988:7f5789ab6700: invalid property id: '112' > 4001.895171965:7f5789ab6700: rainerscript: var 112: '***INVALID PROPERTY > NAME***' > 4001.895177271:7f5789ab6700: eval expr 0x1c9a200, return datatype 'S' > 4001.895186019:7f5789ab6700: SET !var_x_forwarded_for = > 4001.895197686:7f5789ab6700: var 'x_forwarded_for' > 4001.895211853:7f5789ab6700: END SET > 4001.895223237:7f5789ab6700: eval expr 0x1c9a1d0, type 'V[86]' > 4001.895228412:7f5789ab6700: invalid property id: '176' > 4001.895233604:7f5789ab6700: rainerscript: var 176: '**INVALID PROPERTY > NAME**' > 4001.895238609:7f5789ab6700: eval expr 0x1c9a1d0, return datatype 'S' > 4001.895245235:7f5789ab6700: SET !var_x_client = > 4001.895256754:7f5789ab6700: var 'x_client' > 4001.895270913:7f5789ab6700: END SET > > > > Can we send real CEE JSON format using RELP? The reason for this - > unnecessary server-side parsing, we already made CEE json on the client > side and just wants to store this key-value mappings in DB. > > > Thanks. > /---/ > */Best regards,/* > /Eugene Istomin/ > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
