Another run for our app logging:
client have rule for normalizing and action rule for server sending via relp:
template(name="NginxFileFormat" type="list") {
constant(value="{")
property(name="$!date" format="jsonf")
constant(value=", ")
property(name="$!x_client" format="jsonf")
constant(value=", ")
property(name="$!request_method" format="jsonf")
constant(value=", ")
property(name="$!status" format="jsonf")
constant(value=", ")
property(name="$!request_uri" format="jsonf")
constant(value=", ")
property(name="$!gzip_ratio" format="jsonf")
constant(value=", ")
property(name="$!request_time" format="jsonf")
constant(value=", ")
property(name="$!body_bytes_sent" format="jsonf")
constant(value=", ")
property(name="$!user_agent" format="jsonf" position.To="8")
constant(value="}\n")
}
template(name="NginxFileFormatSaver" type="string"
string="/web/_deploy/%$!folder%/logs/nginx/%$month%.%$day%/%$!http_host%")
if $programname == 'nginx' then
action(type="mmnormalize" useRawMsg="on"
ruleBase="/etc/rsyslog.d/rules/nginx")
& action(type="omrelp" target="10.100.101.250" port="20514"
template="NginxFileFormat")
& action(type="omfile" Template="NginxFileFormat"
DynaFile="NginxFileFormatSaver")
& stop
Local messaging works ok:
tailf /web/_deploy/eds.work/logs/nginx/02.21/XXXX
{"date":"Feb 21 19:37:52", "x_client":"XXX", "request_method":"GET",
"status":"200", "request_uri":"XXXX", "gzip_ratio":"2.23",
"request_time":"0.000", "body_bytes_sent":"292", "user_agent":"Mozilla/"}
On the server side we have following conf:
......
module(load="imrelp" Ruleset="nginx")
input(type="imrelp" Port="20514")
template (name="nginx" type="list" sql.option="on") {
constant(value="INSERT INTO nginx (http_host, ........... user_agent)")
constant(value=" VALUES ('")
property(name="$!http_host")
constant(value=",")
property(name="$!var_x_forwarded_for")
.......
property(name="$!var_user_agent")
constant(value="')")
}
ruleset(name="nginx") {
action(type="mmjsonparse")
set $!var_http_host = $http_host;
......
set $!var_user_agent = $user_agent;
action(type="ommysql" server="localhost" serverport="3306" db="rsyslog"
uid="rsyslog" pwd="rsyslog" template="nginx")
}
The question is why rsyslog doesn't work like expected (inserting data into
DB), the errror is *INVALID PROPERTY NAME*
4001.894777365:7f578a2b7700: relp engine is dispatching frame with command
'syslog'
4001.894782968:7f578a2b7700: in 'syslog' command handler
4001.894803276:7f578a2b7700: main Q: qqueueAdd: entry added, size now log 1,
phys 1 entries
4001.894819735:7f578a2b7700: main Q: EnqueueMsg advised worker start
4001.894842354:7f578a2b7700: in destructor: sendbuf 0x7f577c0293a0
4001.894872669:7f578a2b7700: librelp: epoll_set_events sock 11, target bits 01,
current 01
4001.894902997:7f5789ab6700: wti 0x1c92ef0: worker awoke from idle processing
4001.894929072:7f5789ab6700: DeleteProcessedBatch: we deleted 0 objects and
enqueued 0 objects
4001.894936009:7f5789ab6700: doDeleteBatch: delete batch from store, new sizes:
log 1, phys 1
4001.894949456:7f5789ab6700: msg parser: flags 30, from '10.100.101.97', msg
'{"date":"Feb 21 19:33:21", "x_client":"194.204.32.11", "requ'
4001.894959428:7f5789ab6700: parse using parser list 0x1c7d790 (the default
list).
4001.894965890:7f5789ab6700: dropped LF at very end of message (DropTrailingLF
is set)
4001.894972794:7f578a2b7700: librelp: doing epoll_wait
4001.894998278:7f5789ab6700: Parser 'rsyslog.rfc5424' returned -2160
4001.895012989:7f5789ab6700: Message will now be parsed by the legacy syslog
parser (one size fits all... ;)).
4001.895022981:7f5789ab6700: Parser 'rsyslog.rfc3164' returned 0
4001.895031313:7f5789ab6700: processBATCH: batch of 1 elements must be processed
4001.895040008:7f5789ab6700: processBATCH: next msg 0: {"date":"Feb 21
19:33:21", "x_client":"194.204.32.11", "request_method":"GET", "status":"200",
"request_uri":"/session/14/52/4c7
4001.895052532:7f5789ab6700: ACTION 0
[mmjsonparse:action(type="mmjsonparse" ...)]
4001.895066603:7f5789ab6700: executing action 0
4001.895071930:7f5789ab6700: Called action, logging to mmjsonparse
4001.895079697:7f5789ab6700: Action 0 transitioned to state: itx
4001.895087519:7f5789ab6700: entering actionCalldoAction(), state: itx,
actionNbr 0
4001.895098118:7f5789ab6700: mmjsonparse: no JSON cookie: '"Feb 21 19:33:21",
"x_client":"XXXXXX", "request_method":"GET", "status":"200",
"request_uri":"XXXXX", "gzip_ratio":"4.55", "request_time":"0.001",
"body_bytes_sent":"495", "user_agent":"Mozilla/"}'
4001.895115327:7f5789ab6700: Action 0 transitioned to state: rdy
4001.895120802:7f5789ab6700: SET !var_http_host =
4001.895132716:7f5789ab6700: var 'http_host'
4001.895146722:7f5789ab6700: END SET
4001.895160129:7f5789ab6700: eval expr 0x1c9a200, type 'V[86]'
4001.895165988:7f5789ab6700: invalid property id: '112'
4001.895171965:7f5789ab6700: rainerscript: var 112: '***INVALID PROPERTY
NAME***'
4001.895177271:7f5789ab6700: eval expr 0x1c9a200, return datatype 'S'
4001.895186019:7f5789ab6700: SET !var_x_forwarded_for =
4001.895197686:7f5789ab6700: var 'x_forwarded_for'
4001.895211853:7f5789ab6700: END SET
4001.895223237:7f5789ab6700: eval expr 0x1c9a1d0, type 'V[86]'
4001.895228412:7f5789ab6700: invalid property id: '176'
4001.895233604:7f5789ab6700: rainerscript: var 176: '**INVALID PROPERTY NAME**'
4001.895238609:7f5789ab6700: eval expr 0x1c9a1d0, return datatype 'S'
4001.895245235:7f5789ab6700: SET !var_x_client =
4001.895256754:7f5789ab6700: var 'x_client'
4001.895270913:7f5789ab6700: END SET
Can we send real CEE JSON format using RELP? The reason for this - unnecessary
server-side parsing, we already made CEE json on the client side and just wants
to store this key-value mappings in DB.
Thanks.
/---/
*/Best regards,/*
/Eugene Istomin/
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
