Date: Mon, 8 Jun 2015 20:34:45 +0000
From: "David Boles (dboles)" <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Unable to use foreach
David,
Sorry for the delay in responding - my focus was elsewhere temporarily. What
you say about $! and new content below makes sense. I've tried an experiment
with the RSYSLOG_DebugFormat mechanism by running with the following
/etc/rsyslog.conf tweaked in three ways:
--------
module(load="imuxsock")
module(load="mmexternal")
module(load="mmjsonparse")
module(load="ommongodb")
template(name="yyd-log-info" type="list") {
property(name="timestamp")
constant(value=" ")
property(name="hostname")
constant(value=" ")
property(name="msg")
constant(value="\n")
}
# 1
*.* /var/log/debug_pre.log;RSYSLOG_DebugFormat
if ($fromhost-ip == '127.0.0.1' and $syslogfacility-text != 'kern') then {
action(type="mmjsonparse")
if $parsesuccess == "OK" then {
# 2
action(type="mmexternal"
binary="/tmp/snowflake/rewriter.py"
interface.input="fulljson" )
action(type="omfile"
template="yyd-log-info"
file="/var/log/yyd_anon.log")
}
}
# 3
#*.* /var/log/debug_post.log;RSYSLOG_DebugFormat
--------
(1) One mode was with the mmexternal action after # 2 commented out and the
DebugFormat action at # 3, yielding:
--------
Debug line with all properties:
FROMHOST: 'dboles-victim-0', fromhost-ip: '127.0.0.1', HOSTNAME:
'dboles-victim-0', PRI: 12,
syslogtag ':', programname: '', APP-NAME: '', PROCID: '-', MSGID: '-',
TIMESTAMP: 'Jun 8 15:37:24', STRUCTURED-DATA: '-',
msg: ' @cee:{"msg":"[umberlog test] testing ... 1, 2,
3\n","count":"2","el.0.type":"ladybug","el.0.name":"natalie","el.0.spots":"12","el.1.type":"ant","el.1.name":"B289","el.1.role":"worker","el.1.age":"9","pid":"32294","facility":"user","priority":"warn","uid":"1002","gid":"1002","host":"dboles-victim-0","program":"","timestamp":"2015-06-08T15:37:24.449217401-0400"}'
escaped msg: ' @cee:{"msg":"[umberlog test] testing ... 1, 2,
3\n","count":"2","el.0.type":"ladybug","el.0.name":"natalie","el.0.spots":"12","el.1.type":"ant","el.1.name":"B289","el.1.role":"worker","el.1.age":"9","pid":"32294","facility":"user","priority":"warn","uid":"1002","gid":"1002","host":"dboles-victim-0","program":"","timestamp":"2015-06-08T15:37:24.449217401-0400"}'
inputname: imuxsock rawmsg: '<12>Jun 8 15:37:24 : @cee:{"msg":"[umberlog test] testing ... 1, 2,
3\n","count":"2","el.0.type":"ladybug","el.0.name":"natalie","el.0.spots":"12","el.1.type":"ant","el.1.name":"B289","el.1.role":"worker","el.1.age":"9","pid":"32294","facility":"user","priority":"warn","uid":"1002","gid":"1002","host":"dboles-victim-0","program":"","timestamp":"2015-06-08T15:37:24.449217401-0400"}'
$!:{ "msg": "[umberlog test] testing ... 1, 2, 3\n", "count": "2", "el.0.type": "ladybug", "el.0.name": "natalie", "el.0.spots": "12", "el.1.type": "ant", "el.1.name": "B289", "el.1.role": "worker", "el.1.age": "9", "pid":
"32294", "facility": "user", "priority": "warn", "uid": "1002", "gid": "1002", "host": "dboles-victim-0", "program": "", "timestamp": "2015-06-08T15:37:24.449217401-0400" }
$.:
$/:
--------
This makes sense - the mmjsonparse populates the $! element.
(2) The second mode was with the mmexternal action at # 2 enabled and the
DebugFormat action at # 3 - this results in rsyslogd suffering a segfault.
(3) The third mode has the mmexternal action at # 2 enabled and the DebugFormat
action at #1 enabled:
Debug line with all properties:
FROMHOST: 'dboles-victim-0', fromhost-ip: '127.0.0.1', HOSTNAME:
'dboles-victim-0', PRI: 12,
syslogtag ':', programname: '', APP-NAME: '', PROCID: '-', MSGID: '-',
TIMESTAMP: 'Jun 8 15:41:26', STRUCTURED-DATA: '-',
msg: ' @cee:{"msg":"[umberlog test] testing ... 1, 2,
3\n","count":"2","el.0.type":"ladybug","el.0.name":"natalie","el.0.spots":"12","el.1.type":"ant","el.1.name":"B289","el.1.role":"worker","el.1.age":"9","pid":"1828","facility":"user","priority":"warn","uid":"1002","gid":"1002","host":"dboles-victim-0","program":"","timestamp":"2015-06-08T15:41:26.302102005-0400"}'
escaped msg: ' @cee:{"msg":"[umberlog test] testing ... 1, 2,
3\n","count":"2","el.0.type":"ladybug","el.0.name":"natalie","el.0.spots":"12","el.1.type":"ant","el.1.name":"B289","el.1.role":"worker","el.1.age":"9","pid":"1828","facility":"user","priority":"warn","uid":"1002","gid":"1002","host":"dboles-victim-0","program":"","timestamp":"2015-06-08T15:41:26.302102005-0400"}'
inputname: imuxsock rawmsg: '<12>Jun 8 15:41:26 : @cee:{"msg":"[umberlog test] testing ... 1, 2,
3\n","count":"2","el.0.type":"ladybug","el.0.name":"natalie","el.0.spots":"12","el.1.type":"ant","el.1.name":"B289","el.1.role":"worker","el.1.age":"9","pid":"1828","facility":"user","priority":"warn","uid":"1002","gid":"1002","host":"dboles-victim-0","program":"","timestamp":"2015-06-08T15:41:26.302102005-0400"}'
$!:
$.:
$/:
--------
You also asked for what my external Python script gets as input:
{ "msg": " @cee:{\"msg\":\"[umberlog test] testing ... 1, 2,
3\\n\",\"count\":\"2\",\"el.0.type\":\"
ladybug\",\"el.0.name\":\"natalie\",\"el.0.spots\":\"12\",\"el.1.type\":\"ant\",\"el.1.name\":\"B289
\",\"el.1.role\":\"worker\",\"el.1.age\":\"9\",\"pid\":\"4440\",\"facility\":\"user\",\"priority\":\
"warn\",\"uid\":\"1002\",\"gid\":\"1002\",\"host\":\"dboles-victim-0\",\"program\":\"\",\"timestamp\
":\"2015-06-08T16:28:12.877502675-0400\"}", "rawmsg": "<12>Jun 8 16:28:12 :
@cee:{\"msg\":\"[umberl
og test] testing ... 1, 2,
3\\n\",\"count\":\"2\",\"el.0.type\":\"ladybug\",\"el.0.name\":\"natalie\
",\"el.0.spots\":\"12\",\"el.1.type\":\"ant\",\"el.1.name\":\"B289\",\"el.1.role\":\"worker\",\"el.1
.age\":\"9\",\"pid\":\"4440\",\"facility\":\"user\",\"priority\":\"warn\",\"uid\":\"1002\",\"gid\":\
"1002\",\"host\":\"dboles-victim-0\",\"program\":\"\",\"timestamp\":\"2015-06-08T16:28:12.877502675-
0400\"}", "timereported": "2015-06-08T16:28:12.877616-04:00", "hostname":
"dboles-victim-0", "syslog
tag": ":", "inputname": "imuxsock", "fromhost": "dboles-victim-0", "fromhost-ip":
"127.0.0.1", "pri"
: "12", "syslogfacility": "1", "syslogseverity": "4", "timegenerated":
"2015-06-08T16:28:12.877616-0
4:00", "programname": "", "protocol-version": "0", "structured-data": "-", "app-name":
"", "procid":
"-", "msgid": "-", "uuid": null, "$!": { "msg": "[umberlog test] testing ... 1, 2, 3\n",
"count": "
2", "el.0.type": "ladybug", "el.0.name": "natalie", "el.0.spots": "12", "el.1.type":
"ant", "el.1.na
me": "B289", "el.1.role": "worker", "el.1.age": "9", "pid": "4440", "facility": "user",
"priority":
"warn", "uid": "1002", "gid": "1002", "host": "dboles-victim-0", "program": "",
"timestamp": "2015-0
6-08T16:28:12.877502675-0400" } }
________________________________________
From: [email protected] [[email protected]] on
behalf of David Lang [[email protected]]
Sent: Monday, June 01, 2015 5:06 PM
To: rsyslog-users
Subject: Re: [rsyslog] Unable to use foreach
On Sun, 24 May 2015, David Boles (dboles) wrote:
Thanks Rainer / David,
Based on Rainer's input I've pivoted to going down the mmexternal path and have
questions based on experimenting with that.
To answer David's "what are you trying to do" question and make things as
simple and concrete as possible I've created a clean example (log entry generator,
rsyslog configuration, mmexternal plugin, etc.) that captures what I'm trying to do. To
avoid pasting a bunch of stuff into these emails I set up a git repo with the example at:
https://github.com/davidboles/snowflake.git
The README.md explains the elements of that repo. I have the following
questions:
(1) The generated result does not have the { "hostname" : "frodo" } anywhere
in it. The docs say that I can add elements to the JSON elements of the
message, although that sentence is ambiguous as to whether I can do so if I
haven't modified some other element.
sorry for the delay in responding
in rsyslog, you don't have many elements. There are only a handful of built-in
properties and then everything else is in $!
so if you want to modify anything in $!, you must output a completely new $!
string (which is the JSON representation of the tree.
If you are wanting to add a new item "elements", that means that you are going
to modify the $! variable to include your new element.
the section talking about "if you modify the message variable tree" is saying
that if you don't change $!, you can't add new variables, because all variables
are inside of $!.
although, since this talks about the inability to delete things, it may be that
what you output is merged with what's already in $!. I'd have to test this (see
how to test below)
(2) If I uncomment the template line (line 20 of build.conf) then syslog
SEGV's. How do I access the jsonmesg content after getting stuff back through
mmexternal?
it's not line 20 in the currently visible file, but the best way to troubleshoot
"how do I access blah" is to write a file with the format RSYSLOG_DebugFormat
and it will show you all the variables that are defined at that point. In a
situation like yours where you are doing multiple parse actions, it will show
you the variables as of that point in the processing.
so, to see how to access the variables that were set with mmexternal, after that
action add another one
/var/log/test-debug;RSYSLOG_DebugFormat
and then look at the $!: line in that debug output. It will show you exactly
what variables are defined at that point.
I suspect that what you are outputting doesn't quite match
(3) The field "MUTABLE" has the value "Alpha" in the input to the mmexternal
unit (I can see it in /tmp/rewriter.trace entries). What JSON should my
rewriter.py emit to cause that to be rewritten to "Beta"?
I think it would be useful for you to show what the string is that rewriter.py
receives, what it outputs, and what the resulting debug is. I think that will
make it pretty obvious as to what's happening
David Lang
Thanks,
David Boles
________________________________________
From: [email protected] [[email protected]] on
behalf of Rainer Gerhards [[email protected]]
Sent: Friday, May 22, 2015 5:05 PM
To: rsyslog-users
Subject: Re: [rsyslog] Unable to use foreach
External plugin interface:
https://github.com/rsyslog/rsyslog/blob/master/plugins/external/INTERFACE.md
Overview :
https://github.com/rsyslog/rsyslog/blob/master/plugins/external/README.md
Sent from phone, thus brief.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
