David,
Sorry for the delay in responding - my focus was elsewhere temporarily. What
you say about $! and new content below makes sense. I've tried an experiment
with the RSYSLOG_DebugFormat mechanism by running with the following
/etc/rsyslog.conf tweaked in three ways:
--------
module(load="imuxsock")
module(load="mmexternal")
module(load="mmjsonparse")
module(load="ommongodb")
template(name="yyd-log-info" type="list") {
property(name="timestamp")
constant(value=" ")
property(name="hostname")
constant(value=" ")
property(name="msg")
constant(value="\n")
}
# 1
*.* /var/log/debug_pre.log;RSYSLOG_DebugFormat
if ($fromhost-ip == '127.0.0.1' and $syslogfacility-text != 'kern') then {
action(type="mmjsonparse")
if $parsesuccess == "OK" then {
# 2
action(type="mmexternal"
binary="/tmp/snowflake/rewriter.py"
interface.input="fulljson" )
action(type="omfile"
template="yyd-log-info"
file="/var/log/yyd_anon.log")
}
}
# 3
#*.* /var/log/debug_post.log;RSYSLOG_DebugFormat
--------
(1) One mode was with the mmexternal action after # 2 commented out and the
DebugFormat action at # 3, yielding:
--------
Debug line with all properties:
FROMHOST: 'dboles-victim-0', fromhost-ip: '127.0.0.1', HOSTNAME:
'dboles-victim-0', PRI: 12,
syslogtag ':', programname: '', APP-NAME: '', PROCID: '-', MSGID: '-',
TIMESTAMP: 'Jun 8 15:37:24', STRUCTURED-DATA: '-',
msg: ' @cee:{"msg":"[umberlog test] testing ... 1, 2,
3\n","count":"2","el.0.type":"ladybug","el.0.name":"natalie","el.0.spots":"12","el.1.type":"ant","el.1.name":"B289","el.1.role":"worker","el.1.age":"9","pid":"32294","facility":"user","priority":"warn","uid":"1002","gid":"1002","host":"dboles-victim-0","program":"","timestamp":"2015-06-08T15:37:24.449217401-0400"}'
escaped msg: ' @cee:{"msg":"[umberlog test] testing ... 1, 2,
3\n","count":"2","el.0.type":"ladybug","el.0.name":"natalie","el.0.spots":"12","el.1.type":"ant","el.1.name":"B289","el.1.role":"worker","el.1.age":"9","pid":"32294","facility":"user","priority":"warn","uid":"1002","gid":"1002","host":"dboles-victim-0","program":"","timestamp":"2015-06-08T15:37:24.449217401-0400"}'
inputname: imuxsock rawmsg: '<12>Jun 8 15:37:24 : @cee:{"msg":"[umberlog test]
testing ... 1, 2,
3\n","count":"2","el.0.type":"ladybug","el.0.name":"natalie","el.0.spots":"12","el.1.type":"ant","el.1.name":"B289","el.1.role":"worker","el.1.age":"9","pid":"32294","facility":"user","priority":"warn","uid":"1002","gid":"1002","host":"dboles-victim-0","program":"","timestamp":"2015-06-08T15:37:24.449217401-0400"}'
$!:{ "msg": "[umberlog test] testing ... 1, 2, 3\n", "count": "2", "el.0.type":
"ladybug", "el.0.name": "natalie", "el.0.spots": "12", "el.1.type": "ant",
"el.1.name": "B289", "el.1.role": "worker", "el.1.age": "9", "pid": "32294",
"facility": "user", "priority": "warn", "uid": "1002", "gid": "1002", "host":
"dboles-victim-0", "program": "", "timestamp":
"2015-06-08T15:37:24.449217401-0400" }
$.:
$/:
--------
This makes sense - the mmjsonparse populates the $! element.
(2) The second mode was with the mmexternal action at # 2 enabled and the
DebugFormat action at # 3 - this results in rsyslogd suffering a segfault.
(3) The third mode has the mmexternal action at # 2 enabled and the DebugFormat
action at #1 enabled:
Debug line with all properties:
FROMHOST: 'dboles-victim-0', fromhost-ip: '127.0.0.1', HOSTNAME:
'dboles-victim-0', PRI: 12,
syslogtag ':', programname: '', APP-NAME: '', PROCID: '-', MSGID: '-',
TIMESTAMP: 'Jun 8 15:41:26', STRUCTURED-DATA: '-',
msg: ' @cee:{"msg":"[umberlog test] testing ... 1, 2,
3\n","count":"2","el.0.type":"ladybug","el.0.name":"natalie","el.0.spots":"12","el.1.type":"ant","el.1.name":"B289","el.1.role":"worker","el.1.age":"9","pid":"1828","facility":"user","priority":"warn","uid":"1002","gid":"1002","host":"dboles-victim-0","program":"","timestamp":"2015-06-08T15:41:26.302102005-0400"}'
escaped msg: ' @cee:{"msg":"[umberlog test] testing ... 1, 2,
3\n","count":"2","el.0.type":"ladybug","el.0.name":"natalie","el.0.spots":"12","el.1.type":"ant","el.1.name":"B289","el.1.role":"worker","el.1.age":"9","pid":"1828","facility":"user","priority":"warn","uid":"1002","gid":"1002","host":"dboles-victim-0","program":"","timestamp":"2015-06-08T15:41:26.302102005-0400"}'
inputname: imuxsock rawmsg: '<12>Jun 8 15:41:26 : @cee:{"msg":"[umberlog test]
testing ... 1, 2,
3\n","count":"2","el.0.type":"ladybug","el.0.name":"natalie","el.0.spots":"12","el.1.type":"ant","el.1.name":"B289","el.1.role":"worker","el.1.age":"9","pid":"1828","facility":"user","priority":"warn","uid":"1002","gid":"1002","host":"dboles-victim-0","program":"","timestamp":"2015-06-08T15:41:26.302102005-0400"}'
$!:
$.:
$/:
--------
You also asked for what my external Python script gets as input:
{ "msg": " @cee:{\"msg\":\"[umberlog test] testing ... 1, 2,
3\\n\",\"count\":\"2\",\"el.0.type\":\"
ladybug\",\"el.0.name\":\"natalie\",\"el.0.spots\":\"12\",\"el.1.type\":\"ant\",\"el.1.name\":\"B289
\",\"el.1.role\":\"worker\",\"el.1.age\":\"9\",\"pid\":\"4440\",\"facility\":\"user\",\"priority\":\
"warn\",\"uid\":\"1002\",\"gid\":\"1002\",\"host\":\"dboles-victim-0\",\"program\":\"\",\"timestamp\
":\"2015-06-08T16:28:12.877502675-0400\"}", "rawmsg": "<12>Jun 8 16:28:12 :
@cee:{\"msg\":\"[umberl
og test] testing ... 1, 2,
3\\n\",\"count\":\"2\",\"el.0.type\":\"ladybug\",\"el.0.name\":\"natalie\
",\"el.0.spots\":\"12\",\"el.1.type\":\"ant\",\"el.1.name\":\"B289\",\"el.1.role\":\"worker\",\"el.1
.age\":\"9\",\"pid\":\"4440\",\"facility\":\"user\",\"priority\":\"warn\",\"uid\":\"1002\",\"gid\":\
"1002\",\"host\":\"dboles-victim-0\",\"program\":\"\",\"timestamp\":\"2015-06-08T16:28:12.877502675-
0400\"}", "timereported": "2015-06-08T16:28:12.877616-04:00", "hostname":
"dboles-victim-0", "syslog
tag": ":", "inputname": "imuxsock", "fromhost": "dboles-victim-0",
"fromhost-ip": "127.0.0.1", "pri"
: "12", "syslogfacility": "1", "syslogseverity": "4", "timegenerated":
"2015-06-08T16:28:12.877616-0
4:00", "programname": "", "protocol-version": "0", "structured-data": "-",
"app-name": "", "procid":
"-", "msgid": "-", "uuid": null, "$!": { "msg": "[umberlog test] testing ...
1, 2, 3\n", "count": "
2", "el.0.type": "ladybug", "el.0.name": "natalie", "el.0.spots": "12",
"el.1.type": "ant", "el.1.na
me": "B289", "el.1.role": "worker", "el.1.age": "9", "pid": "4440", "facility":
"user", "priority":
"warn", "uid": "1002", "gid": "1002", "host": "dboles-victim-0", "program": "",
"timestamp": "2015-0
6-08T16:28:12.877502675-0400" } }
________________________________________
From: [email protected] [[email protected]] on
behalf of David Lang [[email protected]]
Sent: Monday, June 01, 2015 5:06 PM
To: rsyslog-users
Subject: Re: [rsyslog] Unable to use foreach
On Sun, 24 May 2015, David Boles (dboles) wrote:
> Thanks Rainer / David,
>
> Based on Rainer's input I've pivoted to going down the mmexternal path and
> have questions based on experimenting with that.
>
> To answer David's "what are you trying to do" question and make things as
> simple and concrete as possible I've created a clean example (log entry
> generator, rsyslog configuration, mmexternal plugin, etc.) that captures what
> I'm trying to do. To avoid pasting a bunch of stuff into these emails I set
> up a git repo with the example at:
>
> https://github.com/davidboles/snowflake.git
>
> The README.md explains the elements of that repo. I have the following
> questions:
>
> (1) The generated result does not have the { "hostname" : "frodo" } anywhere
> in it. The docs say that I can add elements to the JSON elements of the
> message, although that sentence is ambiguous as to whether I can do so if I
> haven't modified some other element.
sorry for the delay in responding
in rsyslog, you don't have many elements. There are only a handful of built-in
properties and then everything else is in $!
so if you want to modify anything in $!, you must output a completely new $!
string (which is the JSON representation of the tree.
If you are wanting to add a new item "elements", that means that you are going
to modify the $! variable to include your new element.
the section talking about "if you modify the message variable tree" is saying
that if you don't change $!, you can't add new variables, because all variables
are inside of $!.
although, since this talks about the inability to delete things, it may be that
what you output is merged with what's already in $!. I'd have to test this (see
how to test below)
> (2) If I uncomment the template line (line 20 of build.conf) then syslog
> SEGV's. How do I access the jsonmesg content after getting stuff back through
> mmexternal?
it's not line 20 in the currently visible file, but the best way to troubleshoot
"how do I access blah" is to write a file with the format RSYSLOG_DebugFormat
and it will show you all the variables that are defined at that point. In a
situation like yours where you are doing multiple parse actions, it will show
you the variables as of that point in the processing.
so, to see how to access the variables that were set with mmexternal, after that
action add another one
/var/log/test-debug;RSYSLOG_DebugFormat
and then look at the $!: line in that debug output. It will show you exactly
what variables are defined at that point.
I suspect that what you are outputting doesn't quite match
> (3) The field "MUTABLE" has the value "Alpha" in the input to the mmexternal
> unit (I can see it in /tmp/rewriter.trace entries). What JSON should my
> rewriter.py emit to cause that to be rewritten to "Beta"?
I think it would be useful for you to show what the string is that rewriter.py
receives, what it outputs, and what the resulting debug is. I think that will
make it pretty obvious as to what's happening
David Lang
> Thanks,
>
> David Boles
>
>
>
> ________________________________________
> From: [email protected] [[email protected]]
> on behalf of Rainer Gerhards [[email protected]]
> Sent: Friday, May 22, 2015 5:05 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Unable to use foreach
>
> External plugin interface:
> https://github.com/rsyslog/rsyslog/blob/master/plugins/external/INTERFACE.md
>
> Overview :
> https://github.com/rsyslog/rsyslog/blob/master/plugins/external/README.md
>
> Sent from phone, thus brief.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
