My recommendation is to pass the message on to an mmexternal module
(that you write) that does the required transformation...
action(
type="mmexternal"
name="normalize-json"
binary="/path/to/transform.exec"
interface.input="rawmsg"
output="/path/to/transform.debug"
)
...where transform.exec is any executable that does what you need it to do.
You'll need to understand how external modules work, but it's fairly
simple to implement and this is the perfect use case for one.
Regards,
On 5/29/19 2:27 PM, Noriko Hosoi via rsyslog wrote:
Hello rsyslog list,
[1] is an example of pre-processed log. We'd like to keep fields
defined in the environment variable [3] in the top level, and move the
rest to one level down as in [2]. In this example, 2 fields "test0"
and "test1" are to be moved, but the count could vary. We'd like to do
something like [5] with the rulebase [4]. (*) But this preliminary
attempt does not work since 1) "contains" only takes quoted value
(https://www.rsyslog.com/doc/v8-stable/configuration/filters.html#value-part)
and 2) mmnormalize with [4] replaces(?) the value instead of adds it.
Could you please enlighten me how we could achieve the goal?
(*) Note: this script [5] has a culprit, too. If the undefined field
name is a substring of DEFAULT_KEEP_FIELDS, e.g., "host", "name" and
"addr", then they won't be moved to "undefined" even if the script
worked as I hoped. So, probably, we have to come up with a completely
new idea...
Thank you, in advance.
--noriko
[1] input:
{
"message": "Test Message 0",
"level": "info",
"@timestamp": "2019-05-27T23:49:09.507361+00:00",
"hostname": "ip-10-0-172-2.compute.internal",
"test0": "string0",
"test1": { "test10": 10, "test11": "string11" }
}
[2] output:
{
"message": "Test Message 0",
"level": "info",
"@timestamp": "2019-05-27T23:49:09.507361+00:00",
"hostname": "ip-10-0-172-2.compute.internal",
"undefined": { "test0": "string0", "test1": { "test10": 10,
"test11": "string11" } }
}
[3] DEFAULT_KEEP_FIELDS:
export
DEFAULT_KEEP_FIELDS="CEE,time,@timestamp,hostname,ipaddr4,ipaddr6,level,message"
[4]
==> files/rsyslog/undefined.rulebase <==
version=2
rule=:%undefined:json%
[5]
<<snippet of config script>>
set $.default_keep_fields = `echo $DEFAULT_KEEP_FIELDS`;
foreach ($.i in $!) do {
if not ($.default_keep_fields contains $.i!key) {
action(type="mmnormalize"
ruleBase="/etc/rsyslog.d/undefined.rulebase" variable=$.i!key)
}
}
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
