On 5/29/19 1:27 PM, Noriko Hosoi via rsyslog wrote:
Hello rsyslog list,
[1] is an example of pre-processed log. We'd like to keep fields defined in the environment variable [3] in the top level, and move the rest to one level down as in [2]. In this example, 2
fields "test0" and "test1" are to be moved, but the count could vary. We'd like to do something like [5] with the rulebase [4]. (*) But this preliminary attempt does not work since 1)
"contains" only takes quoted value (https://www.rsyslog.com/doc/v8-stable/configuration/filters.html#value-part) and 2) mmnormalize with [4] replaces(?) the value instead of adds it. Could
you please enlighten me how we could achieve the goal?
Instead of keeping the env. vars. as is, convert them into a lookup table.
Then, loop through the fields in the record, and see if the field is in the
lookup table.
https://www.rsyslog.com/doc/v8-stable/configuration/lookup_tables.html
We're using lookup tables in openshift logging:
https://github.com/openshift/cluster-logging-operator/blob/master/files/rsyslog/65-viaq-formatting.conf#L2
https://github.com/openshift/cluster-logging-operator/blob/master/files/rsyslog/65-viaq-formatting.conf#L195
In rsyslog.sh:
https://github.com/openshift/cluster-logging-operator/blob/master/files/rsyslog/rsyslog.sh
you would need to convert the env. var. values into a lookup table JSON file.
The keys would be the field names from the env. var. The value would be "true" - the 'not
found' value would be "false". e.g.
foreach ($.ii in $!) do {
if (lookup("default_keep_fields", $.ii) == "true") then {
...
}
}
( not sure if you can use the result of the lookup() function in an `if`, or if
you need a temporary variable to hold the result, but that's the idea)
(*) Note: this script [5] has a culprit, too. If the undefined field name is a substring of DEFAULT_KEEP_FIELDS, e.g., "host", "name" and "addr", then they won't be moved to "undefined"
even if the script worked as I hoped. So, probably, we have to come up with a completely new idea...
Thank you, in advance.
--noriko
[1] input:
{
"message": "Test Message 0",
"level": "info",
"@timestamp": "2019-05-27T23:49:09.507361+00:00",
"hostname": "ip-10-0-172-2.compute.internal",
"test0": "string0",
"test1": { "test10": 10, "test11": "string11" }
}
[2] output:
{
"message": "Test Message 0",
"level": "info",
"@timestamp": "2019-05-27T23:49:09.507361+00:00",
"hostname": "ip-10-0-172-2.compute.internal",
"undefined": { "test0": "string0", "test1": { "test10": 10, "test11":
"string11" } }
}
[3] DEFAULT_KEEP_FIELDS:
export
DEFAULT_KEEP_FIELDS="CEE,time,@timestamp,hostname,ipaddr4,ipaddr6,level,message"
[4]
==> files/rsyslog/undefined.rulebase <==
version=2
rule=:%undefined:json%
[5]
<<snippet of config script>>
set $.default_keep_fields = `echo $DEFAULT_KEEP_FIELDS`;
foreach ($.i in $!) do {
if not ($.default_keep_fields contains $.i!key) {
action(type="mmnormalize" ruleBase="/etc/rsyslog.d/undefined.rulebase"
variable=$.i!key)
}
}
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
