There is a key-value option in mmnormalize, last I looked (a couple years ago),
I was buried behind some odd name (iptables or something like that) and really
needed just a little attention to make the separators configurable (is is '=',
':', ': ", etc. is it '&', '|', ',', ' ', etc)
I don't know if that ever got done.
David Lang
On Fri, 10
Aug 2018, eSX via rsyslog wrote:
Date: Fri, 10 Aug 2018 19:40:35 +0800
From: eSX via rsyslog
To: rsyslog-users
Cc: eSX
Subject: [rsyslog] mmfields is not enough for field extraction.
Hi all.
I have read the documents of mmfield.
And I think it's not enough.
There is a typical scenario:
Many key-values are concatenated by separator, but not one field only.
e.g. key-value pairs are concatenated in url-query mode, like:
foo=123=456=bbb==xxx
seems there is no way can split them into json easily in rsyslog? property
replacer? mmnormalize?
I want to split above like:
{
"foo":"123",
"bar ":"456",
"aaa":"bbb",
"empty":""
...
}
It seems mmnormalize could split string like:
{ "fields": [
{ "key": "foo", "value": "123" },
{ "key": "bar", "value": "456" },
{ "key": "aaa", "value": "bbb" }
...
]
}
it's a big waste in extra field name.
property replacer, with $!url:F, but I can't know how many fields splited
also.
with regexp, It's too expensive and inconvenient too.
mmfields can split url-query, but the fields split into json, like:
{
"f1":"foo=123",
"f2":"bar=456 ",
"f3":"aaa=bbb",
...
}
I think it's necessary to implement a method to resolve this.
like:
kv processor in elasticsearch:
https://www.elastic.co/guide/en/elasticsearch/reference/current/kv-processor.html
kv plugins in logstash:
https://www.elastic.co/guide/en/logstash/current/plugins-filters-kv.html
str_to_map in hive:
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF
a simple way, implement str_to_map() is easy and useful or
add a parameter in mmfields for a second separator?
BTW, I think decode functions is necessary too,like:
urldecode/base64
Thanks
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
