Re: [rsyslog] Transforming JSON Field Names in rawmsg

[David Lang]

On Wed, 3 Oct 2018, John Chivian wrote:

Thank you David, problem solved.  Your message set me on the right track 
to discover the true issue, a deficiency in my understanding of the way 
rsyslog receives information back from the plugin.  I had been returning 
just the modified string, not a JSON safe representation of it preceded 
by a field name.


I've updated the script so that it now specifies rawmsg as the field 
being returned, and verified that rawmsg has indeed been changed in 
actions after the mmexternal normalization.


Interesting, I didn't think the built-in properties were able to be modified, 
that actually sounds like a bug to me :-)


you really do want to be working in the $! or $. variable namespaces, even if 
you modify rawmsg, any parsing of rawmsg has long since taken place, so the 
other properties are not getting set accordingly.


David Lang


Many thanks for the shove in the right direction!
John


On 10/3/18 7:17 PM, David Lang wrote:

On Wed, 3 Oct 2018, John Chivian wrote:



Hi David - Examples as requested...

    The generation of a test message...

   $ echo '{ "a": 1,"b": "string","field three": "example","else": 12 
}' | nc localhost 5139


that is not a valid syslog message


    What the script received...

   $ tail -1 /logspool/prefix_input.txt
   { "a": 1,"b": "string","field three": "example","else": 12 }


    The output from the plugin per the action...

   $ tail -1 /logspool/prefix_results.txt
   { "ny_a": 1,"ny_b": "string","ny_field three": 
"example","ny_else": 12 }



    The output from the rawmsg write action immediately afterward...

   $ tail -1 /logspool/mext.out
   { "a": 1,"b": "string","field three": "example","else": 12 }


this is correct, you don't change rawmsg, you change the variables 
under $!


output the message with the template RSYSLOG_DebugFormat and you will 
see the details of what's there.


David Lang
___ 


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Transforming JSON Field Names in rawmsg

[John Chivian]

Thank you David, problem solved.  Your message set me on the right track 
to discover the true issue, a deficiency in my understanding of the way 
rsyslog receives information back from the plugin.  I had been returning 
just the modified string, not a JSON safe representation of it preceded 
by a field name.


I've updated the script so that it now specifies rawmsg as the field 
being returned, and verified that rawmsg has indeed been changed in 
actions after the mmexternal normalization.


Many thanks for the shove in the right direction!
John


On 10/3/18 7:17 PM, David Lang wrote:

On Wed, 3 Oct 2018, John Chivian wrote:



Hi David - Examples as requested...

    The generation of a test message...

   $ echo '{ "a": 1,"b": "string","field three": "example","else": 12 
}' | nc localhost 5139


that is not a valid syslog message


    What the script received...

   $ tail -1 /logspool/prefix_input.txt
   { "a": 1,"b": "string","field three": "example","else": 12 }


    The output from the plugin per the action...

   $ tail -1 /logspool/prefix_results.txt
   { "ny_a": 1,"ny_b": "string","ny_field three": 
"example","ny_else": 12 }



    The output from the rawmsg write action immediately afterward...

   $ tail -1 /logspool/mext.out
   { "a": 1,"b": "string","field three": "example","else": 12 }


this is correct, you don't change rawmsg, you change the variables 
under $!


output the message with the template RSYSLOG_DebugFormat and you will 
see the details of what's there.


David Lang
___ 


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Transforming JSON Field Names in rawmsg

[David Lang]

On Wed, 3 Oct 2018, John Chivian wrote:



Hi David - Examples as requested...

    The generation of a test message...

   $ echo '{ "a": 1,"b": "string","field three": "example","else": 12
   }' | nc localhost 5139


that is not a valid syslog message


    What the script received...

   $ tail -1 /logspool/prefix_input.txt
   { "a": 1,"b": "string","field three": "example","else": 12 }


    The output from the plugin per the action...

   $ tail -1 /logspool/prefix_results.txt
   { "ny_a": 1,"ny_b": "string","ny_field three": "example","ny_else": 12 }


    The output from the rawmsg write action immediately afterward...

   $ tail -1 /logspool/mext.out
   { "a": 1,"b": "string","field three": "example","else": 12 }


this is correct, you don't change rawmsg, you change the variables under $!

output the message with the template RSYSLOG_DebugFormat and you will see the 
details of what's there.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Transforming JSON Field Names in rawmsg

[John Chivian]

Hi David - Examples as requested...

    The generation of a test message...

   $ echo '{ "a": 1,"b": "string","field three": "example","else": 12
   }' | nc localhost 5139


    What the script received...

   $ tail -1 /logspool/prefix_input.txt
   { "a": 1,"b": "string","field three": "example","else": 12 }


    The output from the plugin per the action...

   $ tail -1 /logspool/prefix_results.txt
   { "ny_a": 1,"ny_b": "string","ny_field three": "example","ny_else": 12 }


    The output from the rawmsg write action immediately afterward...

   $ tail -1 /logspool/mext.out
   { "a": 1,"b": "string","field three": "example","else": 12 }


    I've also discovered that if I do...

   set $.nrawmsg = $rawmsg;


...between the two action blocks that $.nrawmsg also contains the 
original rawmsg value, not the transformed one.



   The script that is the external normalization is very simple.

#!/bin/bash

   #!/bin/bash

   while read -r line
   do
  echo "${line}" >> /logspool/prefix_input.txt
  echo "${line}" | sed 's/\([^"]\+\)"\([^"]\+\)":/\1"ny_\2":/g'
   done

   exit


Regards, John



On 10/3/18 12:57 PM, David Lang wrote:

On Wed, 3 Oct 2018, John Chivian wrote:


Thank you David.  You're comment has led me to this...

  ruleset(name="fix_names") {
    action(
  type="mmexternal"
  name="normalize-names"
  binary="/etc/rsyslog.d/transforms/prefix_names.sh"
  interface.input="rawmsg"
  output="/logspool/prefix_results.txt"
    )
    action(
  type="omfile"
  name="mmexternal-debug"
  template="rawmsg"
  File="/logspool/mext.out"
    )
}

I know the external script is functioning because the 
prefix_results.txt file (from the first action) shows the correct 
results, but when I then immediately write out the rawmsg (in the 
second action) I get the original, unmodified value.


can you provide an example of a log message that's send to your script 
and the output that you get from your script?


David Lang

I must be missing something fundamental, and will revisit the 
documentation, but I'd be grateful for any words of wisdom or guidance.


Thanks, John

On 10/2/18 3:11 PM, David Lang wrote:

On Tue, 2 Oct 2018, John Chivian wrote:


Hello Group:

   I am trying to determine the best way to transform the field 
names of a simple JSON object that is rawmsg.  The objects are 
fluid having both numeric and string content, but are always in the 
form...


    { "aStr": "aString","bStr": "bString","cNum": 0,"dStr": 
"cString" }


   I need to add a prefix "ny_" to the field names such that the 
result would be...


    { "ny_aStr": "aString","ny_bStr": "bString","ny_cNum": 
0,"ny_dStr": "cString" }


   I have a sed script with extractions that can do this 
transformation...


's/\([^"]\+\)"\([^"]\+\)":/\1"ny_\2":/g'

   ...but I don't know if it's possible to integrate that into a 
template with the property replacer, or if there's a better cleaner 
way to do it.


   Any and all recommendations are greatly appreciated.


I think you would have to resort to code outside of rsyslog (either 
a custom mm module or mmexternal to call a script of your devising) 
to change the field contents like that for an arbitrary and changing 
list of fields.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a 
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT 
POST if you DON'T LIKE THAT.



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a 
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT 
POST if you DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a 
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST 
if you DON'T LIKE THAT.



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Extracting text from msg field to make structure message

[David Lang]

On Wed, 3 Oct 2018, Lavanya Kanchanapalli via rsyslog wrote:


I am trying to parse log messages and transform them to structured
messages. Is there a way support such operation with rsyslog config


Yes, what I do is use mmnormalize to do the parsing, and then I use a template:

$template structured,"<%PRI%>1 %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% 
%PROCID% %MSGID% %STRUCTURED-DATA% %$!%\n"

to output things as a json structure in the body of the message, making it very 
easy for downstream things to parse it, add metadata, etc.



can do some of it. Is there a way to do the following using rsyslog config?

  1. Map 2 fields to single output name. Ex: "__ts": "2018-09-20
  10:18:56.363" (first 2 fields in example below).
  2. Extract what is left in msg after extracting all known fields based
  on position. Ex: "msg": "Unregistering application nameOfAnApiHere with
  someOtherName with status DOWN"


rather than going into details, look at mmnormalize


  3. Is there a way to use local variables
  

  to hold the values of fields from msg and use the variables in templates?


yes, there is no technical difference between $. variable and $! variables, we 
started with $! variables, so they are the 'default' that all documentation 
uses. We added $. variables so that we have variables that we can set that will 
NOT be included when we output $! somewhere



*1. rsyslog config template definition*


templates are for output only, not for parsing

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Extracting text from msg field to make structure message

[Lavanya Kanchanapalli via rsyslog]

Hi,
I am trying to parse log messages and transform them to structured
messages. Is there a way support such operation with rsyslog config (have
not yet explored the option to write custom parser or message modification
plugin for this)? I found template list properties

which
can do some of it. Is there a way to do the following using rsyslog config?

   1. Map 2 fields to single output name. Ex: "__ts": "2018-09-20
   10:18:56.363" (first 2 fields in example below).
   2. Extract what is left in msg after extracting all known fields based
   on position. Ex: "msg": "Unregistering application nameOfAnApiHere with
   someOtherName with status DOWN"
   3. Is there a way to use local variables
   

   to hold the values of fields from msg and use the variables in templates?

*Example Log message:*
2018-09-20 10:18:56.363  INFO --- [Thread-68] x.y.z.key1Value Unregistering
application nameOfAnApiHere with someOtherName with status DOWN

*1. rsyslog config template definition*

template(name="structure-log-format" type="list") {
constant(value="{")

# This only extracts the first field with value 2018-09-20.
*# TODO: What is a way to map first 2 fields to map to __ts field? *
property(outname="__ts" name="msg" field.number="1"
field.delimiter="32" format="jsonf") constant(value=", ")

constant(value="\"event\":[{")
constant(value="\"payload\":{")
property(outname="_log_" name="syslogtag" format="jsonf")
constant(value=", ")
property(outname="__loglvl" name="msg" field.number="4"
field.delimiter="32" format="jsonf") constant(value=", ")
property(outname="__thread" name="msg" field.number="7"
field.delimiter="32" format="jsonf") constant(value=", ")
property(outname="__key1" name="msg" field.number="8"
field.delimiter="32" format="jsonf") constant(value=", ")
# The following setting will include full message value
starting from "2018-09-20 ... DOWN"
*# TODO: What is a way to only include message starting
from "**Unregistering
... DOWN"?*
property(name="msg" format="jsonf" droplastlf="on" )
constant(value="}")
constant(value="}]} \n")
}

*2. Expected result: *
{
 * "__ts": "2018-09-20 10:18:56.363"*,
  "event": [
{
  "payload": {
"_log_": "catalina",
"__loglvl": "INFO",
"__thread": "Thread-68",
"__key1": "x.y.z.key1Value",
*"msg": "Unregistering application nameOfAnApiHere with
someOtherName with status DOWN"*
  }
}
  ]
}

*3. Actual result:*
{
 * "__ts": "2018-09-20"*,
  "event": [
{
  "payload": {
"_log_": "catalina",
"__loglvl": "INFO",
"__thread": "Thread-68",
"__key1": "x.y.z.key1Value",
*"msg": "2018-09-20 10:18:56.363  INFO 2144 --- [Thread-68]
x.y.z.key1Value Unregistering application nameOfAnApiHere with
someOtherName with status DOWN"*
  }
}
  ]
}

Thank you,
Lavanya K
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Transforming JSON Field Names in rawmsg

[David Lang]

On Wed, 3 Oct 2018, John Chivian wrote:


Thank you David.  You're comment has led me to this...

  ruleset(name="fix_names") {
    action(
  type="mmexternal"
  name="normalize-names"
  binary="/etc/rsyslog.d/transforms/prefix_names.sh"
  interface.input="rawmsg"
  output="/logspool/prefix_results.txt"
    )
    action(
  type="omfile"
  name="mmexternal-debug"
  template="rawmsg"
  File="/logspool/mext.out"
    )
}

I know the external script is functioning because the prefix_results.txt 
file (from the first action) shows the correct results, but when I then 
immediately write out the rawmsg (in the second action) I get the 
original, unmodified value.


can you provide an example of a log message that's send to your script and the 
output that you get from your script?


David Lang

I must be missing something fundamental, and will revisit the 
documentation, but I'd be grateful for any words of wisdom or guidance.


Thanks, John

On 10/2/18 3:11 PM, David Lang wrote:

On Tue, 2 Oct 2018, John Chivian wrote:


Hello Group:

   I am trying to determine the best way to transform the field names 
of a simple JSON object that is rawmsg.  The objects are fluid having 
both numeric and string content, but are always in the form...


    { "aStr": "aString","bStr": "bString","cNum": 0,"dStr": 
"cString" }


   I need to add a prefix "ny_" to the field names such that the 
result would be...


    { "ny_aStr": "aString","ny_bStr": "bString","ny_cNum": 
0,"ny_dStr": "cString" }


   I have a sed script with extractions that can do this 
transformation...


's/\([^"]\+\)"\([^"]\+\)":/\1"ny_\2":/g'

   ...but I don't know if it's possible to integrate that into a 
template with the property replacer, or if there's a better cleaner 
way to do it.


   Any and all recommendations are greatly appreciated.


I think you would have to resort to code outside of rsyslog (either a 
custom mm module or mmexternal to call a script of your devising) to 
change the field contents like that for an arbitrary and changing list 
of fields.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a 
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST 
if you DON'T LIKE THAT.



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Transforming JSON Field Names in rawmsg

[David Lang]

On Wed, 3 Oct 2018, singh.janmejay via rsyslog wrote:


It seems like you can parse the
message(https://www.rsyslog.com/doc/v8-stable/configuration/modules/mmjsonparse.html)
and set new fields (and unset the old fields) in rainerscript
(https://www.rsyslog.com/doc/v8-stable/rainerscript/variable_property_types.html).


that would require that you know every field name ahead of time to put it in the 
rsyslog config, there is no way to walk the structure and modify what you find.


David Lang



On Wed, Oct 3, 2018 at 7:06 PM John Chivian  wrote:


Thank you David.  You're comment has led me to this...

   ruleset(name="fix_names") {
 action(
   type="mmexternal"
   name="normalize-names"
   binary="/etc/rsyslog.d/transforms/prefix_names.sh"
   interface.input="rawmsg"
   output="/logspool/prefix_results.txt"
 )
 action(
   type="omfile"
   name="mmexternal-debug"
   template="rawmsg"
   File="/logspool/mext.out"
 )
}

I know the external script is functioning because the prefix_results.txt
file (from the first action) shows the correct results, but when I then
immediately write out the rawmsg (in the second action) I get the
original, unmodified value.

I must be missing something fundamental, and will revisit the
documentation, but I'd be grateful for any words of wisdom or guidance.

Thanks, John

On 10/2/18 3:11 PM, David Lang wrote:

On Tue, 2 Oct 2018, John Chivian wrote:


Hello Group:

   I am trying to determine the best way to transform the field names
of a simple JSON object that is rawmsg.  The objects are fluid having
both numeric and string content, but are always in the form...

{ "aStr": "aString","bStr": "bString","cNum": 0,"dStr":
"cString" }

   I need to add a prefix "ny_" to the field names such that the
result would be...

{ "ny_aStr": "aString","ny_bStr": "bString","ny_cNum":
0,"ny_dStr": "cString" }

   I have a sed script with extractions that can do this
transformation...

's/\([^"]\+\)"\([^"]\+\)":/\1"ny_\2":/g'

   ...but I don't know if it's possible to integrate that into a
template with the property replacer, or if there's a better cleaner
way to do it.

   Any and all recommendations are greatly appreciated.


I think you would have to resort to code outside of rsyslog (either a
custom mm module or mmexternal to call a script of your devising) to
change the field contents like that for an arbitrary and changing list
of fields.

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if you DON'T LIKE THAT.



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.






___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Transforming JSON Field Names in rawmsg

[singh.janmejay via rsyslog]

It seems like you can parse the
message(https://www.rsyslog.com/doc/v8-stable/configuration/modules/mmjsonparse.html)
and set new fields (and unset the old fields) in rainerscript
(https://www.rsyslog.com/doc/v8-stable/rainerscript/variable_property_types.html).
On Wed, Oct 3, 2018 at 7:06 PM John Chivian  wrote:
>
> Thank you David.  You're comment has led me to this...
>
>ruleset(name="fix_names") {
>  action(
>type="mmexternal"
>name="normalize-names"
>binary="/etc/rsyslog.d/transforms/prefix_names.sh"
>interface.input="rawmsg"
>output="/logspool/prefix_results.txt"
>  )
>  action(
>type="omfile"
>name="mmexternal-debug"
>template="rawmsg"
>File="/logspool/mext.out"
>  )
> }
>
> I know the external script is functioning because the prefix_results.txt
> file (from the first action) shows the correct results, but when I then
> immediately write out the rawmsg (in the second action) I get the
> original, unmodified value.
>
> I must be missing something fundamental, and will revisit the
> documentation, but I'd be grateful for any words of wisdom or guidance.
>
> Thanks, John
>
> On 10/2/18 3:11 PM, David Lang wrote:
> > On Tue, 2 Oct 2018, John Chivian wrote:
> >
> >> Hello Group:
> >>
> >>I am trying to determine the best way to transform the field names
> >> of a simple JSON object that is rawmsg.  The objects are fluid having
> >> both numeric and string content, but are always in the form...
> >>
> >> { "aStr": "aString","bStr": "bString","cNum": 0,"dStr":
> >> "cString" }
> >>
> >>I need to add a prefix "ny_" to the field names such that the
> >> result would be...
> >>
> >> { "ny_aStr": "aString","ny_bStr": "bString","ny_cNum":
> >> 0,"ny_dStr": "cString" }
> >>
> >>I have a sed script with extractions that can do this
> >> transformation...
> >>
> >> 's/\([^"]\+\)"\([^"]\+\)":/\1"ny_\2":/g'
> >>
> >>...but I don't know if it's possible to integrate that into a
> >> template with the property replacer, or if there's a better cleaner
> >> way to do it.
> >>
> >>Any and all recommendations are greatly appreciated.
> >
> > I think you would have to resort to code outside of rsyslog (either a
> > custom mm module or mmexternal to call a script of your devising) to
> > change the field contents like that for an arbitrary and changing list
> > of fields.
> >
> > David Lang
> > ___
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> > if you DON'T LIKE THAT.
>
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Transforming JSON Field Names in rawmsg

[John Chivian]

Thank you David.  You're comment has led me to this...

  ruleset(name="fix_names") {
    action(
  type="mmexternal"
  name="normalize-names"
  binary="/etc/rsyslog.d/transforms/prefix_names.sh"
  interface.input="rawmsg"
  output="/logspool/prefix_results.txt"
    )
    action(
  type="omfile"
  name="mmexternal-debug"
  template="rawmsg"
  File="/logspool/mext.out"
    )
}

I know the external script is functioning because the prefix_results.txt 
file (from the first action) shows the correct results, but when I then 
immediately write out the rawmsg (in the second action) I get the 
original, unmodified value.


I must be missing something fundamental, and will revisit the 
documentation, but I'd be grateful for any words of wisdom or guidance.


Thanks, John

On 10/2/18 3:11 PM, David Lang wrote:

On Tue, 2 Oct 2018, John Chivian wrote:


Hello Group:

   I am trying to determine the best way to transform the field names 
of a simple JSON object that is rawmsg.  The objects are fluid having 
both numeric and string content, but are always in the form...


    { "aStr": "aString","bStr": "bString","cNum": 0,"dStr": 
"cString" }


   I need to add a prefix "ny_" to the field names such that the 
result would be...


    { "ny_aStr": "aString","ny_bStr": "bString","ny_cNum": 
0,"ny_dStr": "cString" }


   I have a sed script with extractions that can do this 
transformation...


's/\([^"]\+\)"\([^"]\+\)":/\1"ny_\2":/g'

   ...but I don't know if it's possible to integrate that into a 
template with the property replacer, or if there's a better cleaner 
way to do it.


   Any and all recommendations are greatly appreciated.


I think you would have to resort to code outside of rsyslog (either a 
custom mm module or mmexternal to call a script of your devising) to 
change the field contents like that for an arbitrary and changing list 
of fields.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a 
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST 
if you DON'T LIKE THAT.



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.