A test for this.
On Fri, Oct 31, 2014 at 7:55 AM, singh.janmejay <[email protected]> wrote: > Added doc for variable-support. > > Also fixed the ambiguous title (mmjsonparse and mmnormalize are now > identified in the title, hence on the index page). > > On Thu, Oct 30, 2014 at 5:08 PM, Rainer Gerhards <[email protected] > > wrote: > >> Right in the mmnormalize doc is the right spot: >> >> >> https://github.com/rsyslog/rsyslog-doc/blob/master/source/configuration/modules/mmnormalize.rst >> >> Rainer >> >> 2014-10-30 12:28 GMT+01:00 singh.janmejay <[email protected]>: >> >> > What is the right place to add doc for this? >> > >> > On Wed, Oct 29, 2014 at 7:05 PM, Rainer Gerhards < >> [email protected] >> > > >> > wrote: >> > >> > > These two are finally merged... >> > > >> > > Rainer >> > > >> > > 2014-10-21 15:58 GMT+02:00 singh.janmejay <[email protected]>: >> > > >> > > > No problem. >> > > > >> > > > -- >> > > > Regards, >> > > > Janmejay >> > > > >> > > > PS: Please blame the typos in this mail on my phone's uncivilized >> soft >> > > > keyboard sporting it's not-so-smart-assist technology. >> > > > >> > > > On Oct 21, 2014 4:03 PM, "Rainer Gerhards" < >> [email protected]> >> > > > wrote: >> > > > >> > > > > Thx for the patches. Just FYI: I am deep inside complex work, will >> > look >> > > > at >> > > > > merging later the week, if things go really bad early next week. >> > > > > >> > > > > Rainer >> > > > > >> > > > > 2014-10-21 9:32 GMT+02:00 singh.janmejay < >> [email protected]>: >> > > > > >> > > > > > On Tue, Oct 14, 2014 at 10:06 AM, singh.janmejay < >> > > > > [email protected] >> > > > > > > >> > > > > > wrote: >> > > > > > >> > > > > > > >> > > > > > > >> > > > > > > On Tue, Oct 14, 2014 at 9:23 AM, David Lang <[email protected]> >> > wrote: >> > > > > > > >> > > > > > >> On Tue, 14 Oct 2014, singh.janmejay wrote: >> > > > > > >> >> > > > > > >> Agree. >> > > > > > >>> >> > > > > > >>> I was not thinking of it like that. Will make the necessary >> > > > changes. >> > > > > > >>> >> > > > > > >>> Another question, do you want me to keep the template >> support >> > and >> > > > add >> > > > > > >>> variable support? >> > > > > > >>> >> > > > > > >>> or should I replace it completely with variable support(im >> > > thinking >> > > > > > >>> replacing makes more sense)? >> > > > > > >>> >> > > > > > >> >> > > > > > >> I think replacing makes more sense. Since we have the >> ability to >> > > > set a >> > > > > > >> variable equal to the output of a template, not having a >> > template >> > > in >> > > > > the >> > > > > > >> action doesn't prevent you from doing anything, you just >> have to >> > > do >> > > > it >> > > > > > in >> > > > > > >> two steps instead of one. >> > > > > > >> >> > > > > > >> >> > > > > > > Sounds good, will rework it in the next patch. >> > > > > > > >> > > > > > > >> > > > > > >> >> > > > > > >> David Lang >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >>> >> > > > > > >>> >> > > > > > >>> On Tue, Oct 14, 2014 at 8:53 AM, David Lang <[email protected]> >> > > wrote: >> > > > > > >>> >> > > > > > >>> On Tue, 14 Oct 2014, singh.janmejay wrote: >> > > > > > >>>> >> > > > > > >>>> Yep, sounds good. >> > > > > > >>>> >> > > > > > >>>>> >> > > > > > >>>>> The only concern I can think of: its different from how >> other >> > > > > actions >> > > > > > >>>>> work. >> > > > > > >>>>> >> > > > > > >>>>> But that shouldn't be a problem if we make other actions >> > accept >> > > > > > values >> > > > > > >>>>> too(and possibly deprecate usage of template in favour of >> > > > variable >> > > > > + >> > > > > > >>>>> exec_template). Something to think about, I guess. >> > > > > > >>>>> >> > > > > > >>>>> I'll make the change and send it back for review. >> > > > > > >>>>> >> > > > > > >>>>> >> > > > > > >>>> it's the difference between input and output. >> > > > > > >>>> >> > > > > > >>>> For other actions, the template is formatting the output of >> > the >> > > > > > module, >> > > > > > >>>> and you almost never are going to be sending just a single >> > > > variable >> > > > > as >> > > > > > >>>> the >> > > > > > >>>> output (I can't think of any case, other than possibly >> rawmsg) >> > > > > > >>>> >> > > > > > >>>> In this case we are defining the input to the message >> > > modification >> > > > > > >>>> module, >> > > > > > >>>> msg, rawmsg, or an already defined variable. >> > > > > > >>>> >> > > > > > >>>> viewing it this way, I think it makes perfect sense that >> it's >> > > not >> > > > a >> > > > > > >>>> template, but let's see what others think. >> > > > > > >>>> >> > > > > > >>>> David Lang >> > > > > > >>>> >> > > > > > >>>> >> > > > > > >>>> >> > > > > > >>>> On Tue, Oct 14, 2014 at 7:59 AM, David Lang < >> [email protected]> >> > > > wrote: >> > > > > > >>>>> >> > > > > > >>>>> As an efficiency thing, I would suggest having it get a >> > > variable >> > > > > > >>>>> instead >> > > > > > >>>>> >> > > > > > >>>>>> of a templage. You can assign a template to a variable if >> > you >> > > > need >> > > > > > to, >> > > > > > >>>>>> but >> > > > > > >>>>>> if you don't (because a prior step already parsed out the >> > data >> > > > to >> > > > > a >> > > > > > >>>>>> variable) you skip the time needed to go through the >> > template >> > > > > > engine. >> > > > > > >>>>>> >> > > > > > >>>>>> I expect that most of the time you are going to have the >> > part >> > > > you >> > > > > > >>>>>> want to >> > > > > > >>>>>> normalize as the contents of a variable being passed in >> via >> > > JSON >> > > > > or >> > > > > > a >> > > > > > >>>>>> structured data field, not as something that you would >> need >> > > the >> > > > > full >> > > > > > >>>>>> power >> > > > > > >>>>>> and complexity of the template string generator to get >> at. >> > > > > > >>>>>> >> > > > > > >>>>>> The template string handler is slower than you would >> think. >> > > When >> > > > > we >> > > > > > >>>>>> added >> > > > > > >>>>>> the ability to have string generation modules in C >> instead >> > of >> > > > > using >> > > > > > >>>>>> the >> > > > > > >>>>>> template engine, it made a noticable difference in the >> total >> > > > > > >>>>>> throughput >> > > > > > >>>>>> of >> > > > > > >>>>>> rsyslog, and if we are just going to be starting from a >> > string >> > > > to >> > > > > > >>>>>> begin >> > > > > > >>>>>> with, running it through the template engine is pure >> > overhead. >> > > > > > >>>>>> >> > > > > > >>>>>> Thanks for doing this sort of thing. I was wanting to get >> > > > > > mmnormalize >> > > > > > >>>>>> and >> > > > > > >>>>>> mmjsonparse to work on arbitrary strings instead of just >> > $msg, >> > > > but >> > > > > > >>>>>> hadn't >> > > > > > >>>>>> had time to deal with it. >> > > > > > >>>>>> >> > > > > > >>>>>> David Lang >> > > > > > >>>>>> >> > > > > > >>>>>> On Tue, 14 Oct 2014, singh.janmejay wrote: >> > > > > > >>>>>> >> > > > > > >>>>>> Date: Tue, 14 Oct 2014 01:03:44 +0530 >> > > > > > >>>>>> >> > > > > > >>>>>> From: singh.janmejay <[email protected]> >> > > > > > >>>>>>> Reply-To: rsyslog-users <[email protected]> >> > > > > > >>>>>>> To: [email protected] >> > > > > > >>>>>>> Subject: [rsyslog] Patch to allow choice of templates in >> > > > > > mmnormalize >> > > > > > >>>>>>> >> > > > > > >>>>>>> >> > > > > > >>>>>>> Hi, >> > > > > > >>>>>>> >> > > > > > >>>>>>> As of now mmnormalize supports only raw-msg or msg via >> the >> > > > > boolean >> > > > > > >>>>>>> flag >> > > > > > >>>>>>> 'useRawMsg'. This patch allows using >> > template="template_name" >> > > > > with >> > > > > > >>>>>>> mmnormalize just like a few other actions(such as omfwd, >> > > > > > >>>>>>> omelasticsearch >> > > > > > >>>>>>> etc) do. >> > > > > > >>>>>>> >> > > > > > >>>>>>> It expects a param called 'template' following the >> > convention >> > > > in >> > > > > > >>>>>>> other >> > > > > > >>>>>>> action-types. And when both useRawMsg and template >> params >> > are >> > > > > set, >> > > > > > it >> > > > > > >>>>>>> prints a warning that template will be ignored and uses >> > > > > > raw-message >> > > > > > >>>>>>> instead. >> > > > > > >>>>>>> >> > > > > > >>>>>>> The patch is called 0003... because its the 3rd patch >> in my >> > > > > > >>>>>>> local-repo. >> > > > > > >>>>>>> But >> > > > > > >>>>>>> since it doesn't have any files common with previous >> > patches, >> > > > it >> > > > > > >>>>>>> should >> > > > > > >>>>>>> apply just fine on any recent commit. >> > > > > > >>>>>>> >> > > > > > >>>>>>> Please let me know if the patch looks good for merging, >> > I'll >> > > > > update >> > > > > > >>>>>>> the >> > > > > > >>>>>>> documentation. >> > > > > > >>>>>>> >> > > > > > >>>>>>> >> > > > > > >>>>>>> _______________________________________________ >> > > > > > >>>>>>> >> > > > > > >>>>>>> rsyslog mailing list >> > > > > > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> > > > > > >>>>>> http://www.rsyslog.com/professional-services/ >> > > > > > >>>>>> What's up with rsyslog? Follow >> > https://twitter.com/rgerhards >> > > > > > >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are >> ARCHIVED >> > > by >> > > > a >> > > > > > >>>>>> myriad >> > > > > > >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO >> NOT >> > > POST >> > > > if >> > > > > > you >> > > > > > >>>>>> DON'T LIKE THAT. >> > > > > > >>>>>> >> > > > > > >>>>>> >> > > > > > >>>>>> >> > > > > > >>>>> >> > > > > > >>>>> >> > > > > > >>>>> _______________________________________________ >> > > > > > >>>>> >> > > > > > >>>> rsyslog mailing list >> > > > > > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> > > > > > >>>> http://www.rsyslog.com/professional-services/ >> > > > > > >>>> What's up with rsyslog? Follow >> https://twitter.com/rgerhards >> > > > > > >>>> NOTE WELL: This is a PUBLIC mailing list, posts are >> ARCHIVED >> > by >> > > a >> > > > > > myriad >> > > > > > >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >> > POST >> > > if >> > > > > you >> > > > > > >>>> DON'T LIKE THAT. >> > > > > > >>>> >> > > > > > >>>> >> > > > > > >>> >> > > > > > >>> >> > > > > > >>> >> > > > > > >>> _______________________________________________ >> > > > > > >> rsyslog mailing list >> > > > > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> > > > > > >> http://www.rsyslog.com/professional-services/ >> > > > > > >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> > > > > > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED >> by >> > a >> > > > > myriad >> > > > > > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >> POST >> > if >> > > > you >> > > > > > >> DON'T LIKE THAT. >> > > > > > >> >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > -- >> > > > > > > Regards, >> > > > > > > Janmejay >> > > > > > > http://codehunk.wordpress.com >> > > > > > > >> > > > > > >> > > > > > >> > > > > > Hi, >> > > > > > >> > > > > > Sorry for the loss the continuity. >> > > > > > >> > > > > > Here are the patches for mmnormalize variable support. >> > > > > > >> > > > > > I have left patch 0001-... as is (it is the same old patch which >> > > > enables >> > > > > > template support) >> > > > > > >> > > > > > The patch 0002-... follows that and replaces support for >> templates >> > > with >> > > > > > variable. >> > > > > > >> > > > > > My mmnormalize invocation now looks like this: >> > > > > > >> > > > > > action(type="mmnormalize" ruleBase="/tmp/foo.rulebase" >> > > > variable="$.foo") >> > > > > > >> > > > > > Variable can be any property or a local/global variable etc. It >> > uses >> > > > > > MsgGetProp, so it should support everything. >> > > > > > >> > > > > > -- >> > > > > > Regards, >> > > > > > Janmejay >> > > > > > http://codehunk.wordpress.com >> > > > > > >> > > > > > _______________________________________________ >> > > > > > rsyslog mailing list >> > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > > > > > http://www.rsyslog.com/professional-services/ >> > > > > > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > > > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by >> a >> > > > myriad >> > > > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST >> if >> > > you >> > > > > > DON'T LIKE THAT. >> > > > > > >> > > > > _______________________________________________ >> > > > > rsyslog mailing list >> > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > > > > http://www.rsyslog.com/professional-services/ >> > > > > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> > > myriad >> > > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> > you >> > > > > DON'T LIKE THAT. >> > > > > >> > > > _______________________________________________ >> > > > rsyslog mailing list >> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > > > http://www.rsyslog.com/professional-services/ >> > > > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> > myriad >> > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> you >> > > > DON'T LIKE THAT. >> > > > >> > > _______________________________________________ >> > > rsyslog mailing list >> > > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > > http://www.rsyslog.com/professional-services/ >> > > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad >> > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> > > DON'T LIKE THAT. >> > > >> > >> > >> > >> > -- >> > Regards, >> > Janmejay >> > http://codehunk.wordpress.com >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> > DON'T LIKE THAT. >> > >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > > > > -- > Regards, > Janmejay > http://codehunk.wordpress.com > -- Regards, Janmejay http://codehunk.wordpress.com
From c66f5f46d3ddac937b6bbea70ab7aa1947fa2c18 Mon Sep 17 00:00:00 2001 From: Janmejay Singh <[email protected]> Date: Fri, 31 Oct 2014 14:40:28 +0530 Subject: [PATCH] added a test for mmnormalize variable support --- tests/Makefile.am | 5 +++++ tests/diag.sh | 6 ++++++ tests/mmnormalize_variable.sh | 13 +++++++++++++ tests/testsuites/date_time_msg | 2 ++ tests/testsuites/mmnormalize_variable.rulebase | 1 + 5 files changed, 27 insertions(+) create mode 100755 tests/mmnormalize_variable.sh create mode 100644 tests/testsuites/date_time_msg create mode 100644 tests/testsuites/mmnormalize_variable.rulebase diff --git a/tests/Makefile.am b/tests/Makefile.am index b5d6f53..55ef2e0 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -133,6 +133,11 @@ TESTS += \ mmpstrucdata.sh endif +if ENABLE_MMNORMALIZE +TESTS += \ + mmnormalize_variable.sh +endif + if ENABLE_MMJSONPARSE TESTS += \ mmjsonparse_simple.sh \ diff --git a/tests/diag.sh b/tests/diag.sh index 1eae524..de887c9 100755 --- a/tests/diag.sh +++ b/tests/diag.sh @@ -165,6 +165,12 @@ case $1 in fi rm -f work2 ;; + 'content-check') + cat rsyslog.out.log | grep -qF "$2" + if [ "$?" -ne "0" ]; then + exit 1 + fi + ;; 'gzip-seq-check') # do the usual sequence check, but for gzip files rm -f work ls -l rsyslog.out.log diff --git a/tests/mmnormalize_variable.sh b/tests/mmnormalize_variable.sh new file mode 100755 index 0000000..c19ae50 --- /dev/null +++ b/tests/mmnormalize_variable.sh @@ -0,0 +1,13 @@ +# added 2014-10-31 by singh.janmejay +# This file is part of the rsyslog project, released under ASL 2.0 +echo =============================================================================== +echo \[mmnormalize_variable.sh\]: basic test for mmnormalize module variable-support +source $srcdir/diag.sh init +source $srcdir/diag.sh startup mmnormalize_variable.conf +./tcpflood -m 1 -I testsuites/date_time_msg +echo doing shutdown +source $srcdir/diag.sh shutdown-when-empty +echo wait on shutdown +source $srcdir/diag.sh wait-shutdown +source $srcdir/diag.sh content-check "h:13 m:20 s:18" +source $srcdir/diag.sh exit diff --git a/tests/testsuites/date_time_msg b/tests/testsuites/date_time_msg new file mode 100644 index 0000000..9b74b1b --- /dev/null +++ b/tests/testsuites/date_time_msg @@ -0,0 +1,2 @@ +<167>Mar 6 16:57:54 172.20.245.8 %PIX-7-710005: at Thu Oct 30 13:20:18 IST 2014 random number is 19597 + diff --git a/tests/testsuites/mmnormalize_variable.rulebase b/tests/testsuites/mmnormalize_variable.rulebase new file mode 100644 index 0000000..4d40d4c --- /dev/null +++ b/tests/testsuites/mmnormalize_variable.rulebase @@ -0,0 +1 @@ +rule=hms:%hr:number%:%min:number%:%sec:number% %zone:word% \ No newline at end of file -- 2.0.4
_______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
