On Tue, Oct 14, 2014 at 10:06 AM, singh.janmejay <[email protected]> wrote:
> > > On Tue, Oct 14, 2014 at 9:23 AM, David Lang <[email protected]> wrote: > >> On Tue, 14 Oct 2014, singh.janmejay wrote: >> >> Agree. >>> >>> I was not thinking of it like that. Will make the necessary changes. >>> >>> Another question, do you want me to keep the template support and add >>> variable support? >>> >>> or should I replace it completely with variable support(im thinking >>> replacing makes more sense)? >>> >> >> I think replacing makes more sense. Since we have the ability to set a >> variable equal to the output of a template, not having a template in the >> action doesn't prevent you from doing anything, you just have to do it in >> two steps instead of one. >> >> > Sounds good, will rework it in the next patch. > > >> >> David Lang >> >> >> >>> >>> >>> On Tue, Oct 14, 2014 at 8:53 AM, David Lang <[email protected]> wrote: >>> >>> On Tue, 14 Oct 2014, singh.janmejay wrote: >>>> >>>> Yep, sounds good. >>>> >>>>> >>>>> The only concern I can think of: its different from how other actions >>>>> work. >>>>> >>>>> But that shouldn't be a problem if we make other actions accept values >>>>> too(and possibly deprecate usage of template in favour of variable + >>>>> exec_template). Something to think about, I guess. >>>>> >>>>> I'll make the change and send it back for review. >>>>> >>>>> >>>> it's the difference between input and output. >>>> >>>> For other actions, the template is formatting the output of the module, >>>> and you almost never are going to be sending just a single variable as >>>> the >>>> output (I can't think of any case, other than possibly rawmsg) >>>> >>>> In this case we are defining the input to the message modification >>>> module, >>>> msg, rawmsg, or an already defined variable. >>>> >>>> viewing it this way, I think it makes perfect sense that it's not a >>>> template, but let's see what others think. >>>> >>>> David Lang >>>> >>>> >>>> >>>> On Tue, Oct 14, 2014 at 7:59 AM, David Lang <[email protected]> wrote: >>>>> >>>>> As an efficiency thing, I would suggest having it get a variable >>>>> instead >>>>> >>>>>> of a templage. You can assign a template to a variable if you need to, >>>>>> but >>>>>> if you don't (because a prior step already parsed out the data to a >>>>>> variable) you skip the time needed to go through the template engine. >>>>>> >>>>>> I expect that most of the time you are going to have the part you >>>>>> want to >>>>>> normalize as the contents of a variable being passed in via JSON or a >>>>>> structured data field, not as something that you would need the full >>>>>> power >>>>>> and complexity of the template string generator to get at. >>>>>> >>>>>> The template string handler is slower than you would think. When we >>>>>> added >>>>>> the ability to have string generation modules in C instead of using >>>>>> the >>>>>> template engine, it made a noticable difference in the total >>>>>> throughput >>>>>> of >>>>>> rsyslog, and if we are just going to be starting from a string to >>>>>> begin >>>>>> with, running it through the template engine is pure overhead. >>>>>> >>>>>> Thanks for doing this sort of thing. I was wanting to get mmnormalize >>>>>> and >>>>>> mmjsonparse to work on arbitrary strings instead of just $msg, but >>>>>> hadn't >>>>>> had time to deal with it. >>>>>> >>>>>> David Lang >>>>>> >>>>>> On Tue, 14 Oct 2014, singh.janmejay wrote: >>>>>> >>>>>> Date: Tue, 14 Oct 2014 01:03:44 +0530 >>>>>> >>>>>> From: singh.janmejay <[email protected]> >>>>>>> Reply-To: rsyslog-users <[email protected]> >>>>>>> To: [email protected] >>>>>>> Subject: [rsyslog] Patch to allow choice of templates in mmnormalize >>>>>>> >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> As of now mmnormalize supports only raw-msg or msg via the boolean >>>>>>> flag >>>>>>> 'useRawMsg'. This patch allows using template="template_name" with >>>>>>> mmnormalize just like a few other actions(such as omfwd, >>>>>>> omelasticsearch >>>>>>> etc) do. >>>>>>> >>>>>>> It expects a param called 'template' following the convention in >>>>>>> other >>>>>>> action-types. And when both useRawMsg and template params are set, it >>>>>>> prints a warning that template will be ignored and uses raw-message >>>>>>> instead. >>>>>>> >>>>>>> The patch is called 0003... because its the 3rd patch in my >>>>>>> local-repo. >>>>>>> But >>>>>>> since it doesn't have any files common with previous patches, it >>>>>>> should >>>>>>> apply just fine on any recent commit. >>>>>>> >>>>>>> Please let me know if the patch looks good for merging, I'll update >>>>>>> the >>>>>>> documentation. >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> >>>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com/professional-services/ >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>> myriad >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>>> DON'T LIKE THAT. >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> DON'T LIKE THAT. >>>> >>>> >>> >>> >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > > > > -- > Regards, > Janmejay > http://codehunk.wordpress.com > Hi, Sorry for the loss the continuity. Here are the patches for mmnormalize variable support. I have left patch 0001-... as is (it is the same old patch which enables template support) The patch 0002-... follows that and replaces support for templates with variable. My mmnormalize invocation now looks like this: action(type="mmnormalize" ruleBase="/tmp/foo.rulebase" variable="$.foo") Variable can be any property or a local/global variable etc. It uses MsgGetProp, so it should support everything. -- Regards, Janmejay http://codehunk.wordpress.com
From 3e99cb46aca4ed6523f8dceb3d66ca88dec49c5c Mon Sep 17 00:00:00 2001 From: Janmejay Singh <[email protected]> Date: Mon, 13 Oct 2014 22:29:39 +0530 Subject: [PATCH 1/2] adds the choice of template for mmnormalize action --- plugins/mmnormalize/mmnormalize.c | 35 ++++++++++++++++++++++++++++++----- 1 file changed, 30 insertions(+), 5 deletions(-) diff --git a/plugins/mmnormalize/mmnormalize.c b/plugins/mmnormalize/mmnormalize.c index ba2e730..039c5d0 100644 --- a/plugins/mmnormalize/mmnormalize.c +++ b/plugins/mmnormalize/mmnormalize.c @@ -45,6 +45,7 @@ #include "errmsg.h" #include "cfsysline.h" #include "dirty.h" +#include "unicode-helper.h" MODULE_TYPE_OUTPUT MODULE_TYPE_NOKEEP @@ -64,6 +65,7 @@ typedef struct _instanceData { uchar *rulebase; /**< name of rulebase to use */ ln_ctx ctxln; /**< context to be used for liblognorm */ char *pszPath; /**< path of normalized data */ + uchar *tmplName; /**< name of template to use */ } instanceData; typedef struct wrkrInstanceData { @@ -81,7 +83,8 @@ static configSettings_t cs; static struct cnfparamdescr actpdescr[] = { { "rulebase", eCmdHdlrGetWord, 1 }, { "path", eCmdHdlrGetWord, 0 }, - { "userawmsg", eCmdHdlrBinary, 0 } + { "userawmsg", eCmdHdlrBinary, 0 }, + { "template", eCmdHdlrGetWord, 0 } }; static struct cnfparamblk actpblk = { CNFPARAMBLK_VERSION, @@ -174,7 +177,8 @@ BEGINfreeInstance CODESTARTfreeInstance free(pData->rulebase); ln_exitCtx(pData->ctxln); - free(pData->pszPath); + free(pData->pszPath); + free(pData->tmplName); ENDfreeInstance @@ -186,6 +190,10 @@ ENDfreeWrkrInstance BEGINdbgPrintInstInfo CODESTARTdbgPrintInstInfo dbgprintf("mmnormalize\n"); + dbgprintf("\ttemplate='%s'\n", pData->tmplName); + dbgprintf("\trulebase='%s'\n", pData->rulebase); + dbgprintf("\tpath='%s'\n", pData->pszPath); + dbgprintf("\tbUseRawMsg='%d'\n", pData->bUseRawMsg); ENDdbgPrintInstInfo @@ -203,7 +211,10 @@ CODESTARTdoAction pMsg = (msg_t*) ppString[0]; if(pWrkrData->pData->bUseRawMsg) { getRawMsg(pMsg, &buf, &len); - } else { + } else if (pWrkrData->pData->tmplName) { + buf = ppString[1]; + len = ustrlen(buf); + } else { buf = getMSG(pMsg); len = getMSGLen(pMsg); } @@ -260,6 +271,8 @@ CODESTARTnewActInst pData->rulebase = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(actpblk.descr[i].name, "userawmsg")) { pData->bUseRawMsg = (int) pvals[i].val.d.n; + } else if(!strcmp(actpblk.descr[i].name, "template")) { + pData->tmplName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(actpblk.descr[i].name, "path")) { cstr = es_str2cstr(pvals[i].val.d.estr, NULL); if (strlen(cstr) < 2) { @@ -282,8 +295,20 @@ CODESTARTnewActInst "param '%s'\n", actpblk.descr[i].name); } } - CODE_STD_STRING_REQUESTnewActInst(1) + + if(pData->bUseRawMsg) { + errmsg.LogError(0, RS_RET_CONFIG_ERROR, + "mmnormalize: 'template' param can't be used with 'useRawMsg'. " + "Ignoring 'template', will use raw message."); + free(pData->tmplName); + pData->tmplName = NULL; + } + + CODE_STD_STRING_REQUESTnewActInst(pData->tmplName ? 2 : 1) CHKiRet(OMSRsetEntry(*ppOMSR, 0, NULL, OMSR_TPL_AS_MSG)); + if (pData->tmplName) { + CHKiRet(OMSRsetEntry(*ppOMSR, 1, ustrdup(pData->tmplName), OMSR_NO_RQD_TPL_OPTS)); + } iRet = buildInstance(pData); CODE_STD_FINALIZERnewActInst @@ -369,7 +394,7 @@ BEGINmodInit() rsRetVal localRet; rsRetVal (*pomsrGetSupportedTplOpts)(unsigned long *pOpts); unsigned long opts; - int bMsgPassingSupported; + int bMsgPassingSupported; CODESTARTmodInit INITLegCnfVars *ipIFVersProvided = CURR_MOD_IF_VERSION; -- 2.0.4
From ca6a9665719574ccf0444553f2abbdff373cc159 Mon Sep 17 00:00:00 2001 From: Janmejay Singh <[email protected]> Date: Tue, 21 Oct 2014 12:06:39 +0530 Subject: [PATCH 2/2] mmnormalize action now takes variable instead of template --- plugins/mmnormalize/mmnormalize.c | 48 ++++++++++++++++++++++----------------- 1 file changed, 27 insertions(+), 21 deletions(-) diff --git a/plugins/mmnormalize/mmnormalize.c b/plugins/mmnormalize/mmnormalize.c index 039c5d0..4b188fb 100644 --- a/plugins/mmnormalize/mmnormalize.c +++ b/plugins/mmnormalize/mmnormalize.c @@ -65,7 +65,7 @@ typedef struct _instanceData { uchar *rulebase; /**< name of rulebase to use */ ln_ctx ctxln; /**< context to be used for liblognorm */ char *pszPath; /**< path of normalized data */ - uchar *tmplName; /**< name of template to use */ + msgPropDescr_t *varDescr; /**< name of variable to use */ } instanceData; typedef struct wrkrInstanceData { @@ -84,7 +84,7 @@ static struct cnfparamdescr actpdescr[] = { { "rulebase", eCmdHdlrGetWord, 1 }, { "path", eCmdHdlrGetWord, 0 }, { "userawmsg", eCmdHdlrBinary, 0 }, - { "template", eCmdHdlrGetWord, 0 } + { "variable", eCmdHdlrGetWord, 0 } }; static struct cnfparamblk actpblk = { CNFPARAMBLK_VERSION, @@ -178,7 +178,8 @@ CODESTARTfreeInstance free(pData->rulebase); ln_exitCtx(pData->ctxln); free(pData->pszPath); - free(pData->tmplName); + msgPropDescrDestruct(pData->varDescr); + free(pData->varDescr); ENDfreeInstance @@ -190,7 +191,7 @@ ENDfreeWrkrInstance BEGINdbgPrintInstInfo CODESTARTdbgPrintInstInfo dbgprintf("mmnormalize\n"); - dbgprintf("\ttemplate='%s'\n", pData->tmplName); + dbgprintf("\tvariable='%s'\n", pData->varDescr->name); dbgprintf("\trulebase='%s'\n", pData->rulebase); dbgprintf("\tpath='%s'\n", pData->pszPath); dbgprintf("\tbUseRawMsg='%d'\n", pData->bUseRawMsg); @@ -204,21 +205,25 @@ ENDtryResume BEGINdoAction msg_t *pMsg; uchar *buf; - int len; + rs_size_t len; int r; struct json_object *json = NULL; + unsigned short freeBuf = 0; CODESTARTdoAction pMsg = (msg_t*) ppString[0]; if(pWrkrData->pData->bUseRawMsg) { getRawMsg(pMsg, &buf, &len); - } else if (pWrkrData->pData->tmplName) { - buf = ppString[1]; - len = ustrlen(buf); + } else if (pWrkrData->pData->varDescr) { + buf = MsgGetProp(pMsg, NULL, pWrkrData->pData->varDescr, &len, &freeBuf, NULL); } else { buf = getMSG(pMsg); len = getMSGLen(pMsg); } r = ln_normalize(pWrkrData->pData->ctxln, (char*)buf, len, &json); + if (freeBuf) { + free(buf); + buf = NULL; + } if(r != 0) { DBGPRINTF("error %d during ln_normalize\n", r); MsgSetParseSuccess(pMsg, 0); @@ -237,15 +242,17 @@ setInstParamDefaults(instanceData *pData) pData->rulebase = NULL; pData->bUseRawMsg = 0; pData->pszPath = strdup("$!"); + pData->varDescr = NULL; } BEGINnewActInst struct cnfparamvals *pvals; int i; int bDestructPValsOnExit; - char *cstr; + char *cstr; + char *varName = NULL; CODESTARTnewActInst - DBGPRINTF("newActInst (mmnormalize)\n"); + DBGPRINTF("newActInst (mmnormalize)\n"); bDestructPValsOnExit = 0; pvals = nvlstGetParams(lst, &actpblk, NULL); @@ -271,8 +278,8 @@ CODESTARTnewActInst pData->rulebase = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(actpblk.descr[i].name, "userawmsg")) { pData->bUseRawMsg = (int) pvals[i].val.d.n; - } else if(!strcmp(actpblk.descr[i].name, "template")) { - pData->tmplName = (uchar*)es_str2cstr(pvals[i].val.d.estr, NULL); + } else if(!strcmp(actpblk.descr[i].name, "variable")) { + varName = es_str2cstr(pvals[i].val.d.estr, NULL); } else if(!strcmp(actpblk.descr[i].name, "path")) { cstr = es_str2cstr(pvals[i].val.d.estr, NULL); if (strlen(cstr) < 2) { @@ -298,18 +305,17 @@ CODESTARTnewActInst if(pData->bUseRawMsg) { errmsg.LogError(0, RS_RET_CONFIG_ERROR, - "mmnormalize: 'template' param can't be used with 'useRawMsg'. " - "Ignoring 'template', will use raw message."); - free(pData->tmplName); - pData->tmplName = NULL; + "mmnormalize: 'variable' param can't be used with 'useRawMsg'. " + "Ignoring 'variable', will use raw message."); + } else if (varName) { + CHKmalloc(pData->varDescr = MALLOC(sizeof(msgPropDescr_t))); + CHKiRet(msgPropDescrFill(pData->varDescr, (uchar*) varName, strlen(varName))); } + free(varName); + varName = NULL; - CODE_STD_STRING_REQUESTnewActInst(pData->tmplName ? 2 : 1) + CODE_STD_STRING_REQUESTnewActInst(1) CHKiRet(OMSRsetEntry(*ppOMSR, 0, NULL, OMSR_TPL_AS_MSG)); - if (pData->tmplName) { - CHKiRet(OMSRsetEntry(*ppOMSR, 1, ustrdup(pData->tmplName), OMSR_NO_RQD_TPL_OPTS)); - } - iRet = buildInstance(pData); CODE_STD_FINALIZERnewActInst if(bDestructPValsOnExit) -- 2.0.4
_______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
