The more I read, the more I think what you really want is a single string. Are we just missing a shell escape property option?
What do you think? Rainer Sent from phone, thus brief. -------- Ursprüngliche Nachricht -------- Von: Brian Knox <[email protected]> Datum: An: rsyslog-users <[email protected]> Betreff: Re: [rsyslog] working with list templates Hm. Interesting question. Let me think... I definitely will not always include one field only - say I wanted to do something in redis like create counts of how many messages I received, by hostname, for each programname. The redis command would be HINCR <programname> <hostname> so HINCR rsyslog myhost.mydomain.org Would create a hash at key "rsyslog" with a field of "myhost.mydomain.org" and increment a counter at that field by 1. The other thing I need to be able to do is make sure I can concatenate fields into a single value... Like if I wanted to make make a combined key of programname and hostname with a delimiter... "INCR rsyslog|myhost.mydomain.org" That case is pretty easy to handle with a string template though... "INCR %programname%|%hostname%" works fine in that case. Brian On 1/7/2013 12:34 PM, Rainer Gerhards wrote: > Just to make sure I did get this straight: This means you always include one > field only? > > Rainer > >> -----Original Message----- >> From: [email protected] [mailto:rsyslog- >> [email protected]] On Behalf Of Brian Knox >> Sent: Monday, January 07, 2013 5:18 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] working with list templates >> >> So - in my case I do not believe I need field names. Redis is more of >> a >> k/v store than a document store like mongo (although it supports more >> data >> structures than simply key / value, like sets, hashes, and lists). >> >> With the hiredis api there are basically two ways of passing a command >> to >> redis. For the sake of example, let's take the command I played in the >> previous example, with would push the content of the msg property into >> a >> list in redis. >> >> Assuming "msg" contains the rsyslog message itself ("my message") >> >> 1) Using sprintf like formatting >> >> redisCommand(context, "LPUSH mylist %s", msg); >> >> 2) Passing an array of char arrays >> >> something like this: >> >> const char *argv[3]; >> argv[0] = "LPUSH"; >> argv[1] = "mylist"; >> argv[2] = msg; >> size_t lens[3] = { 5, 6, 10 }; >> int argc = 3; >> >> redisFormatCommandArgv(&cmd,argc,argv,lens); >> >> The way my current omhiredis module works is to simply call >> >> rc = redisAppendCommand(pData->conn, (char*)message); >> >> This works fine for doing things like incrementing counters >> >> template(name="messages_processed" type="string" string="INCR processed >> 1") >> >> However, if I include the actual log line, the spaces in the log line >> break >> things as then hiredis things the spaces are command delimiters. >> >> So, if I construct the command using a list template: >> >> template(name="redisPushQueue" type="list") { >> constant(value="LPUSH") >> constant(value="queue_key") >> property(name="msg") >> } >> >> So that from the plugin side, I can get "LPUSH", "queue_key", and the >> message from the msg property, and their lengths - then I could easily >> use the ArgV version of the hiredis command runners and formatters. >> >> Make sense? >> >> Brian >> >> >> >> >> >> >> >> On Mon, Jan 7, 2013 at 10:59 AM, Rainer Gerhards >> <[email protected]>wrote: >> >>>> -----Original Message----- >>>> From: [email protected] [mailto:rsyslog- >>>> [email protected]] On Behalf Of Brian Knox >>>> Sent: Monday, January 07, 2013 2:11 PM >>>> To: rsyslog-users >>>> Subject: Re: [rsyslog] working with list templates >>>> >>>> Happy Monday - getting ready to revisit the improvements to the >>>> template >>>> handling I'd like to make to omhiredis. To sum up the thread so >> far: >>>> 1) I can register from the plugin how I would like to handle the >>>> template >>> For an example, look here: >>> >>> >> http://git.adiscon.com/?p=rsyslog.git;a=blob;f=plugins/ommongodb/ommong >> odb.c;h=dd997410743ab7ed1850029f368ecb29f4405449;hb=HEAD#l504 >>> Note the OMSR_TPL_AS_... (JSON) in this case. >>> >>>> 2) I should probably use the JSON method as array will be >> deprecated >>> This is actually a good question. The original plan is to deprecate >> them, >>> HOWEVER, this is under review. Array mode *may* (may!) be a good >>> alternative for more traditional (SQL) like output sources. However, >> array >>> mode lacks field names, so it all comes down to counting positions - >> what >>> in fact IMHO makes them unsuitable. My personal guess is that we will >>> probably deprecate them. You'll see a related request for comment >> here on >>> this list in the not so distant future (I need to clearly describe >> the need >>> first). >>> >>>> So my question is now - if I use a "list" style template >> declaration, >>>> what >>>> sort of JSON object will I end up with? >>>> >>>> For a concrete example: >>>> >>>> template(name="redisPushQueue" type="list") { >>>> constant(value="LPUSH") >>>> constant(value="queue_key") >>>> property(name="msg") >>>> } >>>> >>>> And here's the function I'm passing to: >>>> void redisAppendCommandArgv(redisContext *c, int argc, const char >>>> **argv, const size_t *argvlen); >>>> >>>> So if I could pass this as essentially a JSON array ["LPUSH", >>>> "queue_key", >>>> "msg"] that would make life easy. >>> Well, not sure about redis... don't you need field names? That's >> actually >>> one of the big questions in regard to relational databases. I think >> they >>> need field names in any case, so if they are not inside the template, >> I >>> need to pass them in via a different mechanism. Thus I want to have >> them >>> inside the template. >>> >>> With JSON passing what you guess is actually just one field, namely >> "msg" >>> with the given content. That is because the constants have no "name" >>> parameter, and without a name, they cannot be part of a json object >> (this >>> is intentional, and can be useful to use the same template with >> plugins >>> that expect JSON and plugins that expect text). >>> >>> Rainer >>>> Brian >>>> >>>> >>>> On Sat, Jan 5, 2013 at 3:56 PM, Brian Knox <[email protected]> >> wrote: >>>>> Aha! Thanks for the info David - I wouldn't want to build on top >> of >>>> a >>>>> feature that may be deprecated in the near future. >>>>> >>>>> Brian >>>>> >>>>> >>>>> On Fri, Jan 4, 2013 at 8:30 PM, David Lang <[email protected]> wrote: >>>>> >>>>>> It's probably better to use JSON instead of array. Array is a >> very >>>> early >>>>>> thing and there is some talk of phasing it out rather than >> needing >>>> to >>>>>> support multiple ways of passing groups of messages. >>>>>> >>>>>> David Lang >>>>>> >>>>>> >>>>>> On Fri, 4 Jan 2013, Brian Knox wrote: >>>>>> >>>>>> Aha! I think in this case array passing may be what I need. >>>> Thanks! >>>>>>> Brian >>>>>>> On 1/4/2013 4:00 AM, Rainer Gerhards wrote: >>>>>>> >>>>>>>> Quick hint: you need to set the param passing mode inside the >>>> plugin. >>>>>>>> Not the template type specifies what you get, but the plugin >> asks >>>> the >>>>>>>> engine. What you look for is JSON passing mode (or maybe array >>>> passing). >>>>>>>> Ommongodb should help you, else ping me early next week. >>>>>>>> >>>>>>>> Rainer >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>>> From: [email protected].**com<rsyslog- >>>> [email protected]>[mailto: >>>>>>>>> rsyslog- >>>>>>>>> [email protected]] On Behalf Of Brian Knox >>>>>>>>> Sent: Thursday, January 03, 2013 2:22 PM >>>>>>>>> To: [email protected] >>>>>>>>> Subject: Re: [rsyslog] working with list templates >>>>>>>>> >>>>>>>>> Hi Radu! Thanks for the link, but I already know how to work >>>> with list >>>>>>>>> templates from a conf perspective. I'm looking for good >> examples >>>> of >>>>>>>>> accessing that data from code in an output plugin. Currently >> my >>>> plugin >>>>>>>>> simply does this in CODESTARTdoAction: >>>>>>>>> >>>>>>>>> CHKiRet(writeHiredis(ppString[**0], pData)); >>>>>>>>> >>>>>>>>> This passes the entire output from the template as a string >> to >>>> redis. >>>>>>>>> This works ok when I want to construct redis commands from >>>> templates >>>>>>>>> using individual properties straight to hiredis's >>>> redisAppendCommand >>>>>>>>> function, such as: >>>>>>>>> >>>>>>>>> "HINCR progcount %programname% 1" >>>>>>>>> >>>>>>>>> But it does not work when I want to send whole messages to >> redis >>>>>>>>> (whitespace in the syslog message be interpreted as >> delimiters in >>>> the >>>>>>>>> command, causing errors). >>>>>>>>> >>>>>>>>> hiredis has another function however, redisAppendCommandArgv, >>>> that is >>>>>>>>> variatic. In order to use this I would like to construct a >> list >>>> using >>>>>>>>> the list templates, and then access each individual member of >> the >>>> list >>>>>>>>> from the output plugin code so I can construct the array to >> pass >>>> to >>>>>>>>> redisAppendCommandArgv. So, let's say I construct the >> following >>>>>>>>> template: >>>>>>>>> >>>>>>>>> template(name="redisPushQueue" type="list") { >>>>>>>>> constant(value="LPUSH") >>>>>>>>> constant(value="queue_key") >>>>>>>>> property(name="msg") >>>>>>>>> } >>>>>>>>> >>>>>>>>> From the output plugin code, how do I then access each >> member >>>> of this >>>>>>>>> list individually with the new template code? >>>>>>>>> >>>>>>>>> Make sense? >>>>>>>>> >>>>>>>>> Brian >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> 1) Can >>>>>>>>> On 1/3/2013 7:59 AM, Radu Gheorghe wrote: >>>>>>>>> >>>>>>>>>> Hello Brian, >>>>>>>>>> >>>>>>>>>> Just to make sure we're not missing the obvious: >>>>>>>>>> >> http://www.rsyslog.com/doc/**rsyslog_conf_templates.html<http://www.rsy >>>> slog.com/doc/rsyslog_conf_templates.html> >>>>>>>>>> Besides that, there are a couple of examples here: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2013/1/3 Brian Knox <[email protected]> >>>>>>>>>> >>>>>>>>>> I've been working on some improvements for the omhiredis >> output >>>>>>>>>> plugin this >>>>>>>>>> week. I think I've come up with a better way of combining >>>> templates >>>>>>>>>> with >>>>>>>>>> redis command formatting. In order to do this, I need to, >> given >>>> a >>>>>>>>>> list >>>>>>>>>> style template, get the number of elements in the list and >> the >>>>>>>>>> length of >>>>>>>>>> each element. >>>>>>>>>>> I've only worked with string templates (from a plugin >>>> perspective) >>>>>>>>>> with my >>>>>>>>>> output plugins so far. Is there a good example somewhere >> for >>>>>>>>>> dealing with >>>>>>>>>> list templates? >>>>>>>>>>> On other fronts, I now omhiredis working with batch dequeue >> in >>>>>>>>>> conjunction >>>>>>>>>> with hiredis' pipeline commands. In local tests this let me >>>> push >>>>>>>>>> around >>>>>>>>>> 250,000 redis commands per second to a redis instance (I was >>>> getting >>>>>>>>>> around >>>>>>>>>> 50k commands of the same type with the same test before the >>>>>>>>>> changes). >>>>>>>>>> Rainer - I'm hoping to wrap up this next round of >> improvements >>>> in a >>>>>>>>>> few >>>>>>>>>> days and get a patch your way! >>>>>>>>>>> Brian >>>>>>>>>>> ______________________________**_________________ >>>>>>>>>>> rsyslog mailing list >>>>>>>>>>> >> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco >>>> n.net/mailman/listinfo/rsyslog> >>>>>>>>>>> http://www.rsyslog.com/**professional- >>>> services/<http://www.rsyslog.com/professional-services/> >>>>>>>>>>> What's up with rsyslog? Follow >> https://twitter.com/rgerhards >>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are >> ARCHIVED by >>>> a >>>>>>>>>> myriad >>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >> POST >>>> if >>>>>>>>>> you >>>>>>>>>> DON'T LIKE THAT. >>>>>>>>>>> ______________________________**_________________ >>>>>>>>>> rsyslog mailing list >>>>>>>>>> >> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco >>>> n.net/mailman/listinfo/rsyslog> >>>>>>>>>> http://www.rsyslog.com/**professional- >>>> services/<http://www.rsyslog.com/professional-services/> >>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED >> by >>>> a >>>>>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO >> NOT >>>> POST >>>>>>>>> if you DON'T LIKE THAT. >>>>>>>>> >>>>>>>>> ______________________________**_________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> >> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco >>>> n.net/mailman/listinfo/rsyslog> >>>>>>>>> http://www.rsyslog.com/**professional- >>>> services/<http://www.rsyslog.com/professional-services/> >>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED >> by a >>>>>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO >> NOT >>>> POST >>>>>>>>> if you DON'T LIKE THAT. >>>>>>>>> >>>>>>>> ______________________________**_________________ >>>>>>>> rsyslog mailing list >>>>>>>> >> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco >>>> n.net/mailman/listinfo/rsyslog> >>>>>>>> http://www.rsyslog.com/**professional- >>>> services/<http://www.rsyslog.com/professional-services/> >>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED >> by a >>>>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO >> NOT >>>> POST if >>>>>>>> you DON'T LIKE THAT. >>>>>>>> >>>>>>> ______________________________**_________________ >>>>>>> rsyslog mailing list >>>>>>> >> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco >>>> n.net/mailman/listinfo/rsyslog> >>>>>>> http://www.rsyslog.com/**professional- >>>> services/<http://www.rsyslog.com/professional-services/> >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by >> a >>>> myriad >>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST >> if >>>> you >>>>>>> DON'T LIKE THAT. >>>>>>> >>>>>>> ______________________________**_________________ >>>>>> rsyslog mailing list >>>>>> >> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco >>>> n.net/mailman/listinfo/rsyslog> >>>>>> http://www.rsyslog.com/**professional- >>>> services/<http://www.rsyslog.com/professional-services/> >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by >> a >>>> myriad >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST >> if >>>> you >>>>>> DON'T LIKE THAT. >>>>>> >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >> POST >>>> if you DON'T LIKE THAT. >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> you >>> DON'T LIKE THAT. >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST >> if you DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
