Just to make sure I did get this straight: This means you always include one field only?
Rainer > -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Brian Knox > Sent: Monday, January 07, 2013 5:18 PM > To: rsyslog-users > Subject: Re: [rsyslog] working with list templates > > So - in my case I do not believe I need field names. Redis is more of > a > k/v store than a document store like mongo (although it supports more > data > structures than simply key / value, like sets, hashes, and lists). > > With the hiredis api there are basically two ways of passing a command > to > redis. For the sake of example, let's take the command I played in the > previous example, with would push the content of the msg property into > a > list in redis. > > Assuming "msg" contains the rsyslog message itself ("my message") > > 1) Using sprintf like formatting > > redisCommand(context, "LPUSH mylist %s", msg); > > 2) Passing an array of char arrays > > something like this: > > const char *argv[3]; > argv[0] = "LPUSH"; > argv[1] = "mylist"; > argv[2] = msg; > size_t lens[3] = { 5, 6, 10 }; > int argc = 3; > > redisFormatCommandArgv(&cmd,argc,argv,lens); > > The way my current omhiredis module works is to simply call > > rc = redisAppendCommand(pData->conn, (char*)message); > > This works fine for doing things like incrementing counters > > template(name="messages_processed" type="string" string="INCR processed > 1") > > However, if I include the actual log line, the spaces in the log line > break > things as then hiredis things the spaces are command delimiters. > > So, if I construct the command using a list template: > > template(name="redisPushQueue" type="list") { > constant(value="LPUSH") > constant(value="queue_key") > property(name="msg") > } > > So that from the plugin side, I can get "LPUSH", "queue_key", and the > message from the msg property, and their lengths - then I could easily > use the ArgV version of the hiredis command runners and formatters. > > Make sense? > > Brian > > > > > > > > On Mon, Jan 7, 2013 at 10:59 AM, Rainer Gerhards > <[email protected]>wrote: > > > > -----Original Message----- > > > From: [email protected] [mailto:rsyslog- > > > [email protected]] On Behalf Of Brian Knox > > > Sent: Monday, January 07, 2013 2:11 PM > > > To: rsyslog-users > > > Subject: Re: [rsyslog] working with list templates > > > > > > Happy Monday - getting ready to revisit the improvements to the > > > template > > > handling I'd like to make to omhiredis. To sum up the thread so > far: > > > > > > 1) I can register from the plugin how I would like to handle the > > > template > > > > For an example, look here: > > > > > http://git.adiscon.com/?p=rsyslog.git;a=blob;f=plugins/ommongodb/ommong > odb.c;h=dd997410743ab7ed1850029f368ecb29f4405449;hb=HEAD#l504 > > > > Note the OMSR_TPL_AS_... (JSON) in this case. > > > > > 2) I should probably use the JSON method as array will be > deprecated > > > > This is actually a good question. The original plan is to deprecate > them, > > HOWEVER, this is under review. Array mode *may* (may!) be a good > > alternative for more traditional (SQL) like output sources. However, > array > > mode lacks field names, so it all comes down to counting positions - > what > > in fact IMHO makes them unsuitable. My personal guess is that we will > > probably deprecate them. You'll see a related request for comment > here on > > this list in the not so distant future (I need to clearly describe > the need > > first). > > > > > > > > So my question is now - if I use a "list" style template > declaration, > > > what > > > sort of JSON object will I end up with? > > > > > > For a concrete example: > > > > > > template(name="redisPushQueue" type="list") { > > > constant(value="LPUSH") > > > constant(value="queue_key") > > > property(name="msg") > > > } > > > > > > And here's the function I'm passing to: > > > void redisAppendCommandArgv(redisContext *c, int argc, const char > > > **argv, const size_t *argvlen); > > > > > > So if I could pass this as essentially a JSON array ["LPUSH", > > > "queue_key", > > > "msg"] that would make life easy. > > > > Well, not sure about redis... don't you need field names? That's > actually > > one of the big questions in regard to relational databases. I think > they > > need field names in any case, so if they are not inside the template, > I > > need to pass them in via a different mechanism. Thus I want to have > them > > inside the template. > > > > With JSON passing what you guess is actually just one field, namely > "msg" > > with the given content. That is because the constants have no "name" > > parameter, and without a name, they cannot be part of a json object > (this > > is intentional, and can be useful to use the same template with > plugins > > that expect JSON and plugins that expect text). > > > > Rainer > > > > > > Brian > > > > > > > > > On Sat, Jan 5, 2013 at 3:56 PM, Brian Knox <[email protected]> > wrote: > > > > > > > Aha! Thanks for the info David - I wouldn't want to build on top > of > > > a > > > > feature that may be deprecated in the near future. > > > > > > > > Brian > > > > > > > > > > > > On Fri, Jan 4, 2013 at 8:30 PM, David Lang <[email protected]> wrote: > > > > > > > >> It's probably better to use JSON instead of array. Array is a > very > > > early > > > >> thing and there is some talk of phasing it out rather than > needing > > > to > > > >> support multiple ways of passing groups of messages. > > > >> > > > >> David Lang > > > >> > > > >> > > > >> On Fri, 4 Jan 2013, Brian Knox wrote: > > > >> > > > >> Aha! I think in this case array passing may be what I need. > > > Thanks! > > > >>> > > > >>> Brian > > > >>> On 1/4/2013 4:00 AM, Rainer Gerhards wrote: > > > >>> > > > >>>> Quick hint: you need to set the param passing mode inside the > > > plugin. > > > >>>> Not the template type specifies what you get, but the plugin > asks > > > the > > > >>>> engine. What you look for is JSON passing mode (or maybe array > > > passing). > > > >>>> Ommongodb should help you, else ping me early next week. > > > >>>> > > > >>>> Rainer > > > >>>> > > > >>>> -----Original Message----- > > > >>>>> From: [email protected].**com<rsyslog- > > > [email protected]>[mailto: > > > >>>>> rsyslog- > > > >>>>> [email protected]] On Behalf Of Brian Knox > > > >>>>> Sent: Thursday, January 03, 2013 2:22 PM > > > >>>>> To: [email protected] > > > >>>>> Subject: Re: [rsyslog] working with list templates > > > >>>>> > > > >>>>> Hi Radu! Thanks for the link, but I already know how to work > > > with list > > > >>>>> templates from a conf perspective. I'm looking for good > examples > > > of > > > >>>>> accessing that data from code in an output plugin. Currently > my > > > plugin > > > >>>>> simply does this in CODESTARTdoAction: > > > >>>>> > > > >>>>> CHKiRet(writeHiredis(ppString[**0], pData)); > > > >>>>> > > > >>>>> This passes the entire output from the template as a string > to > > > redis. > > > >>>>> This works ok when I want to construct redis commands from > > > templates > > > >>>>> using individual properties straight to hiredis's > > > redisAppendCommand > > > >>>>> function, such as: > > > >>>>> > > > >>>>> "HINCR progcount %programname% 1" > > > >>>>> > > > >>>>> But it does not work when I want to send whole messages to > redis > > > >>>>> (whitespace in the syslog message be interpreted as > delimiters in > > > the > > > >>>>> command, causing errors). > > > >>>>> > > > >>>>> hiredis has another function however, redisAppendCommandArgv, > > > that is > > > >>>>> variatic. In order to use this I would like to construct a > list > > > using > > > >>>>> the list templates, and then access each individual member of > the > > > list > > > >>>>> from the output plugin code so I can construct the array to > pass > > > to > > > >>>>> redisAppendCommandArgv. So, let's say I construct the > following > > > >>>>> template: > > > >>>>> > > > >>>>> template(name="redisPushQueue" type="list") { > > > >>>>> constant(value="LPUSH") > > > >>>>> constant(value="queue_key") > > > >>>>> property(name="msg") > > > >>>>> } > > > >>>>> > > > >>>>> From the output plugin code, how do I then access each > member > > > of this > > > >>>>> list individually with the new template code? > > > >>>>> > > > >>>>> Make sense? > > > >>>>> > > > >>>>> Brian > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> 1) Can > > > >>>>> On 1/3/2013 7:59 AM, Radu Gheorghe wrote: > > > >>>>> > > > >>>>>> Hello Brian, > > > >>>>>> > > > >>>>>> Just to make sure we're not missing the obvious: > > > >>>>>> > > > > http://www.rsyslog.com/doc/**rsyslog_conf_templates.html<http://www.rsy > > > slog.com/doc/rsyslog_conf_templates.html> > > > >>>>>> > > > >>>>>> Besides that, there are a couple of examples here: > > > >>>>>> > > > >>>>>> > > > >>>>>> 2013/1/3 Brian Knox <[email protected]> > > > >>>>>> > > > >>>>>> I've been working on some improvements for the omhiredis > output > > > >>>>>>> > > > >>>>>> plugin this > > > >>>>> > > > >>>>>> week. I think I've come up with a better way of combining > > > templates > > > >>>>>>> > > > >>>>>> with > > > >>>>> > > > >>>>>> redis command formatting. In order to do this, I need to, > given > > > a > > > >>>>>>> > > > >>>>>> list > > > >>>>> > > > >>>>>> style template, get the number of elements in the list and > the > > > >>>>>>> > > > >>>>>> length of > > > >>>>> > > > >>>>>> each element. > > > >>>>>>> > > > >>>>>>> I've only worked with string templates (from a plugin > > > perspective) > > > >>>>>>> > > > >>>>>> with my > > > >>>>> > > > >>>>>> output plugins so far. Is there a good example somewhere > for > > > >>>>>>> > > > >>>>>> dealing with > > > >>>>> > > > >>>>>> list templates? > > > >>>>>>> > > > >>>>>>> On other fronts, I now omhiredis working with batch dequeue > in > > > >>>>>>> > > > >>>>>> conjunction > > > >>>>> > > > >>>>>> with hiredis' pipeline commands. In local tests this let me > > > push > > > >>>>>>> > > > >>>>>> around > > > >>>>> > > > >>>>>> 250,000 redis commands per second to a redis instance (I was > > > getting > > > >>>>>>> > > > >>>>>> around > > > >>>>> > > > >>>>>> 50k commands of the same type with the same test before the > > > >>>>>>> > > > >>>>>> changes). > > > >>>>> > > > >>>>>> Rainer - I'm hoping to wrap up this next round of > improvements > > > in a > > > >>>>>>> > > > >>>>>> few > > > >>>>> > > > >>>>>> days and get a patch your way! > > > >>>>>>> > > > >>>>>>> Brian > > > >>>>>>> ______________________________**_________________ > > > >>>>>>> rsyslog mailing list > > > >>>>>>> > > > > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco > > > n.net/mailman/listinfo/rsyslog> > > > >>>>>>> http://www.rsyslog.com/**professional- > > > services/<http://www.rsyslog.com/professional-services/> > > > >>>>>>> What's up with rsyslog? Follow > https://twitter.com/rgerhards > > > >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are > ARCHIVED by > > > a > > > >>>>>>> > > > >>>>>> myriad > > > >>>>> > > > >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > POST > > > if > > > >>>>>>> > > > >>>>>> you > > > >>>>> > > > >>>>>> DON'T LIKE THAT. > > > >>>>>>> > > > >>>>>>> ______________________________**_________________ > > > >>>>>> rsyslog mailing list > > > >>>>>> > > > > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco > > > n.net/mailman/listinfo/rsyslog> > > > >>>>>> http://www.rsyslog.com/**professional- > > > services/<http://www.rsyslog.com/professional-services/> > > > >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > > > >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED > by > > > a > > > >>>>>> > > > >>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO > NOT > > > POST > > > >>>>> if you DON'T LIKE THAT. > > > >>>>> > > > >>>>> ______________________________**_________________ > > > >>>>> rsyslog mailing list > > > >>>>> > > > > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco > > > n.net/mailman/listinfo/rsyslog> > > > >>>>> http://www.rsyslog.com/**professional- > > > services/<http://www.rsyslog.com/professional-services/> > > > >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > > > >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED > by a > > > >>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO > NOT > > > POST > > > >>>>> if you DON'T LIKE THAT. > > > >>>>> > > > >>>> ______________________________**_________________ > > > >>>> rsyslog mailing list > > > >>>> > > > > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco > > > n.net/mailman/listinfo/rsyslog> > > > >>>> http://www.rsyslog.com/**professional- > > > services/<http://www.rsyslog.com/professional-services/> > > > >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > > > >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED > by a > > > >>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO > NOT > > > POST if > > > >>>> you DON'T LIKE THAT. > > > >>>> > > > >>> > > > >>> ______________________________**_________________ > > > >>> rsyslog mailing list > > > >>> > > > > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco > > > n.net/mailman/listinfo/rsyslog> > > > >>> http://www.rsyslog.com/**professional- > > > services/<http://www.rsyslog.com/professional-services/> > > > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > > > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by > a > > > myriad > > > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if > > > you > > > >>> DON'T LIKE THAT. > > > >>> > > > >>> ______________________________**_________________ > > > >> rsyslog mailing list > > > >> > > > > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco > > > n.net/mailman/listinfo/rsyslog> > > > >> http://www.rsyslog.com/**professional- > > > services/<http://www.rsyslog.com/professional-services/> > > > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > > > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by > a > > > myriad > > > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if > > > you > > > >> DON'T LIKE THAT. > > > >> > > > > > > > > > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com/professional-services/ > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > POST > > > if you DON'T LIKE THAT. > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
