> -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Brian Knox > Sent: Monday, January 07, 2013 2:11 PM > To: rsyslog-users > Subject: Re: [rsyslog] working with list templates > > Happy Monday - getting ready to revisit the improvements to the > template > handling I'd like to make to omhiredis. To sum up the thread so far: > > 1) I can register from the plugin how I would like to handle the > template
For an example, look here: http://git.adiscon.com/?p=rsyslog.git;a=blob;f=plugins/ommongodb/ommongodb.c;h=dd997410743ab7ed1850029f368ecb29f4405449;hb=HEAD#l504 Note the OMSR_TPL_AS_... (JSON) in this case. > 2) I should probably use the JSON method as array will be deprecated This is actually a good question. The original plan is to deprecate them, HOWEVER, this is under review. Array mode *may* (may!) be a good alternative for more traditional (SQL) like output sources. However, array mode lacks field names, so it all comes down to counting positions - what in fact IMHO makes them unsuitable. My personal guess is that we will probably deprecate them. You'll see a related request for comment here on this list in the not so distant future (I need to clearly describe the need first). > > So my question is now - if I use a "list" style template declaration, > what > sort of JSON object will I end up with? > > For a concrete example: > > template(name="redisPushQueue" type="list") { > constant(value="LPUSH") > constant(value="queue_key") > property(name="msg") > } > > And here's the function I'm passing to: > void redisAppendCommandArgv(redisContext *c, int argc, const char > **argv, const size_t *argvlen); > > So if I could pass this as essentially a JSON array ["LPUSH", > "queue_key", > "msg"] that would make life easy. Well, not sure about redis... don't you need field names? That's actually one of the big questions in regard to relational databases. I think they need field names in any case, so if they are not inside the template, I need to pass them in via a different mechanism. Thus I want to have them inside the template. With JSON passing what you guess is actually just one field, namely "msg" with the given content. That is because the constants have no "name" parameter, and without a name, they cannot be part of a json object (this is intentional, and can be useful to use the same template with plugins that expect JSON and plugins that expect text). Rainer > > Brian > > > On Sat, Jan 5, 2013 at 3:56 PM, Brian Knox <[email protected]> wrote: > > > Aha! Thanks for the info David - I wouldn't want to build on top of > a > > feature that may be deprecated in the near future. > > > > Brian > > > > > > On Fri, Jan 4, 2013 at 8:30 PM, David Lang <[email protected]> wrote: > > > >> It's probably better to use JSON instead of array. Array is a very > early > >> thing and there is some talk of phasing it out rather than needing > to > >> support multiple ways of passing groups of messages. > >> > >> David Lang > >> > >> > >> On Fri, 4 Jan 2013, Brian Knox wrote: > >> > >> Aha! I think in this case array passing may be what I need. > Thanks! > >>> > >>> Brian > >>> On 1/4/2013 4:00 AM, Rainer Gerhards wrote: > >>> > >>>> Quick hint: you need to set the param passing mode inside the > plugin. > >>>> Not the template type specifies what you get, but the plugin asks > the > >>>> engine. What you look for is JSON passing mode (or maybe array > passing). > >>>> Ommongodb should help you, else ping me early next week. > >>>> > >>>> Rainer > >>>> > >>>> -----Original Message----- > >>>>> From: [email protected].**com<rsyslog- > [email protected]>[mailto: > >>>>> rsyslog- > >>>>> [email protected]] On Behalf Of Brian Knox > >>>>> Sent: Thursday, January 03, 2013 2:22 PM > >>>>> To: [email protected] > >>>>> Subject: Re: [rsyslog] working with list templates > >>>>> > >>>>> Hi Radu! Thanks for the link, but I already know how to work > with list > >>>>> templates from a conf perspective. I'm looking for good examples > of > >>>>> accessing that data from code in an output plugin. Currently my > plugin > >>>>> simply does this in CODESTARTdoAction: > >>>>> > >>>>> CHKiRet(writeHiredis(ppString[**0], pData)); > >>>>> > >>>>> This passes the entire output from the template as a string to > redis. > >>>>> This works ok when I want to construct redis commands from > templates > >>>>> using individual properties straight to hiredis's > redisAppendCommand > >>>>> function, such as: > >>>>> > >>>>> "HINCR progcount %programname% 1" > >>>>> > >>>>> But it does not work when I want to send whole messages to redis > >>>>> (whitespace in the syslog message be interpreted as delimiters in > the > >>>>> command, causing errors). > >>>>> > >>>>> hiredis has another function however, redisAppendCommandArgv, > that is > >>>>> variatic. In order to use this I would like to construct a list > using > >>>>> the list templates, and then access each individual member of the > list > >>>>> from the output plugin code so I can construct the array to pass > to > >>>>> redisAppendCommandArgv. So, let's say I construct the following > >>>>> template: > >>>>> > >>>>> template(name="redisPushQueue" type="list") { > >>>>> constant(value="LPUSH") > >>>>> constant(value="queue_key") > >>>>> property(name="msg") > >>>>> } > >>>>> > >>>>> From the output plugin code, how do I then access each member > of this > >>>>> list individually with the new template code? > >>>>> > >>>>> Make sense? > >>>>> > >>>>> Brian > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> 1) Can > >>>>> On 1/3/2013 7:59 AM, Radu Gheorghe wrote: > >>>>> > >>>>>> Hello Brian, > >>>>>> > >>>>>> Just to make sure we're not missing the obvious: > >>>>>> > http://www.rsyslog.com/doc/**rsyslog_conf_templates.html<http://www.rsy > slog.com/doc/rsyslog_conf_templates.html> > >>>>>> > >>>>>> Besides that, there are a couple of examples here: > >>>>>> > >>>>>> > >>>>>> 2013/1/3 Brian Knox <[email protected]> > >>>>>> > >>>>>> I've been working on some improvements for the omhiredis output > >>>>>>> > >>>>>> plugin this > >>>>> > >>>>>> week. I think I've come up with a better way of combining > templates > >>>>>>> > >>>>>> with > >>>>> > >>>>>> redis command formatting. In order to do this, I need to, given > a > >>>>>>> > >>>>>> list > >>>>> > >>>>>> style template, get the number of elements in the list and the > >>>>>>> > >>>>>> length of > >>>>> > >>>>>> each element. > >>>>>>> > >>>>>>> I've only worked with string templates (from a plugin > perspective) > >>>>>>> > >>>>>> with my > >>>>> > >>>>>> output plugins so far. Is there a good example somewhere for > >>>>>>> > >>>>>> dealing with > >>>>> > >>>>>> list templates? > >>>>>>> > >>>>>>> On other fronts, I now omhiredis working with batch dequeue in > >>>>>>> > >>>>>> conjunction > >>>>> > >>>>>> with hiredis' pipeline commands. In local tests this let me > push > >>>>>>> > >>>>>> around > >>>>> > >>>>>> 250,000 redis commands per second to a redis instance (I was > getting > >>>>>>> > >>>>>> around > >>>>> > >>>>>> 50k commands of the same type with the same test before the > >>>>>>> > >>>>>> changes). > >>>>> > >>>>>> Rainer - I'm hoping to wrap up this next round of improvements > in a > >>>>>>> > >>>>>> few > >>>>> > >>>>>> days and get a patch your way! > >>>>>>> > >>>>>>> Brian > >>>>>>> ______________________________**_________________ > >>>>>>> rsyslog mailing list > >>>>>>> > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco > n.net/mailman/listinfo/rsyslog> > >>>>>>> http://www.rsyslog.com/**professional- > services/<http://www.rsyslog.com/professional-services/> > >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by > a > >>>>>>> > >>>>>> myriad > >>>>> > >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if > >>>>>>> > >>>>>> you > >>>>> > >>>>>> DON'T LIKE THAT. > >>>>>>> > >>>>>>> ______________________________**_________________ > >>>>>> rsyslog mailing list > >>>>>> > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco > n.net/mailman/listinfo/rsyslog> > >>>>>> http://www.rsyslog.com/**professional- > services/<http://www.rsyslog.com/professional-services/> > >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by > a > >>>>>> > >>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > POST > >>>>> if you DON'T LIKE THAT. > >>>>> > >>>>> ______________________________**_________________ > >>>>> rsyslog mailing list > >>>>> > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco > n.net/mailman/listinfo/rsyslog> > >>>>> http://www.rsyslog.com/**professional- > services/<http://www.rsyslog.com/professional-services/> > >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > POST > >>>>> if you DON'T LIKE THAT. > >>>>> > >>>> ______________________________**_________________ > >>>> rsyslog mailing list > >>>> > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco > n.net/mailman/listinfo/rsyslog> > >>>> http://www.rsyslog.com/**professional- > services/<http://www.rsyslog.com/professional-services/> > >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > POST if > >>>> you DON'T LIKE THAT. > >>>> > >>> > >>> ______________________________**_________________ > >>> rsyslog mailing list > >>> > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco > n.net/mailman/listinfo/rsyslog> > >>> http://www.rsyslog.com/**professional- > services/<http://www.rsyslog.com/professional-services/> > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > >>> DON'T LIKE THAT. > >>> > >>> ______________________________**_________________ > >> rsyslog mailing list > >> > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco > n.net/mailman/listinfo/rsyslog> > >> http://www.rsyslog.com/**professional- > services/<http://www.rsyslog.com/professional-services/> > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > >> DON'T LIKE THAT. > >> > > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
