-----Original Message-----
From: [email protected] [mailto:rsyslog-
[email protected]] On Behalf Of Brian Knox
Sent: Monday, January 07, 2013 5:18 PM
To: rsyslog-users
Subject: Re: [rsyslog] working with list templates
So - in my case I do not believe I need field names. Redis is more of
a
k/v store than a document store like mongo (although it supports more
data
structures than simply key / value, like sets, hashes, and lists).
With the hiredis api there are basically two ways of passing a command
to
redis. For the sake of example, let's take the command I played in the
previous example, with would push the content of the msg property into
a
list in redis.
Assuming "msg" contains the rsyslog message itself ("my message")
1) Using sprintf like formatting
redisCommand(context, "LPUSH mylist %s", msg);
2) Passing an array of char arrays
something like this:
const char *argv[3];
argv[0] = "LPUSH";
argv[1] = "mylist";
argv[2] = msg;
size_t lens[3] = { 5, 6, 10 };
int argc = 3;
redisFormatCommandArgv(&cmd,argc,argv,lens);
The way my current omhiredis module works is to simply call
rc = redisAppendCommand(pData->conn, (char*)message);
This works fine for doing things like incrementing counters
template(name="messages_processed" type="string" string="INCR processed
1")
However, if I include the actual log line, the spaces in the log line
break
things as then hiredis things the spaces are command delimiters.
So, if I construct the command using a list template:
template(name="redisPushQueue" type="list") {
constant(value="LPUSH")
constant(value="queue_key")
property(name="msg")
}
So that from the plugin side, I can get "LPUSH", "queue_key", and the
message from the msg property, and their lengths - then I could easily
use the ArgV version of the hiredis command runners and formatters.
Make sense?
Brian
On Mon, Jan 7, 2013 at 10:59 AM, Rainer Gerhards
<[email protected]>wrote:
-----Original Message-----
From: [email protected] [mailto:rsyslog-
[email protected]] On Behalf Of Brian Knox
Sent: Monday, January 07, 2013 2:11 PM
To: rsyslog-users
Subject: Re: [rsyslog] working with list templates
Happy Monday - getting ready to revisit the improvements to the
template
handling I'd like to make to omhiredis. To sum up the thread so
far:
1) I can register from the plugin how I would like to handle the
template
For an example, look here:
http://git.adiscon.com/?p=rsyslog.git;a=blob;f=plugins/ommongodb/ommong
odb.c;h=dd997410743ab7ed1850029f368ecb29f4405449;hb=HEAD#l504
Note the OMSR_TPL_AS_... (JSON) in this case.
2) I should probably use the JSON method as array will be
deprecated
This is actually a good question. The original plan is to deprecate
them,
HOWEVER, this is under review. Array mode *may* (may!) be a good
alternative for more traditional (SQL) like output sources. However,
array
mode lacks field names, so it all comes down to counting positions -
what
in fact IMHO makes them unsuitable. My personal guess is that we will
probably deprecate them. You'll see a related request for comment
here on
this list in the not so distant future (I need to clearly describe
the need
first).
So my question is now - if I use a "list" style template
declaration,
what
sort of JSON object will I end up with?
For a concrete example:
template(name="redisPushQueue" type="list") {
constant(value="LPUSH")
constant(value="queue_key")
property(name="msg")
}
And here's the function I'm passing to:
void redisAppendCommandArgv(redisContext *c, int argc, const char
**argv, const size_t *argvlen);
So if I could pass this as essentially a JSON array ["LPUSH",
"queue_key",
"msg"] that would make life easy.
Well, not sure about redis... don't you need field names? That's
actually
one of the big questions in regard to relational databases. I think
they
need field names in any case, so if they are not inside the template,
I
need to pass them in via a different mechanism. Thus I want to have
them
inside the template.
With JSON passing what you guess is actually just one field, namely
"msg"
with the given content. That is because the constants have no "name"
parameter, and without a name, they cannot be part of a json object
(this
is intentional, and can be useful to use the same template with
plugins
that expect JSON and plugins that expect text).
Rainer
Brian
On Sat, Jan 5, 2013 at 3:56 PM, Brian Knox <[email protected]>
wrote:
Aha! Thanks for the info David - I wouldn't want to build on top
of
a
feature that may be deprecated in the near future.
Brian
On Fri, Jan 4, 2013 at 8:30 PM, David Lang <[email protected]> wrote:
It's probably better to use JSON instead of array. Array is a
very
early
thing and there is some talk of phasing it out rather than
needing
to
support multiple ways of passing groups of messages.
David Lang
On Fri, 4 Jan 2013, Brian Knox wrote:
Aha! I think in this case array passing may be what I need.
Thanks!
Brian
On 1/4/2013 4:00 AM, Rainer Gerhards wrote:
Quick hint: you need to set the param passing mode inside the
plugin.
Not the template type specifies what you get, but the plugin
asks
the
engine. What you look for is JSON passing mode (or maybe array
passing).
Ommongodb should help you, else ping me early next week.
Rainer
-----Original Message-----
From: [email protected].**com<rsyslog-
[email protected]>[mailto:
rsyslog-
[email protected]] On Behalf Of Brian Knox
Sent: Thursday, January 03, 2013 2:22 PM
To: [email protected]
Subject: Re: [rsyslog] working with list templates
Hi Radu! Thanks for the link, but I already know how to work
with list
templates from a conf perspective. I'm looking for good
examples
of
accessing that data from code in an output plugin. Currently
my
plugin
simply does this in CODESTARTdoAction:
CHKiRet(writeHiredis(ppString[**0], pData));
This passes the entire output from the template as a string
to
redis.
This works ok when I want to construct redis commands from
templates
using individual properties straight to hiredis's
redisAppendCommand
function, such as:
"HINCR progcount %programname% 1"
But it does not work when I want to send whole messages to
redis
(whitespace in the syslog message be interpreted as
delimiters in
the
command, causing errors).
hiredis has another function however, redisAppendCommandArgv,
that is
variatic. In order to use this I would like to construct a
list
using
the list templates, and then access each individual member of
the
list
from the output plugin code so I can construct the array to
pass
to
redisAppendCommandArgv. So, let's say I construct the
following
template:
template(name="redisPushQueue" type="list") {
constant(value="LPUSH")
constant(value="queue_key")
property(name="msg")
}
From the output plugin code, how do I then access each
member
of this
list individually with the new template code?
Make sense?
Brian
1) Can
On 1/3/2013 7:59 AM, Radu Gheorghe wrote:
Hello Brian,
Just to make sure we're not missing the obvious:
http://www.rsyslog.com/doc/**rsyslog_conf_templates.html<http://www.rsy
slog.com/doc/rsyslog_conf_templates.html>
Besides that, there are a couple of examples here:
2013/1/3 Brian Knox <[email protected]>
I've been working on some improvements for the omhiredis
output
plugin this
week. I think I've come up with a better way of combining
templates
with
redis command formatting. In order to do this, I need to,
given
a
list
style template, get the number of elements in the list and
the
length of
each element.
I've only worked with string templates (from a plugin
perspective)
with my
output plugins so far. Is there a good example somewhere
for
dealing with
list templates?
On other fronts, I now omhiredis working with batch dequeue
in
conjunction
with hiredis' pipeline commands. In local tests this let me
push
around
250,000 redis commands per second to a redis instance (I was
getting
around
50k commands of the same type with the same test before the
changes).
Rainer - I'm hoping to wrap up this next round of
improvements
in a
few
days and get a patch your way!
Brian
______________________________**_________________
rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco
n.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-
services/<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow
https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are
ARCHIVED by
a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST
if
you
DON'T LIKE THAT.
______________________________**_________________
rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco
n.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-
services/<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
by
a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO
NOT
POST
if you DON'T LIKE THAT.
______________________________**_________________
rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco
n.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-
services/<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO
NOT
POST
if you DON'T LIKE THAT.
______________________________**_________________
rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco
n.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-
services/<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO
NOT
POST if
you DON'T LIKE THAT.
______________________________**_________________
rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco
n.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-
services/<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if
you
DON'T LIKE THAT.
______________________________**_________________
rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco
n.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-
services/<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST
if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if you DON'T LIKE THAT.
