On Tue, Jul 16, 2013 at 8:36 AM, David Lang <[email protected]> wrote:
> On Tue, 16 Jul 2013, Rainer Gerhards wrote:
>
> Couple of points:
>>
>> $!all-json is a legacy property. It is better to access native json
>> pathes.
>>
>> Overall idea: to handle cases like this, the idea is that the data to be
>> sent is put into differet subtress and a subtree template [1] is used for
>> forwarding. IIRC, I did a blog post about this, but I have to admit I
>> didn't find it on quick search :-(.
>>
>
> the problem is similar to the issue I raised about trusted properties, how
> do you know what you are dealing with.
>
yup, but in a different flavor: I think everyone can pretend things if he
is able to reconfigure rsyslog. I think the point here is to provide the
necessary facility to rightful users. And if e.g. mmjsonparse always parses
into the root, you have essentially lost, you can't properly work with
subtrees. I remember we discussed adding a capability to write to different
subtrees, not sure though if we did implement it.
> It may be worth defining _rsyslog_local!* as a tree that is not included
> in all-json
all-json is really just an old legacy hack. I don't want to enhance it any
further. It's essentially dead ad just there so that old things continue to
work.
> so that people can define whatever they want (and I believe that 'real'
> json trees cannot start with aa '_', so it wouldn't even be a namespace
> conflict, assuming that rsyslog can tolerate it)
>
>
Thinking about the full json tree ("$!"), I don't see this brings benefit
over properly working with subtrees. Does it? It adds complexity in any
case (and hurts performance due to this added complexity).
> is the blog post you are thinking of http://blog.gerhards.net/2012/**
> 03/json-and-rsyslog-templates.**html<http://blog.gerhards.net/2012/03/json-and-rsyslog-templates.html>
>
>
Unfortunatley not. I remember s/t about subtree templates and how to use
them in situations like the one currently discussed. MAybe I just *wanted*
to write it...
Rainer
> David Lang
>
> As David said, this has not been discussed much so far. Out of my head, I
>> don't even know if all pieces have the necessary plumbing (e.g. the
>> parsers
>> need to be able to store their parsing result into a specific subtree in
>> order for the whole idea to make sense). If it's just a small thing that
>> is
>> missing, I may be able to add it before I depart for vacation. But please
>> ask for this only if you intend to actually deploy and test it (non-stable
>> versions are often not supported in enterprise envs), as I would like to
>> save my limited pre-vacation time to things that really make a difference
>> ;)
>>
>> Rainer
>>
>>
>> [1]
>> http://www.rsyslog.com/doc/**rsyslog_conf_templates.html<http://www.rsyslog.com/doc/rsyslog_conf_templates.html>
>>
>>
>> On Tue, Jul 16, 2013 at 8:07 AM, David Lang <[email protected]> wrote:
>>
>> On Mon, 15 Jul 2013, Erik Steffl wrote:
>>>
>>> On 07/15/2013 09:37 PM, David Lang wrote:
>>>
>>>>
>>>> On Mon, 15 Jul 2013, Erik Steffl wrote:
>>>>>
>>>>> Here's what I'm trying to do:
>>>>>
>>>>>>
>>>>>> receive cee message, put it into a json data structure and forward
>>>>>> the whole thing to another server.
>>>>>>
>>>>>> input: <135> time host tag @cee{"orignal":"message"}
>>>>>>
>>>>>> output: <135> time host tag @cee{"cluster":"someName", "message":
>>>>>> {"original":"message"}}
>>>>>>
>>>>>> So what I did was:
>>>>>>
>>>>>> set $!myClusterName = "someName";
>>>>>>
>>>>>> then use these as part of the template:
>>>>>>
>>>>>> constant(value="\"cluster\":{"****)
>>>>>>
>>>>>> property("$!myClusterName")
>>>>>> ...
>>>>>> constant(value="\"message\":")
>>>>>> property("$!all-json")
>>>>>>
>>>>>> and it almost works but $!myClusterName becomes part of the
>>>>>> $!all-json. Looked at docs but it seems all variables that are set
>>>>>> using set keyword become part of $!all-json.
>>>>>>
>>>>>> using property("msg") also sort of works but it includes @cee (which
>>>>>> I don't think makes sense in that part of message, it's already all
>>>>>> json).
>>>>>>
>>>>>> Any way to get $!all-json without the custom variables or "msg"
>>>>>> property without @cee? Or should I be forwarding it in some entirely
>>>>>> different way?
>>>>>>
>>>>>>
>>>>> what if you just do something like:
>>>>>
>>>>> set $!cluster = "somename";
>>>>>
>>>>> won't this include the 'cluster:"somename"' string inside the all-json
>>>>> string?
>>>>>
>>>>>
>>>> yes it will but I want exact opposite :) I am trying to not change the
>>>> original message (it might have a field named "cluster" already) or it
>>>> might not even be a cee json message etc. (I can figure out whether the
>>>> message is json formatted or not but I have to set the variables before
>>>> that since I need them no matter what the format it)
>>>>
>>>> trying to achieve something like (as described above but this might be
>>>> more obvious), pseudo code follows:
>>>>
>>>> set $!message = $!all-json
>>>> set $!cluster = "clusterName"
>>>>
>>>> after the two lines above the $!all-json has two fields - "message"
>>>> where the value is original $!all-json and cluster, where the value is
>>>> my
>>>> variable (which of course does not work cause I (as far as I know) don't
>>>> have a way to set the variable without changing $all-json.
>>>>
>>>> But only if the message is a cee message (otherwise it, of course does
>>>> not make sense). If it's not cee message then I still need cluster name
>>>> but
>>>> $!all-json!message is just the original message string.
>>>>
>>>> I.e. I am not trying to add info to the parsed message (that works,
>>>> discovered that during my failed attempts), I would like the intact
>>>> parsed
>>>> message to become part of outgoing message.
>>>>
>>>> Does that explain what I'm trying to do? Any ideas how to achieve that?
>>>>
>>>>
>>> Ok, that's a little simpler than what I was thinking you were doing.
>>>
>>> As far as I know, this is something that rsyslog does not currently
>>> handle
>>> well, we need some way to blacklist items so they don't appear in
>>> all-json
>>> without deleteing them entirely. There has occasionally be talk about
>>> implementing more control along these lines, but there hasn't been enough
>>> discussion to find good examples and people needing this control to
>>> discuss
>>> possible syntax options with.
>>>
>>>
>>> In the short term, I think the way to do what you are trying to do is to
>>> output in two formats, depending on if the json parse was successful
>>> (there
>>> is a variable set that says if the parse found json to parse), one where
>>> you output all-json, the other where you format your own string, just
>>> including the message body.
>>>
>>> I'm not understanding why you want the output to always be in json
>>> format,
>>> but only include the variable if the input was json, I would have thought
>>> you always wanted to have the variable there.
>>>
>>> If you are concerned about overwriting an existing variable, then what
>>> you
>>> need to do is to reserve a subtree, and if you find that the subtree
>>> exists, copy the contents of the subtree to subtree!old and then put your
>>> variable in subtree!variable (taking advantage of the fact that json is a
>>> tree structure, not just a name=variable flat structure.
>>>
>>> David Lang
>>>
>>> ______________________________****_________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>> >
>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>
>>> >
>>>
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>>> ______________________________**_________________
>> rsyslog mailing list
>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>> ______________________________**_________________
> rsyslog mailing list
> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
