Minor change, have made it:
foreach($.bar in $!foo) do {...}
(note the 'do' before body)
Added do because it kinda keeps syntax similar to if (expr) then {...}
Here is the change:
https://github.com/janmejay/rsyslog/commit/d4229ed42afb759adfc4d44661fda211e37c72a0
Need to clarify desired semantics of 'set' in rscript (started separate
mail thread for that). Will send a pull request for this once we close it
on that thread.
On Wed, Nov 12, 2014 at 11:07 AM, David Lang <[email protected]> wrote:
> On Wed, 12 Nov 2014, singh.janmejay wrote:
>
> How do we want foreach support to be?
>>
>> Possible places are:
>> 1. a foreach loop-construct along-side flow-control structures in rscript
>>
>
> My understanding of how rscript works is that it's mostly parsed at
> startup time, not at message processing time, so I don't see how it can be
> doen in rscript.
>
> 2. a foreach value-producing templateElement in template
>>
>> The tradeoff is, someone looking to do some useful re-structuring of
>> output, may want to call exec-template several times in approach-1, but
>> they can just emit out what they want by placing property-extractors in
>> foreach block in approch-2.
>>
>> The counter argument is, templates can't assign variables, execute
>> functions, do arithmetic-operations etc, so usefulness of foreach in a
>> template will be limited to most basic of message transformation cases.
>> Additionally, string-template as of now has no clean way of implementing
>> foreach. It'll require some involved syntax.
>>
>
> the result of templates can be assigned to variables.
>
> To me approach-1 seems better suited for such a thing (purely because
>> if-else, the other flow control structure is implemented at this level,
>> and
>> variable assignment, functions etc make arbitrary transformation easier).
>>
>> Once we have decided where to place foreach, let us zero in on its syntax.
>> To seed the thought, how about this:
>>
>> set $.grault = "";
>> foreach($!foo as $.bar) {
>> set $.baz = $.bar!quux;
>> foreach($.baz as $.corge) {
>> if ( $.corge contains "#" ) then {
>> set $.grault = $.grault & $.corge;
>> }
>> }
>> }
>>
>
> in spite of everything I said above, if this can be implemented reasonably
> (without killing performance when it's not used), this would be extremely
> powerful.
>
>
> Alternatively:
>> foreach($.bar <- $!foo) {...}
>> foreach($.bar in $!foo) {...}
>>
>
> I like this a little better than 'as', but they are almost equal and
> either is far better than the punctuation versions.
>
> David Lang
>
>
> foreach($.bar : $!foo) {...}
>>
>>
>> On Fri, Oct 31, 2014 at 6:46 PM, singh.janmejay <[email protected]
>> >
>> wrote:
>>
>> Sure, I'll fork on github.
>>>
>>> On Fri, Oct 31, 2014 at 6:42 PM, Rainer Gerhards <
>>> [email protected]
>>>
>>>> wrote:
>>>>
>>>
>>> 2014-10-31 14:08 GMT+01:00 singh.janmejay <[email protected]>:
>>>>
>>>> Cool, I'll implement $!foo!bar[0].
>>>>>
>>>>> +1
>>>>>
>>>>
>>>>
>>>> Let us process this patch-set, because is kinda hard to keep track of
>>>>> old patches and re-send in one shot.
>>>>>
>>>>>
>>>>> would you mind cloning on github and maintain a feature branch there?
>>>> That would make it much easier for me, as I could merge the branch when
>>>> you
>>>> are done. If not, it's no problem and I'll maintain that branch.
>>>>
>>>>
>>>> i'll send the new patch once done(i'll now only get to work on it on
>>>>
>>>>> monday).
>>>>>
>>>>>
>>>>> I haven't had a chance to look as I am now busy building test
>>>> environments and looking at the testbench [yes, one guy!] ;) But I see
>>>> Pavel has asked some questions. He recently did a lot of work on the
>>>> lib,so
>>>> it is best to coordinate that part with him.
>>>>
>>>> Rainer
>>>>
>>>> Do existing patches look ok except for the indexed-addressing feature?
>>>>
>>>>>
>>>>> On Fri, Oct 31, 2014 at 4:15 PM, David Lang <[email protected]> wrote:
>>>>>
>>>>> On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>>>>
>>>>>> Yes, I didn't have a need to address tokens individually, but you
>>>>>> have
>>>>>>
>>>>>>> a
>>>>>>> point.
>>>>>>>
>>>>>>> Any suggestions on what we want to do for addressing array elements?
>>>>>>>
>>>>>>> I wonder if its possible to do in $!... notation without breaking
>>>>>>> backward
>>>>>>> compatibility. How about a function?
>>>>>>>
>>>>>>> I'll be happy to implement support for addressing it in $!...
>>>>>>> notation
>>>>>>> if
>>>>>>> don't mind breaking a corner case in backward compatibility. Eg.
>>>>>>> $!foo!bar![0] ? Its kinda ugly though, or so I think.
>>>>>>>
>>>>>>>
>>>>>> I was thinking just $!foo!bar[0] it's a bit ugly, but not too bad. It
>>>>>> does mean that you can't have '[' in a variable name, but I don't
>>>>>> think
>>>>>> that's likely to be a real problem. I don't think there's ever a
>>>>>> really
>>>>>> clean way to do something like $!foo[2]!bar[2]!baz no matter what your
>>>>>> syntax, it gets messy.
>>>>>>
>>>>>> for templates, we probably need some sort of foreach(array, pattern)
>>>>>> function that takes the pattern and repeats it for each item in the
>>>>>> array.
>>>>>>
>>>>>> David Lang
>>>>>>
>>>>>>
>>>>>> On Fri, Oct 31, 2014 at 3:44 PM, David Lang <[email protected]> wrote:
>>>>>>
>>>>>>>
>>>>>>> On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> It writes it as a json array, here is a fragment from my manual
>>>>>>>> tests:
>>>>>>>>
>>>>>>>>
>>>>>>>>> [ "15", "26", "15" ]
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> right, but how do you access it in rsyslog?
>>>>>>>>
>>>>>>>> if you have { 'foo': { 'bar': '10' } } you access this as $!foo!bar
>>>>>>>> and
>>>>>>>> get the result '10'
>>>>>>>>
>>>>>>>> what would you use to access the value '26' in your example?
>>>>>>>>
>>>>>>>> we also don't have anything like foreach() in our template language,
>>>>>>>> which
>>>>>>>> makes it hard to make use of these values as anything other than a
>>>>>>>> JSON
>>>>>>>> string.
>>>>>>>>
>>>>>>>> I'm not saying that it's not useful, but I am pointing out the
>>>>>>>> problems
>>>>>>>> that we will have using it.
>>>>>>>>
>>>>>>>> David Lang
>>>>>>>>
>>>>>>>>
>>>>>>>> It was using time in hh:mm:ss format and tokening by colon(:). I'll
>>>>>>>> add
>>>>>>>>
>>>>>>>> tests for it soon, but until then pasting output here is the best I
>>>>>>>>> can
>>>>>>>>> do.
>>>>>>>>>
>>>>>>>>> The idea behind this is to generate structured content from
>>>>>>>>> semi-structured
>>>>>>>>> or unstructured log messages. So array is a good representation for
>>>>>>>>> tokenized-value (it is multi-valued by nature, and array is a good
>>>>>>>>> way to
>>>>>>>>> represent that).
>>>>>>>>>
>>>>>>>>> But eventually we should allow user to register value-transformers
>>>>>>>>> so that
>>>>>>>>> it can be pre-processed before its emitted. May be have a canned
>>>>>>>>> set
>>>>>>>>> of
>>>>>>>>> transformers, and allow user to plug in new ones.
>>>>>>>>>
>>>>>>>>> My first instinct was to utilize variable support for this, infact
>>>>>>>>> this
>>>>>>>>> was
>>>>>>>>> the motivator for variable support. But it still leads to a fairly
>>>>>>>>> complex
>>>>>>>>> config for an access log with 15 - 20 fields, especially given
>>>>>>>>> those
>>>>>>>>> fields
>>>>>>>>> can have colon separated entries inside comma separated entries
>>>>>>>>> etc.
>>>>>>>>>
>>>>>>>>> So I felt the need for a simpler way of doing it, hence this and
>>>>>>>>> other
>>>>>>>>> (recurse) field-type.
>>>>>>>>>
>>>>>>>>> On Fri, Oct 31, 2014 at 3:23 PM, David Lang <[email protected]> wrote:
>>>>>>>>>
>>>>>>>>> On Fri, 31 Oct 2014, singh.janmejay wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Tokenizer followed by tokenizer is something that I have in mind
>>>>>>>>>> too.
>>>>>>>>>> But
>>>>>>>>>>
>>>>>>>>>> I
>>>>>>>>>>
>>>>>>>>>>> promised myself that i'd write a test for that instead of testing
>>>>>>>>>>> it
>>>>>>>>>>> manually :-). Will add that patch on this thread once I get a
>>>>>>>>>>> chance to
>>>>>>>>>>> work on it.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> At least in the short term, you can use the ability to call
>>>>>>>>>>>
>>>>>>>>>> mmnormalize
>>>>>>>>>> on
>>>>>>>>>> a variable to parse subvariables.
>>>>>>>>>>
>>>>>>>>>> How are the resulting fields addressed? Rsyslog hasn't had array
>>>>>>>>>> addressing yet.
>>>>>>>>>>
>>>>>>>>>> David Lang
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> However, since you are asking about those kind of forms, let met
>>>>>>>>>> discuss
>>>>>>>>>>
>>>>>>>>>> something else that I was thinking about.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> The idea is to have another field type called recurse.
>>>>>>>>>>>
>>>>>>>>>>> Similar to how tokenized uses a ctx to parse matching text,
>>>>>>>>>>> recurse will
>>>>>>>>>>> parse it using the current context. AFAIK, the context is
>>>>>>>>>>> stateless, so
>>>>>>>>>>> I
>>>>>>>>>>> don't see any problems with that. I also plan to support tag
>>>>>>>>>>> based
>>>>>>>>>>> picking
>>>>>>>>>>> of which rules the text may match, and if it matches something
>>>>>>>>>>> else, it
>>>>>>>>>>> should be considered no-match.
>>>>>>>>>>>
>>>>>>>>>>> Instead of typing it out here, i'll attach a picture I took after
>>>>>>>>>>> thinking
>>>>>>>>>>> through it briefly(i'll attach it to the next mail).
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>
>>>>>>>>>>> Lognorm mailing list
>>>>>>>>>> [email protected]
>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Lognorm mailing list
>>>>>>>>>> [email protected]
>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>>
>>>>>>>> Lognorm mailing list
>>>>>>>> [email protected]
>>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Lognorm mailing list
>>>>>>>> [email protected]
>>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>> Lognorm mailing list
>>>>>> [email protected]
>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>
>>>>>> _______________________________________________
>>>>>> Lognorm mailing list
>>>>>> [email protected]
>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Janmejay
>>>>> http://codehunk.wordpress.com
>>>>>
>>>>> _______________________________________________
>>>>> Lognorm mailing list
>>>>> [email protected]
>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>>
>>>>>
>>>>>
>>>> _______________________________________________
>>>> Lognorm mailing list
>>>> [email protected]
>>>> http://lists.adiscon.net/mailman/listinfo/lognorm
>>>>
>>>>
>>>>
>>>
>>> --
>>> Regards,
>>> Janmejay
>>> http://codehunk.wordpress.com
>>>
>>>
>>
>>
>>
> _______________________________________________
> Lognorm mailing list
> [email protected]
> http://lists.adiscon.net/mailman/listinfo/lognorm
>
> _______________________________________________
> Lognorm mailing list
> [email protected]
> http://lists.adiscon.net/mailman/listinfo/lognorm
>
>
--
Regards,
Janmejay
http://codehunk.wordpress.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
