Sorry, this should be discussed on rsyslog mailing list, adding rsyslog as well.
Here are the pull requests(in order): https://github.com/rsyslog/libestr/pull/1 https://github.com/rsyslog/liblognorm/pull/8 https://github.com/rsyslog/rsyslog/pull/149 On Wed, Nov 12, 2014 at 9:55 AM, singh.janmejay <[email protected]> wrote: > The array-subscripting patches are ready, I'll send a pull request on > github. > > It is of the form $!foo[1]!bar[2]!baz. > > On Wed, Nov 12, 2014 at 9:52 AM, singh.janmejay <[email protected]> > wrote: > >> How do we want foreach support to be? >> >> Possible places are: >> 1. a foreach loop-construct along-side flow-control structures in rscript >> 2. a foreach value-producing templateElement in template >> >> The tradeoff is, someone looking to do some useful re-structuring of >> output, may want to call exec-template several times in approach-1, but >> they can just emit out what they want by placing property-extractors in >> foreach block in approch-2. >> >> The counter argument is, templates can't assign variables, execute >> functions, do arithmetic-operations etc, so usefulness of foreach in a >> template will be limited to most basic of message transformation cases. >> Additionally, string-template as of now has no clean way of implementing >> foreach. It'll require some involved syntax. >> >> To me approach-1 seems better suited for such a thing (purely because >> if-else, the other flow control structure is implemented at this level, and >> variable assignment, functions etc make arbitrary transformation easier). >> >> Once we have decided where to place foreach, let us zero in on its >> syntax. To seed the thought, how about this: >> >> set $.grault = ""; >> foreach($!foo as $.bar) { >> set $.baz = $.bar!quux; >> foreach($.baz as $.corge) { >> if ( $.corge contains "#" ) then { >> set $.grault = $.grault & $.corge; >> } >> } >> } >> >> Alternatively: >> foreach($.bar <- $!foo) {...} >> foreach($.bar in $!foo) {...} >> foreach($.bar : $!foo) {...} >> >> >> >> On Fri, Oct 31, 2014 at 6:46 PM, singh.janmejay <[email protected] >> > wrote: >> >>> Sure, I'll fork on github. >>> >>> On Fri, Oct 31, 2014 at 6:42 PM, Rainer Gerhards < >>> [email protected]> wrote: >>> >>>> 2014-10-31 14:08 GMT+01:00 singh.janmejay <[email protected]>: >>>> >>>>> Cool, I'll implement $!foo!bar[0]. >>>>> >>>>> +1 >>>> >>>> >>>>> Let us process this patch-set, because is kinda hard to keep track of >>>>> old patches and re-send in one shot. >>>>> >>>>> >>>> would you mind cloning on github and maintain a feature branch there? >>>> That would make it much easier for me, as I could merge the branch when you >>>> are done. If not, it's no problem and I'll maintain that branch. >>>> >>>> >>>> i'll send the new patch once done(i'll now only get to work on it on >>>>> monday). >>>>> >>>>> >>>> I haven't had a chance to look as I am now busy building test >>>> environments and looking at the testbench [yes, one guy!] ;) But I see >>>> Pavel has asked some questions. He recently did a lot of work on the lib,so >>>> it is best to coordinate that part with him. >>>> >>>> Rainer >>>> >>>> Do existing patches look ok except for the indexed-addressing feature? >>>>> >>>>> On Fri, Oct 31, 2014 at 4:15 PM, David Lang <[email protected]> wrote: >>>>> >>>>>> On Fri, 31 Oct 2014, singh.janmejay wrote: >>>>>> >>>>>> Yes, I didn't have a need to address tokens individually, but you >>>>>>> have a >>>>>>> point. >>>>>>> >>>>>>> Any suggestions on what we want to do for addressing array elements? >>>>>>> >>>>>>> I wonder if its possible to do in $!... notation without breaking >>>>>>> backward >>>>>>> compatibility. How about a function? >>>>>>> >>>>>>> I'll be happy to implement support for addressing it in $!... >>>>>>> notation if >>>>>>> don't mind breaking a corner case in backward compatibility. Eg. >>>>>>> $!foo!bar![0] ? Its kinda ugly though, or so I think. >>>>>>> >>>>>> >>>>>> I was thinking just $!foo!bar[0] it's a bit ugly, but not too bad. It >>>>>> does mean that you can't have '[' in a variable name, but I don't think >>>>>> that's likely to be a real problem. I don't think there's ever a really >>>>>> clean way to do something like $!foo[2]!bar[2]!baz no matter what your >>>>>> syntax, it gets messy. >>>>>> >>>>>> for templates, we probably need some sort of foreach(array, pattern) >>>>>> function that takes the pattern and repeats it for each item in the >>>>>> array. >>>>>> >>>>>> David Lang >>>>>> >>>>>> >>>>>> On Fri, Oct 31, 2014 at 3:44 PM, David Lang <[email protected]> wrote: >>>>>>> >>>>>>> On Fri, 31 Oct 2014, singh.janmejay wrote: >>>>>>>> >>>>>>>> It writes it as a json array, here is a fragment from my manual >>>>>>>> tests: >>>>>>>> >>>>>>>>> >>>>>>>>> [ "15", "26", "15" ] >>>>>>>>> >>>>>>>>> >>>>>>>> right, but how do you access it in rsyslog? >>>>>>>> >>>>>>>> if you have { 'foo': { 'bar': '10' } } you access this as $!foo!bar >>>>>>>> and >>>>>>>> get the result '10' >>>>>>>> >>>>>>>> what would you use to access the value '26' in your example? >>>>>>>> >>>>>>>> we also don't have anything like foreach() in our template >>>>>>>> language, which >>>>>>>> makes it hard to make use of these values as anything other than a >>>>>>>> JSON >>>>>>>> string. >>>>>>>> >>>>>>>> I'm not saying that it's not useful, but I am pointing out the >>>>>>>> problems >>>>>>>> that we will have using it. >>>>>>>> >>>>>>>> David Lang >>>>>>>> >>>>>>>> >>>>>>>> It was using time in hh:mm:ss format and tokening by colon(:). >>>>>>>> I'll add >>>>>>>> >>>>>>>>> tests for it soon, but until then pasting output here is the best >>>>>>>>> I can >>>>>>>>> do. >>>>>>>>> >>>>>>>>> The idea behind this is to generate structured content from >>>>>>>>> semi-structured >>>>>>>>> or unstructured log messages. So array is a good representation for >>>>>>>>> tokenized-value (it is multi-valued by nature, and array is a good >>>>>>>>> way to >>>>>>>>> represent that). >>>>>>>>> >>>>>>>>> But eventually we should allow user to register value-transformers >>>>>>>>> so that >>>>>>>>> it can be pre-processed before its emitted. May be have a canned >>>>>>>>> set of >>>>>>>>> transformers, and allow user to plug in new ones. >>>>>>>>> >>>>>>>>> My first instinct was to utilize variable support for this, infact >>>>>>>>> this >>>>>>>>> was >>>>>>>>> the motivator for variable support. But it still leads to a fairly >>>>>>>>> complex >>>>>>>>> config for an access log with 15 - 20 fields, especially given >>>>>>>>> those >>>>>>>>> fields >>>>>>>>> can have colon separated entries inside comma separated entries >>>>>>>>> etc. >>>>>>>>> >>>>>>>>> So I felt the need for a simpler way of doing it, hence this and >>>>>>>>> other >>>>>>>>> (recurse) field-type. >>>>>>>>> >>>>>>>>> On Fri, Oct 31, 2014 at 3:23 PM, David Lang <[email protected]> wrote: >>>>>>>>> >>>>>>>>> On Fri, 31 Oct 2014, singh.janmejay wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Tokenizer followed by tokenizer is something that I have in mind >>>>>>>>>> too. >>>>>>>>>> But >>>>>>>>>> >>>>>>>>>> I >>>>>>>>>>> promised myself that i'd write a test for that instead of >>>>>>>>>>> testing it >>>>>>>>>>> manually :-). Will add that patch on this thread once I get a >>>>>>>>>>> chance to >>>>>>>>>>> work on it. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> At least in the short term, you can use the ability to call >>>>>>>>>> mmnormalize >>>>>>>>>> on >>>>>>>>>> a variable to parse subvariables. >>>>>>>>>> >>>>>>>>>> How are the resulting fields addressed? Rsyslog hasn't had array >>>>>>>>>> addressing yet. >>>>>>>>>> >>>>>>>>>> David Lang >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> However, since you are asking about those kind of forms, let met >>>>>>>>>> discuss >>>>>>>>>> >>>>>>>>>> something else that I was thinking about. >>>>>>>>>>> >>>>>>>>>>> The idea is to have another field type called recurse. >>>>>>>>>>> >>>>>>>>>>> Similar to how tokenized uses a ctx to parse matching text, >>>>>>>>>>> recurse will >>>>>>>>>>> parse it using the current context. AFAIK, the context is >>>>>>>>>>> stateless, so >>>>>>>>>>> I >>>>>>>>>>> don't see any problems with that. I also plan to support tag >>>>>>>>>>> based >>>>>>>>>>> picking >>>>>>>>>>> of which rules the text may match, and if it matches something >>>>>>>>>>> else, it >>>>>>>>>>> should be considered no-match. >>>>>>>>>>> >>>>>>>>>>> Instead of typing it out here, i'll attach a picture I took after >>>>>>>>>>> thinking >>>>>>>>>>> through it briefly(i'll attach it to the next mail). >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> >>>>>>>>>> Lognorm mailing list >>>>>>>>>> [email protected] >>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Lognorm mailing list >>>>>>>>>> [email protected] >>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>> Lognorm mailing list >>>>>>>> [email protected] >>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Lognorm mailing list >>>>>>>> [email protected] >>>>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> Lognorm mailing list >>>>>> [email protected] >>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm >>>>>> >>>>>> _______________________________________________ >>>>>> Lognorm mailing list >>>>>> [email protected] >>>>>> http://lists.adiscon.net/mailman/listinfo/lognorm >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> Janmejay >>>>> http://codehunk.wordpress.com >>>>> >>>>> _______________________________________________ >>>>> Lognorm mailing list >>>>> [email protected] >>>>> http://lists.adiscon.net/mailman/listinfo/lognorm >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> Lognorm mailing list >>>> [email protected] >>>> http://lists.adiscon.net/mailman/listinfo/lognorm >>>> >>>> >>> >>> >>> -- >>> Regards, >>> Janmejay >>> http://codehunk.wordpress.com >>> >> >> >> >> -- >> Regards, >> Janmejay >> http://codehunk.wordpress.com >> > > > > -- > Regards, > Janmejay > http://codehunk.wordpress.com > -- Regards, Janmejay http://codehunk.wordpress.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
