El vie, 17 sept 2021 a las 9:15, Mariusz Kruk via rsyslog (<[email protected]>) escribió: > > Hmm... > > OK, so a simple setup of: > > [cut] > > ruleset(name="whatever" queue.type="direct") { > > set $.msg="whatever"; > > } > > set $.msg=$msg; > > call whatever > > template using $.msg > > output with the template > > [cut] > > Which from my tests shows that $.msg retains the $msg value even though > it was supposed to be overwritten by the "whatever" ruleset is caused > not by the queue mechanics but by non-linearity of execution and race?
I'll check. Maybe my memory served me incorrectly - but I don't think so. Rainer > > MK > > On 17.09.2021 08:55, Rainer Gerhards via rsyslog wrote: > > The default queue mode is "direct" - no matter if it is explicitly > > specified or not. > > > > There are indeed semantic differences between the "direct" queue > > (a.k.a. "no real queue exists") and any other (real) queue type. As > > you say, in direct mode, the message object is NOT copied. So anything > > done to it, will potentially affect other uses of the very same > > message object. I real queues, the message is copied and thus no > > processing outside the scope of the queue is affected. > > > > NOTE WELL: if you have parts with direct AND real queues AND modify > > messages in "direct queue constructs" AND have actions/rulesets with > > real queues running in parallel, you have a race. If the modification > > made by the direct queue part depends on whether processing of the > > real queues has already started. For "regular" rule sets this is no > > problem, because main processing is sequential. However, if you do > > very complex setups with multipele rule sets calling into each other > > and using different queue modes, this can affect you. > > > > It is best, also for maintainability, to have a simple sequential > > structure in top level processing, spawning off topic rule sets (if at > > all required). then do all "global" modifications at the beginning. > > And do local (intentionally non outside-visible) modifications in the > > topic rule sets. > > > > I hope this helps, > > Rainer > > > > El vie, 17 sept 2021 a las 6:17, rajeshksv via rsyslog > > (<[email protected]>) escribió: > >> Agreed. We don't need a direct queue before ruleset. I added it only to > >> understand what's going with DIRECT Queues (Since its a very novel concept > >> - Haven't heard something like this before) > >> > >> If you remove the DIRECT queue from ruleset(in Scenario I), then > >> modifications done in ruleset are visible to the main lane. It's working as > >> expected. But since I am interested in understanding DIRECT Queues, I added > >> it. > >> > >>>>> Also, although the maintainers will need to say for sure, user variable > >> defined in a call may not be visible to the parent. Same as in most > >> programming languages. > >> Rsyslog works in a different way, in few scenarios, user variables defined > >> in a call will be visible to the parent (If ruleset is not backed by > >> Queue). So, its not similar to other languages ;) > >> > >>>>> "Note that mmnormalize should only be called once on each message. > >> Behaviour is undefined if multiple calls to mmnormalize happen for the > >> same message." > >> Its a documentation bug, Rainer has raised a PR and will be resolved soon - > >> https://github.com/rsyslog/rsyslog-doc/pull/931/files. Its safe to use it > >> multiple times. > >> > >> > >> On Fri, Sep 17, 2021 at 12:56 AM David Lang via rsyslog < > >> [email protected]> wrote: > >> > >>> On Thu, 16 Sep 2021, Mariusz Kruk via rsyslog wrote: > >>> > >>> https://www.rsyslog.com/doc/v8-stable/configuration/modules/mmnormalize.html > >>>> "Note that mmnormalize should only be called once on each message. > >>>> Behaviour is undefined if multiple calls to mmnormalize happen for the > >>>> same message." > >>> I called that out and it should have been removed from current > >>> documentation. > >>> > >>>> From my tests it turns out that even with direct queue the message gets > >>>> copied when entering a separate queue and thus the results are not > >>>> inherited on the ruleset (and queue) exit. > >>>> > >>>> So it seems to be the mmnormalize that's causing the OP's variable to be > >>>> retained after the ruleset exit. > >>> but if you don't specify any queue at all, then things inside a ruleset > >>> will > >>> affect things outside the ruleset. > >>> > >>> David Lang > >>> > >>>> On 16.09.2021 20:54, David Lang wrote: > >>>>> There is no reason you can't call mmnormalize multiple times, you can > >>>>> specify where to put the results (by default, they go under $!) > >>>>> > >>>>> I'm not really sure what a direct queue on a ruleset means, you should > >>>>> just call the ruleset without it having any queue on it. > >>>>> > >>>>> David Lang > >>>>> > >>>>> On Thu, 16 Sep 2021, Mariusz Kruk via rsyslog wrote: > >>>>> > >>>>>> I believe it's not about queue, but about mmnormalize. I haven't used > >>>>>> mmnormalize much myself but from the docs I understand that > >>>>>> mmnormalize should only be called once on a message and is supposed > >>>>>> to parse the message into a set of "global" properties. > >>>>>> > >>>>>> > >>>>>> On 16.09.2021 15:21, rajeshksv via rsyslog wrote: > >>>>>>> Hi Rsyslog Users, > >>>>>>> > >>>>>>> I am trying to understand how queues work in Rsyslog. In case of > >>>>>>> non-direct > >>>>>>> queues, message is copied and placed in the queue and message > >>>>>>> modifications > >>>>>>> done by queue workers won't have any impact on the original message. > >>>>>>> Makes > >>>>>>> sense here. > >>>>>>> > >>>>>>> However, I am a little confused when it comes to direct Queue. What > >>>>>>> happens in Direct Queue ? Will the message be copied or is it the same > >>>>>>> message ? I tested out two scenarios (wrt rulesets and actions) and > >>>>>>> both > >>>>>>> of them gave different results. > >>>>>>> > >>>>>>> When ruleset is backed by a direct Queue, message modifications done > >>> in > >>>>>>> ruleset don't reflect back in original flow where as in case of > >>> actions > >>>>>>> (such mmnormalize, mmkubernetes) which are by default backed by direct > >>>>>>> Queue, message modifications done with action reflects in original > >>> flow > >>>>>>> Scenario 1: > >>>>>>> > >>>>>>> template(name="abc" type="string" string="%$!var1% %$.var2% %msg%") > >>>>>>> > >>>>>>> ruleset(name="relay.htp1" queue.type="Direct") { > >>>>>>> call rs1 > >>>>>>> * // $!var1, $.var2 aren't available here* > >>>>>>> action(type="omfile" file="/tmp/output.log" template="abc") > >>>>>>> call relay.htp > >>>>>>> } > >>>>>>> > >>>>>>> ruleset(name="rs1" queue.type="Direct"){ > >>>>>>> set $!var1 = "hello"; > >>>>>>> set $.var2 = "bye"; > >>>>>>> } > >>>>>>> > >>>>>>> input(type="imfile" > >>>>>>> File="/tmp/input.log" > >>>>>>> Ruleset="relay.htp1" > >>>>>>> Tag="tag") > >>>>>>> > >>>>>>> > >>>>>>> Scenario 2: > >>>>>>> > >>>>>>> module(load = "mmnormalize") > >>>>>>> ruleset(name = "relay.htp1" queue.type="Direct") { > >>>>>>> action(type = "mmnormalize" > >>>>>>> ruleBase="/etc/rsyslog.d/service.rulebase" > >>>>>>> path="$!msg") > >>>>>>> * // $!msg will be available here even though action is backed by > >>>>>>> a > >>>>>>> default Queue. * > >>>>>>> } > >>>>>>> > >>>>>>> input(type="imfile" > >>>>>>> File="/tmp/input.log" > >>>>>>> Ruleset="relay.htp1" > >>>>>>> Tag="tag") > >>>>>>> > >>>>>>> > >>>>>>> How come $!var1, $.var2 aren't available in scenario1 whereas $!msg is > >>>>>>> available when both are using Direct Queue. Am I missing something > >>>>>>> here ? > >>>>>>> > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com/professional-services/ > >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > >>>>>> POST if you DON'T LIKE THAT. > >>>>>> > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> https://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com/professional-services/ > >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >>> of > >>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>> DON'T > >>>> LIKE THAT. > >>> _______________________________________________ > >>> rsyslog mailing list > >>> https://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com/professional-services/ > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>> DON'T LIKE THAT. > >> > >> > >> -- > >> Regards, > >> Rajesh KSV > >> _______________________________________________ > >> rsyslog mailing list > >> https://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > > _______________________________________________ > > rsyslog mailing list > > https://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > > LIKE THAT. > _______________________________________________ > rsyslog mailing list > https://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
