> > Which from my tests shows that $.msg retains the $msg value even though > > it was supposed to be overwritten by the "whatever" ruleset is caused > > not by the queue mechanics but by non-linearity of execution and race? > > I'll check. Maybe my memory served me incorrectly - but I don't think so.
OK, it's actually a kind of bug, but one that we better not fix. Here is what happens: Call, by design, is able to call a rule set either synchronously or asynchronously. We did this, because practice showed that both modes are needed. For various reasons (would need to dig out, but IIRC "backwards compatibility" as among them), we decided to make async calls if the ruleset has a queue assigned and sync if not. Now comes the bug: to know if a "queue is assigned" we just check if queue parameters are given. I overlooked the case of someone explicitly specifying a "direct queue", aka "no queue". As such, queue="direct" triggers async calls. That in turn means while the actually the same message object IS modified, the mentioned race occurs. In many cases (OS scheduling reasons), the next action will be executed before the async rule set (active thread vs. waiting thread). So the output uses the old value before the ruleset sets the new one. Can we fix this? Yes, it's easy, just check queue type and if it is "direct", do a sync call. BUT: this would potentially break existing configurations, something we do only if there is really a very good reason to do so. I do not see the case strong enough for a breaking fix. I think I will add a warning message when a direct queue type is detected but explicitly set. So users can become aware of the issue. Any objections? Rainer > > Rainer > > > > > MK > > > > > > On 17.09.2021 08:55, Rainer Gerhards via rsyslog wrote: > > > The default queue mode is "direct" - no matter if it is explicitly > > > specified or not. > > > > > > There are indeed semantic differences between the "direct" queue > > > (a.k.a. "no real queue exists") and any other (real) queue type. As > > > you say, in direct mode, the message object is NOT copied. So anything > > > done to it, will potentially affect other uses of the very same > > > message object. I real queues, the message is copied and thus no > > > processing outside the scope of the queue is affected. > > > > > > NOTE WELL: if you have parts with direct AND real queues AND modify > > > messages in "direct queue constructs" AND have actions/rulesets with > > > real queues running in parallel, you have a race. If the modification > > > made by the direct queue part depends on whether processing of the > > > real queues has already started. For "regular" rule sets this is no > > > problem, because main processing is sequential. However, if you do > > > very complex setups with multipele rule sets calling into each other > > > and using different queue modes, this can affect you. > > > > > > It is best, also for maintainability, to have a simple sequential > > > structure in top level processing, spawning off topic rule sets (if at > > > all required). then do all "global" modifications at the beginning. > > > And do local (intentionally non outside-visible) modifications in the > > > topic rule sets. > > > > > > I hope this helps, > > > Rainer > > > > > > El vie, 17 sept 2021 a las 6:17, rajeshksv via rsyslog > > > (<[email protected]>) escribió: > > >> Agreed. We don't need a direct queue before ruleset. I added it only to > > >> understand what's going with DIRECT Queues (Since its a very novel > > >> concept > > >> - Haven't heard something like this before) > > >> > > >> If you remove the DIRECT queue from ruleset(in Scenario I), then > > >> modifications done in ruleset are visible to the main lane. It's working > > >> as > > >> expected. But since I am interested in understanding DIRECT Queues, I > > >> added > > >> it. > > >> > > >>>>> Also, although the maintainers will need to say for sure, user > > >>>>> variable > > >> defined in a call may not be visible to the parent. Same as in most > > >> programming languages. > > >> Rsyslog works in a different way, in few scenarios, user variables > > >> defined > > >> in a call will be visible to the parent (If ruleset is not backed by > > >> Queue). So, its not similar to other languages ;) > > >> > > >>>>> "Note that mmnormalize should only be called once on each message. > > >> Behaviour is undefined if multiple calls to mmnormalize happen for the > > >> same message." > > >> Its a documentation bug, Rainer has raised a PR and will be resolved > > >> soon - > > >> https://github.com/rsyslog/rsyslog-doc/pull/931/files. Its safe to use it > > >> multiple times. > > >> > > >> > > >> On Fri, Sep 17, 2021 at 12:56 AM David Lang via rsyslog < > > >> [email protected]> wrote: > > >> > > >>> On Thu, 16 Sep 2021, Mariusz Kruk via rsyslog wrote: > > >>> > > >>> https://www.rsyslog.com/doc/v8-stable/configuration/modules/mmnormalize.html > > >>>> "Note that mmnormalize should only be called once on each message. > > >>>> Behaviour is undefined if multiple calls to mmnormalize happen for the > > >>>> same message." > > >>> I called that out and it should have been removed from current > > >>> documentation. > > >>> > > >>>> From my tests it turns out that even with direct queue the message > > >>>> gets > > >>>> copied when entering a separate queue and thus the results are not > > >>>> inherited on the ruleset (and queue) exit. > > >>>> > > >>>> So it seems to be the mmnormalize that's causing the OP's variable to > > >>>> be > > >>>> retained after the ruleset exit. > > >>> but if you don't specify any queue at all, then things inside a ruleset > > >>> will > > >>> affect things outside the ruleset. > > >>> > > >>> David Lang > > >>> > > >>>> On 16.09.2021 20:54, David Lang wrote: > > >>>>> There is no reason you can't call mmnormalize multiple times, you can > > >>>>> specify where to put the results (by default, they go under $!) > > >>>>> > > >>>>> I'm not really sure what a direct queue on a ruleset means, you should > > >>>>> just call the ruleset without it having any queue on it. > > >>>>> > > >>>>> David Lang > > >>>>> > > >>>>> On Thu, 16 Sep 2021, Mariusz Kruk via rsyslog wrote: > > >>>>> > > >>>>>> I believe it's not about queue, but about mmnormalize. I haven't used > > >>>>>> mmnormalize much myself but from the docs I understand that > > >>>>>> mmnormalize should only be called once on a message and is supposed > > >>>>>> to parse the message into a set of "global" properties. > > >>>>>> > > >>>>>> > > >>>>>> On 16.09.2021 15:21, rajeshksv via rsyslog wrote: > > >>>>>>> Hi Rsyslog Users, > > >>>>>>> > > >>>>>>> I am trying to understand how queues work in Rsyslog. In case of > > >>>>>>> non-direct > > >>>>>>> queues, message is copied and placed in the queue and message > > >>>>>>> modifications > > >>>>>>> done by queue workers won't have any impact on the original message. > > >>>>>>> Makes > > >>>>>>> sense here. > > >>>>>>> > > >>>>>>> However, I am a little confused when it comes to direct Queue. What > > >>>>>>> happens in Direct Queue ? Will the message be copied or is it the > > >>>>>>> same > > >>>>>>> message ? I tested out two scenarios (wrt rulesets and actions) and > > >>>>>>> both > > >>>>>>> of them gave different results. > > >>>>>>> > > >>>>>>> When ruleset is backed by a direct Queue, message modifications done > > >>> in > > >>>>>>> ruleset don't reflect back in original flow where as in case of > > >>> actions > > >>>>>>> (such mmnormalize, mmkubernetes) which are by default backed by > > >>>>>>> direct > > >>>>>>> Queue, message modifications done with action reflects in original > > >>> flow > > >>>>>>> Scenario 1: > > >>>>>>> > > >>>>>>> template(name="abc" type="string" string="%$!var1% %$.var2% %msg%") > > >>>>>>> > > >>>>>>> ruleset(name="relay.htp1" queue.type="Direct") { > > >>>>>>> call rs1 > > >>>>>>> * // $!var1, $.var2 aren't available here* > > >>>>>>> action(type="omfile" file="/tmp/output.log" template="abc") > > >>>>>>> call relay.htp > > >>>>>>> } > > >>>>>>> > > >>>>>>> ruleset(name="rs1" queue.type="Direct"){ > > >>>>>>> set $!var1 = "hello"; > > >>>>>>> set $.var2 = "bye"; > > >>>>>>> } > > >>>>>>> > > >>>>>>> input(type="imfile" > > >>>>>>> File="/tmp/input.log" > > >>>>>>> Ruleset="relay.htp1" > > >>>>>>> Tag="tag") > > >>>>>>> > > >>>>>>> > > >>>>>>> Scenario 2: > > >>>>>>> > > >>>>>>> module(load = "mmnormalize") > > >>>>>>> ruleset(name = "relay.htp1" queue.type="Direct") { > > >>>>>>> action(type = "mmnormalize" > > >>>>>>> ruleBase="/etc/rsyslog.d/service.rulebase" > > >>>>>>> path="$!msg") > > >>>>>>> * // $!msg will be available here even though action is backed > > >>>>>>> by a > > >>>>>>> default Queue. * > > >>>>>>> } > > >>>>>>> > > >>>>>>> input(type="imfile" > > >>>>>>> File="/tmp/input.log" > > >>>>>>> Ruleset="relay.htp1" > > >>>>>>> Tag="tag") > > >>>>>>> > > >>>>>>> > > >>>>>>> How come $!var1, $.var2 aren't available in scenario1 whereas $!msg > > >>>>>>> is > > >>>>>>> available when both are using Direct Queue. Am I missing something > > >>>>>>> here ? > > >>>>>>> > > >>>>>> _______________________________________________ > > >>>>>> rsyslog mailing list > > >>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog > > >>>>>> http://www.rsyslog.com/professional-services/ > > >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > > >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > >>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > > >>>>>> POST if you DON'T LIKE THAT. > > >>>>>> > > >>>> _______________________________________________ > > >>>> rsyslog mailing list > > >>>> https://lists.adiscon.net/mailman/listinfo/rsyslog > > >>>> http://www.rsyslog.com/professional-services/ > > >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > > >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > >>>> myriad > > >>> of > > >>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > >>> DON'T > > >>>> LIKE THAT. > > >>> _______________________________________________ > > >>> rsyslog mailing list > > >>> https://lists.adiscon.net/mailman/listinfo/rsyslog > > >>> http://www.rsyslog.com/professional-services/ > > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > >>> DON'T LIKE THAT. > > >> > > >> > > >> -- > > >> Regards, > > >> Rajesh KSV > > >> _______________________________________________ > > >> rsyslog mailing list > > >> https://lists.adiscon.net/mailman/listinfo/rsyslog > > >> http://www.rsyslog.com/professional-services/ > > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > >> DON'T LIKE THAT. > > > _______________________________________________ > > > rsyslog mailing list > > > https://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com/professional-services/ > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > > DON'T LIKE THAT. > > _______________________________________________ > > rsyslog mailing list > > https://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > > LIKE THAT. _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
