The default queue mode is "direct" - no matter if it is explicitly specified or not.
There are indeed semantic differences between the "direct" queue (a.k.a. "no real queue exists") and any other (real) queue type. As you say, in direct mode, the message object is NOT copied. So anything done to it, will potentially affect other uses of the very same message object. I real queues, the message is copied and thus no processing outside the scope of the queue is affected. NOTE WELL: if you have parts with direct AND real queues AND modify messages in "direct queue constructs" AND have actions/rulesets with real queues running in parallel, you have a race. If the modification made by the direct queue part depends on whether processing of the real queues has already started. For "regular" rule sets this is no problem, because main processing is sequential. However, if you do very complex setups with multipele rule sets calling into each other and using different queue modes, this can affect you. It is best, also for maintainability, to have a simple sequential structure in top level processing, spawning off topic rule sets (if at all required). then do all "global" modifications at the beginning. And do local (intentionally non outside-visible) modifications in the topic rule sets. I hope this helps, Rainer El vie, 17 sept 2021 a las 6:17, rajeshksv via rsyslog (<[email protected]>) escribió: > > Agreed. We don't need a direct queue before ruleset. I added it only to > understand what's going with DIRECT Queues (Since its a very novel concept > - Haven't heard something like this before) > > If you remove the DIRECT queue from ruleset(in Scenario I), then > modifications done in ruleset are visible to the main lane. It's working as > expected. But since I am interested in understanding DIRECT Queues, I added > it. > > >>> Also, although the maintainers will need to say for sure, user variable > defined in a call may not be visible to the parent. Same as in most > programming languages. > Rsyslog works in a different way, in few scenarios, user variables defined > in a call will be visible to the parent (If ruleset is not backed by > Queue). So, its not similar to other languages ;) > > >>> "Note that mmnormalize should only be called once on each message. > Behaviour is undefined if multiple calls to mmnormalize happen for the > same message." > Its a documentation bug, Rainer has raised a PR and will be resolved soon - > https://github.com/rsyslog/rsyslog-doc/pull/931/files. Its safe to use it > multiple times. > > > On Fri, Sep 17, 2021 at 12:56 AM David Lang via rsyslog < > [email protected]> wrote: > > > On Thu, 16 Sep 2021, Mariusz Kruk via rsyslog wrote: > > > > > > > https://www.rsyslog.com/doc/v8-stable/configuration/modules/mmnormalize.html > > > > > > "Note that mmnormalize should only be called once on each message. > > > Behaviour is undefined if multiple calls to mmnormalize happen for the > > > same message." > > > > I called that out and it should have been removed from current > > documentation. > > > > > From my tests it turns out that even with direct queue the message gets > > > copied when entering a separate queue and thus the results are not > > > inherited on the ruleset (and queue) exit. > > > > > > So it seems to be the mmnormalize that's causing the OP's variable to be > > > retained after the ruleset exit. > > > > but if you don't specify any queue at all, then things inside a ruleset > > will > > affect things outside the ruleset. > > > > David Lang > > > > > On 16.09.2021 20:54, David Lang wrote: > > >> There is no reason you can't call mmnormalize multiple times, you can > > >> specify where to put the results (by default, they go under $!) > > >> > > >> I'm not really sure what a direct queue on a ruleset means, you should > > >> just call the ruleset without it having any queue on it. > > >> > > >> David Lang > > >> > > >> On Thu, 16 Sep 2021, Mariusz Kruk via rsyslog wrote: > > >> > > >>> I believe it's not about queue, but about mmnormalize. I haven't used > > >>> mmnormalize much myself but from the docs I understand that > > >>> mmnormalize should only be called once on a message and is supposed > > >>> to parse the message into a set of "global" properties. > > >>> > > >>> > > >>> On 16.09.2021 15:21, rajeshksv via rsyslog wrote: > > >>>> Hi Rsyslog Users, > > >>>> > > >>>> I am trying to understand how queues work in Rsyslog. In case of > > >>>> non-direct > > >>>> queues, message is copied and placed in the queue and message > > >>>> modifications > > >>>> done by queue workers won't have any impact on the original message. > > >>>> Makes > > >>>> sense here. > > >>>> > > >>>> However, I am a little confused when it comes to direct Queue. What > > >>>> happens in Direct Queue ? Will the message be copied or is it the same > > >>>> message ? I tested out two scenarios (wrt rulesets and actions) and > > >>>> both > > >>>> of them gave different results. > > >>>> > > >>>> When ruleset is backed by a direct Queue, message modifications done > > in > > >>>> ruleset don't reflect back in original flow where as in case of > > actions > > >>>> (such mmnormalize, mmkubernetes) which are by default backed by direct > > >>>> Queue, message modifications done with action reflects in original > > flow > > >>>> > > >>>> Scenario 1: > > >>>> > > >>>> template(name="abc" type="string" string="%$!var1% %$.var2% %msg%") > > >>>> > > >>>> ruleset(name="relay.htp1" queue.type="Direct") { > > >>>> call rs1 > > >>>> * // $!var1, $.var2 aren't available here* > > >>>> action(type="omfile" file="/tmp/output.log" template="abc") > > >>>> call relay.htp > > >>>> } > > >>>> > > >>>> ruleset(name="rs1" queue.type="Direct"){ > > >>>> set $!var1 = "hello"; > > >>>> set $.var2 = "bye"; > > >>>> } > > >>>> > > >>>> input(type="imfile" > > >>>> File="/tmp/input.log" > > >>>> Ruleset="relay.htp1" > > >>>> Tag="tag") > > >>>> > > >>>> > > >>>> Scenario 2: > > >>>> > > >>>> module(load = "mmnormalize") > > >>>> ruleset(name = "relay.htp1" queue.type="Direct") { > > >>>> action(type = "mmnormalize" > > >>>> ruleBase="/etc/rsyslog.d/service.rulebase" > > >>>> path="$!msg") > > >>>> * // $!msg will be available here even though action is backed by a > > >>>> default Queue. * > > >>>> } > > >>>> > > >>>> input(type="imfile" > > >>>> File="/tmp/input.log" > > >>>> Ruleset="relay.htp1" > > >>>> Tag="tag") > > >>>> > > >>>> > > >>>> How come $!var1, $.var2 aren't available in scenario1 whereas $!msg is > > >>>> available when both are using Direct Queue. Am I missing something > > >>>> here ? > > >>>> > > >>> _______________________________________________ > > >>> rsyslog mailing list > > >>> https://lists.adiscon.net/mailman/listinfo/rsyslog > > >>> http://www.rsyslog.com/professional-services/ > > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > >>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > > >>> POST if you DON'T LIKE THAT. > > >>> > > > _______________________________________________ > > > rsyslog mailing list > > > https://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com/professional-services/ > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of > > > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T > > > LIKE THAT. > > _______________________________________________ > > rsyslog mailing list > > https://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > > -- > Regards, > Rajesh KSV > _______________________________________________ > rsyslog mailing list > https://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
