> On Jan 16, 2015, at 12:25 PM, David Lang <[email protected]> wrote: > > On Fri, 16 Jan 2015, Dave Caplinger wrote: > >> ... It would be handy if we could optionally turn off [out-of-order >> delivery] >> for an output queue in order to deliver the queued messages in-order >> even if there is an additional disk write penalty to pay (for longer). > > One issue is that disk queues are very slow compared to memory queues, so > it's > possible that if you force all messages to be written to the queue while you > are > also pulling messages from the queue that this will slow you down so much > that > you will never catch up. I think there is room for improvement here, but that > would be pretty major surgery.
I understand; I would want to test things to really understand the performance penalty, but there are mitigating factors for some common cases as well. For example: filesystem buffer can help speed reading data previously written to disk if your outage was short enough to not get "too far" behind, because the data is still actually in RAM so you don't actually have to pay physical IOPS to touch the disk to retrieve it. Also, if the queue consumption rate is high and the reason for entering DAQ was a connectivity failure rather than the input rate overrunning the output rate, you should be able to leave DAQ mode relatively quickly. (Having stream compression on your output queue can help you reach a very high queue consumption rate even for relatively bandwidth-constrained remote destination.) These factors are why I was thinking maybe the penalty isn't really as large as I initially thought, for some cases at least. However, the fact that you indicate having this option would be "major surgery" to Rsyslog is dissuading me from wanting to bother going down this path. > I'll also point out that even without disk assisted queues, you can get > messages > out of order for several reasons. > > 1. UDP packets can pass each other 'on the wire' in a sufficiently complex > network. Can't control the first sender, but at least relay-to-relay we can make this TCP. (For ordering, not prevention of any possible loss.) > 2. since rsyslog processes messages in batches, when you have multiple > threads > working, thread 1 can grab messages 1-100 and a millisecond later thread 2 > can > grab messages 101-200 from the queue, message 101 will be sent long before > message 100 (possibly before message 2 gets processed, depending on the > ruleset) This kind of variation is acceptable for my case (see below), especially if the message rate is high (because the time variation between batches is low). > 3. If you have redundant relay systems, one may get delayed (it may go down > before relaying all it's messages and send them when it comes back up for > example) > > Even before rsyslog implemented batches and had the potential to send > messages > out of order, there were still conditions that could cause out-of-order > delivery. When I took the Simple Event Correlator class we were taught to not > do > if A followed by B followed by C then X and instead do if A set flagA, if B > set > flagB, if C set flagC, if flagA,flagB,flagC then X. To clarify, I'm not looking for *guaranteed* delivery order, just "generally in order." We do perform event correlation, but in some cases it's within time windows. So as you described: A followed by B followed by C, all within T time. Having some variation around a moving "now" pointer in time is fine; the events still wind up within the same (T +/- some small variation) -width window. It's when logs arrive *significantly* out of sequence that you wind up having to manage state for multiple T-width windows for the same scenario, and it means you can't really be confident that you're done with a certain time window (you can be perpetually waiting for the last event in the chain). It's certainly an edge case; normally connectivity interruptions are either "very brief" (absorbed by in-memory queue), or "short" (absorbed by DAQ for a few minutes/hours depending on log volume). But if they are very long, the time difference between the oldest and newest logs (which are being delivered in roughly alternating batches during the DAQ burn-down) can be quite large, like "yesterday, now, yesterday, now, yesterday..." -- Dave Caplinger, Director of Architecture | Ph: (402) 361-3063 | Solutionary — An NTT Group Security Company _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
