On Fri, 23 Oct 2015, Tomas Heinrich wrote:
On 10/23/15 15:06, David Lang wrote:
On Thu, 22 Oct 2015, Tomas Heinrich wrote:
Hello,
I've been reading some RELP documentation[0] and it mentions a planned
capability to only acknowledge reception after the _output_ module of
the recipient acknowledges delivery.
I don't think this has been implemented yet but I was wondering
whether this is still on the roadmap and whether it is close to or far
from being completed.
I don't believe that this can be done reasonably with the current
architecture.
Right, that's what I think too. But the docs mention it and they are probably
pretty old, so I was wondering whether there was some provision conceived for
this use case - or whether this idea was abandoned (for the time being).
The problem is that the queues get in the way and the
input module can't easily know when the message has been fully processed
when it could end up going through multiple queues and get forked to
multiple copies in the process.
Yeah, the async processing is the problem in this case. And then there are
the partial successes and message duplication...
What you can do is to use a disk-only queue with syncing turned on and
checkpintinterval set to 1 (and IIRC another parameter or two) and this
will guarantee that the message won't be lost.
Well, this would ack the enqueuing and not the actual delivery. I think
anything else than a direct queue would be problematic from a reliability
POV.
Or you could set the main queue type to direct (and not use any action
queues), to have it processed immediately (I'm not sure if this would
quite do what you want, but it would be very close)
There could be even a stronger requirement - not only guaranteeing delivery
in an infinite time (over a chain of relays) but also signaling a failure and
aborting the sending attempts after a timeout so that the original sender
could perhaps fail-over to a different destination.
Unfortunantly, both of these solutions will absolutly kill performance
(by a factor of ~1000x or so)
Yes, with the current architecture this mode of operation seems unfeasible.
Thanks for you thoughts David.
absolute guaranteed delivery through multiple relays is especially problematic.
What do you do if the log message goes to multiple destinations and some work
and others don't?
Do you really want to freeze the sending box if there is a problem further down
the line that delays the delivery?
If you really want absolute guaranteed delivery to a single destination, you are
looking for a database that provides ACID guarantees, not a general purpose
logging system. There are good reasons why people don't put logs in databases,
with performance being one of the biggest ones.
There are some 'log' solutions that claim to give you end-to-end acknowlegement,
but if you look into them, you find that performance suffers and your app needs
to keep track of a LOT of outstanding state because it can take long enough for
the ack to come back that you can end up with a LOT of data in flight.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
