The sender gets an application level ack from the receiver. so it knows that the
message got into the main queue on the receiver.
The message could be lost if the receiver crashes before it's written to
whatever outputs are configured. You can eliminate this window, but it means
eliminating all memory caching and queues, which lowers the performance by a
factor of 1000x or so. it's not only converting the queues to be 'direct' of
'disk only', it's doing a fsync after each disk I/O. Traditional syslog did this
(but without the application acknowledgement so it still had the network
issues), which is why it had a reputation of loosing messages once you started
getting close to 100 logs/sec (on 15k rpm drives, on 2600 rpm drives you would
run the risk of loosing data at around 25 logs/sec)
David Lang
On Thu, 12 Feb 2015, [email protected] wrote:
Hello David,
Thank you for these explainations.
Actually, in my current job we use keepalived for this.
I was wondering, this should lead to the same behavior if one of the remote
syslog server dies, no ?
The sender can't know the state of the message, as with my current failover
configuration.
Regards,
Smana
----- Mail original -----
De: "David Lang" <[email protected]>
À: "rsyslog-users" <[email protected]>
Envoyé: Mercredi 11 Février 2015 15:55:21
Objet: Re: [rsyslog] Relp failover : duplicate logs
On Wed, 11 Feb 2015, [email protected] wrote:
I'll think about a third part application for loadbalancing.
pacemaker can operate in two modes.
1. the IP shifts from one to the other.
In this case, your config would get simplified because you would not need to
have two outputs and the failover between them. You would only have one output
and if the server fails, you reconnect and hit the second box.
2. the IP is shared between them at the same time
This is load balancing, but it only load balances connections, not individual
logs. So you would need to have multiple systems sending or have the senders
reconnect every 1000 or 10000 messages to spread the load across the multiple
servers.
This is what you use when the back-end processing/storage of the logs is
significant and you need to distribute the load. In most cases this load is not
the resources rsyslog consumes, but rather whatever rsyslog is feeding the logs
into.
In either case, the big advantage of doing it on the receiving side is that the
sender doesn't need to know about it.
This simplifies the config on the sender (you don't need the server2 portion of
the config)
This also means that if you change things on the receiver side you don't have to
change the senders (other than that when you go to a load balanced config you
may want to set the rebind interval on the senders to make it balance more
evenly) If you change the number of servers receiving, build new ones and shift
load to them without taking the old ones down first, etc. It's all transparent
to the senders.
David Lang
Regards,
Smana
----- Mail original -----
De: "David Lang" <[email protected]>
À: "rsyslog-users" <[email protected]>
Envoyé: Mercredi 11 Février 2015 14:23:22
Objet: Re: [rsyslog] Relp failover : duplicate logs
On Wed, 11 Feb 2015, [email protected] wrote:
Is there a way to keep a failover architeture without duplicating logs please ?
unfortunantly no. If you work through exactly what can go wrong here, you will
see that there is no way to both ensure all logs are delivered and none are
duplicated without the sender tagging each message and the receiver then weeding
out duplicates
The problem is that the communications between the two systems is not
instantanious, you are using TCP, which will not loose data under normal
conditions, but will loose data under failure conditions. RELP adds application
level acknowlegements to the process, so that if the sender doesn't get an
acknowledgement from the receiving application it will assume the message didn't
get through and re-send it. But this means that if something goes wrong that
prevents these acknowlegements from getting through, the receiver will thing
they have the message, all properly acknowleged, but the sender will thing it
never got to the receiver and will re-send it, causing a duplicate.
In cases like this where the software exits without the ability to save any
state (kill -9, reboot, server failure, etc) the receiver has no way of saving
it's exact state, so when it comes up again it can't know what messages had been
received previously, so it can't tell that the messages it's receiveing are the
same as ones that it received before the outage.
You should also look into things like pacemaker http://clusterlabs.org/ that
will let you have multiple systems share a single IP address and fail it over if
one dies. This will allow you to reduce the outage from a system failure/reboot
to seconds.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
