On Thu, 20 Jan 2011, Dra?en Ka?ar wrote:
Robert Jennings wrote:
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Dra?en Kacar
How does your failover procedure work exactly? If it's crashing the OS
(or turning off power), then there's nothing which can send TCP close
packet, so your clients won't know they should close the connection.
If that's the case, there should be retransmitting TCP timeout on the
client, which is usually 13-15 minutes by default. The exact number
depends on the OS and the network MTU. Did you wait that long?
The failover simulates a network link going down, so effectively the
network cable is just yanked.
In that case TCP FIN packets can't get to the client, so you get the
behavior you're describing.
FIN packets don't get there, but once the new box is up, it should be
sending a RESET packet
David Lang
I have switched to using UDP which is working fine, I never waited a
full 15minutes but that is an unacceotable period of downtime.
Sure, but that timeout is configurable on the OS level. Write timeouts
should be configurable on the application level (in this case that's the
client rsyslog you're using), but I don't know if your version has that
feature. Check the documentation.
And there's an option of using TCP keepalive (which also has configurable
timeouts), so client can check for the availability of server. I do not
know if rsyslog can set keepalive option on its outgoing TCP connections,
though. But that can be easily added. It can be added even if the
executable you're using doesn't support it.
Is it known how RELP would behave in a heartbeat failover scenario? I
have yet to test it as im running the stock rhel5 version of rsyslog
3.22
It should be able to detect that the remote server is unavailable when it
tries to send the first message under those conditions. I haven't tried
it, though.
With UDP behaviour is as expected, with a ~20-30 second failover period
where messages are lost (settings not tweaked in anyway)
I am using 3.22 simply as it is shipped with RHEL5/CENTOS5, but would be
interest to try a more recent version.
Which leads me to think perhaps the best solution may be:
Rsyslog clients UDP -> prd-syslog-000
Syslog clients & devices UDP -> prd-syslog-000
Prd-syslog-002 TCP -> prd-syslog-001
So prd-syslog-002 is only ever hit when prd-syslog-001 is down, and can
reliably queue and push messages to prd-syslog-001 once it is back
online.
Thanks,
Rob
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of [email protected]
Sent: Wednesday, 19 January 2011 2:00 PM
To: rsyslog-users
Subject: Re: [rsyslog] rsyslog with heartbeat
how long did you wait for the failover to take place? the first thing
that
will have to happen is that the sender will have to notice that the TCP
connection doesn't work anymore, close it, and open another one.
Since the system doesn't know what messages have actually been sent to
the
server (as opposed to mearly sent to the TCP stack on the sending
machine), there will be some logs lost at failover time. the longer it
takes the client to notice the problem, the worse this will be.
version 3.22 is fairly old at this point, Rainer will have to weigh in
on
how long it should take to notice the failure.
to fully avoid the lost messages issue, you would need to use RELP for
your transport.
I have opted to use UDP and accept that while the boxes are failing
over,
some logs will be lost (you can tune heartbeat for pretty short failover
times, subsecond if you really push it)
David Lang
On Wed,
19 Jan 2011, Robert Jennings wrote:
I am running 2 rsyslog servers (prd-syslog-001, prd-syslog-002)
sharing
a virtual IP managed by heartbeat (prd-syslog-000)
clients are configured to send logs to the virtualIP
*.info;mail.none;authpriv.none;cron.none
@@prd-syslog-000
authpriv.*
@@prd-syslog-000
local7.*
@@prd-syslog-000
I have found when the primary host goes down and the secondary takes
over, logs are not arriving at the secondary host unless I restart
rsyslog on the client, this was not the behaviour I was expecting. Is
there any default configuration that may be stopping logs from being
sent when the server sitting behind the virtual ip changes?
The functionality I was trying to achive is as follows:
* clients log to prd-syslog-000
* heartbeat controls prd-syslog-000 address (prd-syslog-001
primary, prd-syslog-002 on failover)
* prd-syslog-002 is configured to forward logs to prd-syslog-001
* during downtime of prd-syslog-001 these logs will sit queued
* when prd-syslog-001 comes back up it will take over
prd-syslog-000 and start recieving client logs
* prd-syslog-002 will flush its queue back to prd-syslog-001
The reason I am using this setup as opposed to logging to two rsyslog
hosts is to support devices that can only log to one host, and provide
one location with a consistent set of logs instead of two machines
with
out of sync logs.
running rsyslog-3.22
Thanks,
Rob
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
--
.-. .-. Yes, I am an agent of Satan, but my duties are largely
(_ \ / _) ceremonial.
|
| [email protected]
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
