Though this goes back to my original problem - when replacing the log
server, I don't want to have to log in to each server running rsyslog
to HUP or restart the rsyslog daemon. Just knowing which machines I
need to log into is too much of a pain.
On ב', אוג 18, 2014 at 9:13 PM, David Lang <[email protected]> wrote:
a kill -HUP also tells rsyslog to close all files and outbound
connections, and only re-open them as needed, a cron log rotation
script should do a HUP as part of it's process.
David Lang
On Mon, 18 Aug 2014, Oded Arbel wrote:
The target is not a load balancer - quite the opposite, its a single
server that every few days/weeks will be replaced by a new server.
The
"RebindInterval" option might be a good idea, if not for:
1. Your 2nd issue from below: how can I force syslog to re-resolve
the
IP address of the server
2. The log may be idle or mostly idle for long periods (night times,
for example, or weekends) - setting X to a low number will decrease
performance while setting it to a high number might cause logs to be
missing for a long time when replacing the server. A time based
configuration would have worked much better here.
On Mon, Aug 18, 2014 at 1:55 PM, David Lang <[email protected]> wrote:
If you are sending logs to a load balancer, it's a good idea to set
the option to reconnect every X messages (set X to some largish
number for efficiency, 1000 or so is a good starting point)
see if that solves your problem
there are two possible issues here
1. the sending machine never detects that the tcp session is dead
2. that when this happens, it retries using the same IP for the
same name
David Lang
On Mon, 18 Aug 2014, Oded Arbel wrote:
Hi Radu, thanks for the response.
I've ran rsyslog on one of the machines, as you specified below.
After that
I redeployed my logging server and changed the IP. There was no
change in
the behavior or syslog - I still get the same logs before and
after the
change. The logs are a little verbose, so here's a snippet of what
I see
regarding my omfwd action (this is from after the change, but it
was
looking identical before - except for the timestamp):
5545.609123832:7f1331cdf700: actionCommitAll: action 4, state 1,
nbr to
commit 0 isTransactional 1
5545.609128757:7f1331cdf700: doTransaction: have commitTransaction
IF,
using that, pWrkrInfo 0x196d090
5545.609133137:7f1331cdf700: entering
actionCallCommitTransaction(), state:
itx, actionNbr 4, nMsgs 1
5545.609137473:7f1331cdf700: logging.server
5545.609142140:7f1331cdf700: logging.server:5545/tcp
5545.609147889:7f1331cdf700: omfwd: add 297 bytes to send buffer
(curr offs
0)
5545.609152831:7f1331cdf700: omfwd: endTransaction, offsSndBuf
297, iRet
-2121
5545.609183045:7f1331cdf700: omfwd: TCP sent 297 bytes, requested
297
5545.609188736:7f1331cdf700: Action 4 transitioned to state: rdy
5545.609193196:7f1331cdf700: omfwd: beginTransaction
5545.609197183:7f1331cdf700: logging.server
5545.609201276:7f1331cdf700: Action 4 transitioned to state: itx
5545.609205543:7f1331cdf700: Action 4 transitioned to state: rdy
5545.609209770:7f1331cdf700: actionCommit, in retry loop, iRet 0
Another thing I figured out - which makes me more troubled about
the
apparent change of behavior between rsyslog 5.x and 8.x - is that
when I
deploy a new logging server and assign the elastic IP to it, the
IP that
rsyslog sees does change: all machines are running on EC2 and so
address
each other using EC2's "private IPs" (a class A non-routable
network),
while the elastic IP is a public IP mapped to the actual internal
IP of the
logging server.
So the behavior can be explained by rsyslog not doing DNS lookups
for its
remote targets: once it starts and resolves the logging.server
name to the
intenal IP, it keeps sending there, even after the elastic IP has
moved.
While the old machine is still running (because moving the elastic
IP does
not change its private IP, nor terminates it) the logs keep
showing up on
the old machine. Once the old machine is turned off, I can see
rsyslog
reopening the omfwd connection and everything starts working again.
A solution for my setup might be a setting in rsyslog to either
always
re-resolve the DNS record before submitting a new message, or at
least
occasionally refresh the cached DNS result (every 60 seconds or
so).
From: Radu Gheorghe <[email protected]>
Hi Oded,
I've never seen this issue, but maybe you can see something in
the debug
log?
To get the debug log, I usually start rsyslog with something like:
# rsyslogd -dn > /var/log/rsyslog.log
Best regards,
Radu
On Mon, Aug 11, 2014 at 4:08 PM, Oded Arbel <[email protected]>
wrote:
Hi list,
I've been using the rsyslog version 5.8 delivered with Ubuntu
12.04
(which my servers are currently running with) to forward message
to a
central logstash server over TCP, and that worked fine - the
forwarding part, I had problems with multiline messages and other
formatting issues, so I upgraded to rsyslog 8 and started to use
the
omfwd module with a complex template - and now those problems are
solved, but I have a new one.
The central logging server is on EC2 with an elastic IP and I
update
its configuration from time to time by basically building a new
server
and moving the elastic IP. With the old 5.8 installation, this
used to
work fine - as soon as the elastic IP moved, all the rsyslog
daemons
would lose the connection to the server and reconnect. With
version
8's omfwd, this doesn't happen - rsyslog doesn't notice the
connection
dropping and doesn't reconnect. I have to log in to each servers
and
restart the service to get it to deliver messages again.
There are messages sent to the rsyslogds during that time, and
these
are logged to the local files correctly, but not delivered to the
central server (neither the old nor the new one). Also, nothing
about
this is logged to the local syslog file.
Please advise?
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
