what do the impstats look like?
with tcp forwarding, if there is a problem connecting to the destination the
queue will fill.
If the queue is filling up on the sender, then it means that it's not getting
through to the receiver. If the connection is idle for a bit, it could be that a
firewall or NAT device is timing out the connection and then blocking it (and
iptables on the receiver could be that firewall). a tcpdump at both ends when
there is a problem would probably be very useful.
I would suggest a much smaller rebindInterval, 5m messages is a lot.
each time rebindinterval messages are sent, rsyslog will close and re-open the
connection. This has an overhead, but it's not that large, so something in the
low thousands should be good (a rebind every second or a few rebinds a second is
not an issue). Note that this may affect the effective compression that you end
up with if it doesn't compress the first messages as efficiently as later ones.
But even with that 5m messages is large.
David Lang
On Mon, 17 Nov 2014, chenlin rao wrote:
Date: Mon, 17 Nov 2014 17:49:56 +0800
From: chenlin rao <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: [rsyslog] help for sudden buffer size
I have a 100 servers using rsyslog v7 sending messages to a VIP which has
10 really rsyslog servers. And I use impstats to monitor them.
every node has 300k msgs "enqueued" per 5 minutes. But some node, 10 or so,
may has 10~200k msgs "size" per 5 minutes! If I restart rsyslogd on one
node, it would be fine for several hours, or days maybe. But during the
time, another node may run into such buffering situation...
I found large netstat send-Q at my nodes, but no recv-Q at rsyslog server.
But have no idea to check such queue at LVS server. So, how can I determine
the root cause? Or which configuration should I change to solve this?
My omfwd configure as follow:
action (
type="omfwd"
Target="172.16.26.249"
Port="514"
Protocol="tcp"
RebindInterval="5000000"
ZipLevel="3"
compression.stream.flushOnTXEnd="off"
name="action_webInfoLog-yf-logs-vip.weibo.cn-514_wwwlog"
queue.filename=
"action_webInfoLog-yf-logs.vip.weibo.cn-514_wwwlog"
queue.size="300000"
queue.dequeuebatchsize="1000"
queue.maxdiskspace="20G"
queue.discardseverity="3"
queue.checkpointinterval="10"
queue.type="linkedlist"
queue.workerthreads="1"
queue.timeoutshutdown="10"
queue.timeoutactioncompletion="10"
queue.timeoutenqueue="20"
queue.timeoutworkerthreadshutdown="10"
queue.workerthreadminimummessages="15000"
queue.maxfilesize="500M"
queue.saveonshutdown="on"
)
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
