On Tue, 10 Feb 2015, Chastity Blackwell wrote:
On Tue, 2015-02-10 at 08:33 -0600, Dave Caplinger wrote:
I'm surprised you need to bother with the HUP if you are using dynafiles
to write to distinct quarter-hour output files. Does lsof indicate that
rsyslog really does still have those older files open? Even so, I wouldn't
expect the extra HUP once every 15 minutes to have much of an impact unless
you have very high log volume since the HUP would pause rsyslog briefly to
flush all output queues and close/reopen files. (Maybe HUP doesn't flush
the queues; I don't recall...?)
I don't believe that HUPing flushes the queues; all it does is close
open filehandles. And yes, with the dynamic file naming, rsyslog will
keep a file open for quite a while (if not indefinitely), because there
is no process that "knows" that file will never get written to again
(dynamic file naming is not aware of chronology, as far as I can tell,
only that it makes a decision based on arbitrary conditions, one of
which can be time). The HUPing is not causing the problem in this case.
HUP does not flush the queue, it closes the connections. If the messages have
been written and is waiting to be flushed to disk the HUP will flush the
partially written messages to disk before closing the filehandle.
What I suspect is more likely is that something else is contending for
bandwidth over the link the VPN uses. If you can catch it in the act, does
running iperf between the two sites reveal lower throughput? (You'll need
to do a similar test outside of the problem window as a baseline of course.)
This is what I'm guessing as well.
Our solution to the "remote sender with low outbound bandwidth" issue was to
1) make sure we weren't forwarding things that we really didn't need, and 2)
enable stream compression. We initially set up a 60-second compressed
batching solution using output dynafiles from rsyslog that were gzip'd and
transferred by another process (then consumed on the other side, uncompressed,
and re-injected into the log stream), and it was effective. However, that
solution was complex (with many moving parts that could break), and stream
compression in rsyslog performs even better than this, especially for long-lived
TCP connections between the two rsyslog nodes.
We were aware that we were reaching the limits of our VPN, so we've
already got plans to switch to another solution, in this case shipping
logs over the open internet (where we have a 10 Gbit connection, as
opposed to the VPN's 1 Gbit) using rsyslog with TLS, but that is not
something we are quite ready to implement yet. I'll likely need to
escalate that on the priority list in light of this problem.
even on that connection, compression is a good idea,
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
