solving this for the general case is more complicated, the logs may be going to
more than one desination from the queue that has the failed sender.
Also, compressing the logs is significantly more fragile if the system were to
go down while it'w writing in a compressed format. This is because the better
compression algorithms need to buffer a chunk of data before they compress is,
and the boundry of the chunk does not always match up with the end of a log
message. This can result in a file that's truncated in the middle of a message.
And if the system then restarts and writes more data to the file, the new data
can be lost because of the missing chunk.
David Lang
On Thu, 27 Feb 2014, Anwar El fatayri wrote:
Is there a way to detect when the connection between the client and the server
breaks down ?
If this is possible, a simple code with the following algorithm does the job :
If (the connection is down) {
check if the size of spool files is bigger than X bytes ;
if (bigger than X) compress;
wait untill the connection is back again and then decompress; (Here rsyslog
should be configured to delay the dequeuing process)
}
I just don't know how RELP detects that the connection with the server was lost
to start saving data on disk and vice versa.
Thanks.
Anwar El Fatayri
Date: Wed, 26 Feb 2014 09:32:21 -0800
From: [email protected]
To: [email protected]
Subject: Re: [rsyslog] Compress/Decompress spool files in $WorkDirectory
On Wed, 26 Feb 2014, Anwar El fatayri wrote:
Hey,At the moment, we have a centralized logging architecture which is not
reliable (via Ossec) and this is why we're trying to use rsyslog - RELP
instead of Ossec to reliably centralize our logs. However, we have more than
300 hosts per Datacenter that generate lots of logs. In case of any breakdown
(server goes down for example), rsyslog clients will save all messages to disk
($WorkDirectory) and this could be really costly. I tried to compress spool
files using logrotate. The problem is that these compressed files are not sent
when the server is up again. So is it possible to compress spool files using
rsyslog ?? If not, Can we configure rsyslog to decompress spool files before
dequeuing ??
Right now rsyslog does not deal with compressed spool files. The feeling is that
disk is cheap, and compressed files do not allow efficient random access, so we
want to have uncompressed access to the data.
That said, I can see where it would be useful to be able to compress the
intermediate spool files (not the one being read, not the one being written to,
but any others that exist). But it's not currently implemented, and I think you
are the first to ask for it.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
