On Thu, Feb 27, 2014 at 5:09 PM, Anwar El fatayri <
[email protected]> wrote:
> Is there a way to detect when the connection between the client and the
> server breaks down ?
> If this is possible, a simple code with the following algorithm does the
> job :
> If (the connection is down) {
> check if the size of spool files is bigger than X bytes ;
> if (bigger than X) compress;
> wait untill the connection is back again and then decompress; (Here
> rsyslog should be configured to delay the dequeuing process)
> }
> I just don't know how RELP detects that the connection with the server was
> lost to start saving data on disk and vice versa.
>
you need to look at ./runtime/queue.c. It is the rsyslog core that detects
this condition, omrelp just tells the core that it failed. Adding
compression should indeed not be too problematic, check the interaction
with ./runtime/stream.c. That class already supports compression, but the
queue doesn't use it. Also, the stream handler needs to be thought to do
decompression, what it currently does not do.
Patches are happily accepted.
Rainer
> Thanks.
> Anwar El Fatayri
>
>
>
> > Date: Wed, 26 Feb 2014 09:32:21 -0800
> > From: [email protected]
> > To: [email protected]
> > Subject: Re: [rsyslog] Compress/Decompress spool files in $WorkDirectory
> >
> > On Wed, 26 Feb 2014, Anwar El fatayri wrote:
> >
> > > Hey,At the moment, we have a centralized logging architecture which is
> not
> > > reliable (via Ossec) and this is why we're trying to use rsyslog - RELP
> > > instead of Ossec to reliably centralize our logs. However, we have
> more than
> > > 300 hosts per Datacenter that generate lots of logs. In case of any
> breakdown
> > > (server goes down for example), rsyslog clients will save all messages
> to disk
> > > ($WorkDirectory) and this could be really costly. I tried to compress
> spool
> > > files using logrotate. The problem is that these compressed files are
> not sent
> > > when the server is up again. So is it possible to compress spool files
> using
> > > rsyslog ?? If not, Can we configure rsyslog to decompress spool files
> before
> > > dequeuing ??
> >
> > Right now rsyslog does not deal with compressed spool files. The feeling
> is that
> > disk is cheap, and compressed files do not allow efficient random
> access, so we
> > want to have uncompressed access to the data.
> >
> > That said, I can see where it would be useful to be able to compress the
> > intermediate spool files (not the one being read, not the one being
> written to,
> > but any others that exist). But it's not currently implemented, and I
> think you
> > are the first to ask for it.
> >
> > David Lang
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
