On Fri, Feb 28, 2014 at 6:02 AM, David Lang <[email protected]> wrote:
> solving this for the general case is more complicated, the logs may be
> going to more than one desination from the queue that has the failed sender.
>
> Also, compressing the logs is significantly more fragile if the system
> were to go down while it'w writing in a compressed format. This is because
> the better
> compression algorithms need to buffer a chunk of data before they compress
> is, and the boundry of the chunk does not always match up with the end of a
> log message. This can result in a file that's truncated in the middle of a
> message. And if the system then restarts and writes more data to the file,
> the new data can be lost because of the missing chunk.
>
>
yeah, that's right. My bad, I thought that's so obvious I didn't even
mention it. In any case, it can be mitigated by writing smaller compression
blocks. This is what omfile does and - IIRC - that's implemented inside the
file stream class layer.
Rainer
> David Lang
>
>
> On Thu, 27 Feb 2014, Anwar El fatayri wrote:
>
> Is there a way to detect when the connection between the client and the
>> server breaks down ?
>> If this is possible, a simple code with the following algorithm does the
>> job :
>> If (the connection is down) {
>> check if the size of spool files is bigger than X bytes ;
>> if (bigger than X) compress;
>> wait untill the connection is back again and then decompress; (Here
>> rsyslog should be configured to delay the dequeuing process)
>> }
>> I just don't know how RELP detects that the connection with the server
>> was lost to start saving data on disk and vice versa.
>> Thanks.
>> Anwar El Fatayri
>>
>>
>>
>> Date: Wed, 26 Feb 2014 09:32:21 -0800
>>> From: [email protected]
>>> To: [email protected]
>>> Subject: Re: [rsyslog] Compress/Decompress spool files in $WorkDirectory
>>>
>>> On Wed, 26 Feb 2014, Anwar El fatayri wrote:
>>>
>>> Hey,At the moment, we have a centralized logging architecture which is
>>>> not
>>>> reliable (via Ossec) and this is why we're trying to use rsyslog - RELP
>>>> instead of Ossec to reliably centralize our logs. However, we have more
>>>> than
>>>> 300 hosts per Datacenter that generate lots of logs. In case of any
>>>> breakdown
>>>> (server goes down for example), rsyslog clients will save all messages
>>>> to disk
>>>> ($WorkDirectory) and this could be really costly. I tried to compress
>>>> spool
>>>> files using logrotate. The problem is that these compressed files are
>>>> not sent
>>>> when the server is up again. So is it possible to compress spool files
>>>> using
>>>> rsyslog ?? If not, Can we configure rsyslog to decompress spool files
>>>> before
>>>> dequeuing ??
>>>>
>>>
>>> Right now rsyslog does not deal with compressed spool files. The feeling
>>> is that
>>> disk is cheap, and compressed files do not allow efficient random
>>> access, so we
>>> want to have uncompressed access to the data.
>>>
>>> That said, I can see where it would be useful to be able to compress the
>>> intermediate spool files (not the one being read, not the one being
>>> written to,
>>> but any others that exist). But it's not currently implemented, and I
>>> think you
>>> are the first to ask for it.
>>>
>>> David Lang
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
