Re: [rsyslog] Delete log files after sending logs to Elasticsearch

[David Lang]

On Mon, 19 Sep 2016, Alec Swan wrote:


I am using rsyslog to read log files and send logs to Elasticsearch. I am
looking for ideas on how to delete log files after their content is
successfully sent to Elasticsearch.


rsyslog doesn't have any options to remove a file when it's done processing it 
(rsyslog assumes that it's reading a file that something else is writing to).


What I would suggest is to use something like logger to read the file and 
deliver the lines to rsyslog, then remove the file after it's all been delivered 
to rsyslog.


David Lang


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Delete log files after sending logs to Elasticsearch

[Alec Swan]

Thanks for sharing your thoughts. I am not using Logstash and hence using
fingerprint or logstash plugins are not an option.

Alec

On Tue, Sep 20, 2016 at 1:42 AM, mosto...@gmail.com 
wrote:

> Working with Elasticsearch you could find many workarounds:
>
>  - ES allows you to "fingerprint" messages, so if this value matches with
> your log hash(event), you are safe to delete that event
>
>  - Using logstash RELP or eve ruby plugin to exec a "delete log line"
> after inserting it on ES
>
>  - With redis you could use push with queue (and deleting line) and
> rpoplpush + cron to check if events are already stored or should be queued
> again
>
>  - ...
>
>
> El 20/09/16 a las 06:10, Radu Gheorghe escribió:
>
> Hi Alec,
>>
>> A dirty one would be to have a cron job that:
>> - looks at the last modified date of a file
>> - queries Elasticsearch by timestamp. To see if the latest log from
>> that file got in. If it did (and maybe also if the Elasticsearch
>> snapshot completed successfully :p), the file can be deleted
>>
>> Best regards,
>> Radu
>> --
>> Performance Monitoring * Log Analytics * Search Analytics
>> Solr & Elasticsearch Support * http://sematext.com/
>>
>>
>> On Tue, Sep 20, 2016 at 12:01 AM, Alec Swan  wrote:
>>
>>> Hello,
>>>
>>> I am using rsyslog to read log files and send logs to Elasticsearch. I am
>>> looking for ideas on how to delete log files after their content is
>>> successfully sent to Elasticsearch.
>>>
>>> Thanks,
>>>
>>> Alec
>>> ___
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.