Thanks for sharing your thoughts. I am not using Logstash and hence using fingerprint or logstash plugins are not an option.
Alec On Tue, Sep 20, 2016 at 1:42 AM, mosto...@gmail.com <mosto...@gmail.com> wrote: > Working with Elasticsearch you could find many workarounds: > > - ES allows you to "fingerprint" messages, so if this value matches with > your log hash(event), you are safe to delete that event > > - Using logstash RELP or eve ruby plugin to exec a "delete log line" > after inserting it on ES > > - With redis you could use push with queue (and deleting line) and > rpoplpush + cron to check if events are already stored or should be queued > again > > - ... > > > El 20/09/16 a las 06:10, Radu Gheorghe escribió: > > Hi Alec, >> >> A dirty one would be to have a cron job that: >> - looks at the last modified date of a file >> - queries Elasticsearch by timestamp. To see if the latest log from >> that file got in. If it did (and maybe also if the Elasticsearch >> snapshot completed successfully :p), the file can be deleted >> >> Best regards, >> Radu >> -- >> Performance Monitoring * Log Analytics * Search Analytics >> Solr & Elasticsearch Support * http://sematext.com/ >> >> >> On Tue, Sep 20, 2016 at 12:01 AM, Alec Swan <alecs...@gmail.com> wrote: >> >>> Hello, >>> >>> I am using rsyslog to read log files and send logs to Elasticsearch. I am >>> looking for ideas on how to delete log files after their content is >>> successfully sent to Elasticsearch. >>> >>> Thanks, >>> >>> Alec >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
