Patrick,
I've given up on ELK (ElasticSearch/Logstash/Kibana), and I'm moving to Splunk. Amavisd-new ability to log in JSON format is a very great feature, and I would like to be able to pipe my JSON logs to Splunk.
[...]
Joolee wrote:
It wouldn't be that hard to create a plugin for that using the amavis custom hooks api.
Obtaining a JSON report by a custom hook is possible, although it would miss some last-minute updates, e.g. on the timing report, as it is not run late enough in the processing. You'd still need a way to merge reports from multiple concurrent child processes into a single stream, which involves locking or some other approach (e.g. message passing). Also some queuing is desired to decouple feeders from consumers. Using a file as an intermediate medium to feed Splunk seems like a poor choice. 2014-10-05 20:17, Jernej Porenta wrote:
a while ago, Mark Martinec wrote a script that pulls Redis logs out to standard output, which can be easily fed into splunk. With a little help of a skilled perl programmer, I am totally sure you can extend attached script to do whatever you want ;)
Indeed, my little program offers all that: locking and queuing is handled by Redis, so the consumer process (e.g. Splunk) would be nicely decoupled from amavisd. Even better would be to persuade Splunk folks to provide an input module to pull JSON records from a Redis queue directly. Mark
