On 2015-08-03 08:40, Benning, Markus wrote:
Am 2015-07-26 16:52, schrieb Phil Daws:
what would be the most suitable way of generating JSON from
amavisd-new for injecting into ES using td-agent ?
Hello Phil,
i wrote some words about it on my blog:
https://markusbenning.de/blog/?p=10
The logging output is very verbose. I posted a patch to filter
report_json output last year:
http://lists.amavis.org/pipermail/amavis-users/2014-December/003371.html
I dont know td-agent, but i used logstash and now saftpresse to get
data from log to ES.
Markus
Don't know about td-agent either, but the most reliable and
efficient way to get JSON logging from amavisd is through
a redis server. Redis is used as a queue, so it can smooth out
any inrush of events, or weather over outages of a log service.
Trying to do the same through a regular text log (syslog or
stderr) is IMO a wrong tool for the job. Decoupling an event
produced from a logger seems beneficial, whereas synchronously
logging can bog down the event producer.
Collecting entries from a redis server is fairly trivial,
so it can be implemented in any language with little work.
Even logstash is perfectly capable of reading JSON entries
from a redis server, and feeding them to Elasticsearch :
amavisd.conf:
@storage_redis_dsn = ( { server => '[::1]:6379', db_id => 1 } );
$redis_logging_key = 'logstash-amavis';
$redis_logging_queue_size_limit = 300000;
# takes about 250 MB of redis memory per 100000 log entries
logstash:
input {
redis {
type => "amavis"
host => "::1"
db => 1
data_type => "list"
key => "logstash-amavis"
codec => json {}
}
}
output {
elasticsearch {
host => "127.0.0.1"
port => "9200"
protocol => "http"
template_overwrite => true
index_type => "%{type}"
codec => json {}
idle_flush_time => 5
flush_size => 1000
}
}
If logstash is not an option, I have a sample perl program
to pull JSON log entries from a queue on a Redis server, and
write them to stdout (so it can feed a system like Splunk,
which may not have a plugin for reading from a Redis server).
Available on request.
Mark
