To share some more insights and describe how I managed to get Graylog to
perform better:
Grok patterns are very costly as mentioned earlier so I went ahead and
looked for a solution that better
indexed those custom logs (in my case sidewinder firewall SEF-format) than
having an input with
~ 12 different grok patterns and regexen - introduce logstash with "kv"
/key-values) filters:
input {
syslog {
port => "50017"
tags => "sidewinder-logs"
}
}
filter {
if "sidewinder-logs" in [tags] {
kv {
prefix => "sw_"
field_split => ","
}
}
}
output {
if "sidewinder-logs" in [tags] {
gelf { chunksize => 2000
host => "127.0.0.1"
port => 12205
}
}
}
This basically does all the indexing for us no matter how much fields vary
in our log messages.
Logstash seems to handle the field extraction very well without any major
CPU strains. With this setup
I achieved a very steady message I/O in our graylog setup as in buffers
almost never fill up beyond 10% (due to bursts)
which is what I was aiming for.
I will give graylog 2.2 a go once it is out to see if message throughput
will stay as good as it is now.
Some more insights into my server.conf:
output_batch_size = 2000
output_flush_interval = 1
processbuffer_processors = 6
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = false
Note for future "special" logs: I´m gonna check if logstash has more CPU
friendly ways to index and output with GELF :)
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/d49d0558-45fb-4dcb-b154-aa0077231d3d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
