Jochen, given the issues with remote storage for the journal... Do you recommend high volume nodes be physical? I run into issues with my receiving nodes where once a node has significant messages in the journal it can take considerable time to recover. Often during these periods the system in question will appear nearly idle, no io-wait, no significant cpu usage/etc. Output to elasticsearch during these times often falls to zero while incoming messages continue to flow into the journal. The odd thing is elasticsearch will be busy during this time but nowhere near fully utilized, 50-60% overall. Once in this state output from the graylog nodes will become very sporadic.
On Fri, Jan 6, 2017 at 7:01 AM, Jerri Son <[email protected]> wrote: > To share some more insights and describe how I managed to get Graylog to > perform better: > > Grok patterns are very costly as mentioned earlier so I went ahead and > looked for a solution that better > indexed those custom logs (in my case sidewinder firewall SEF-format) than > having an input with > ~ 12 different grok patterns and regexen - introduce logstash with "kv" > /key-values) filters: > > input { > syslog { > port => "50017" > tags => "sidewinder-logs" > } > > } > > filter { > if "sidewinder-logs" in [tags] { > kv { > prefix => "sw_" > field_split => "," > > } > } > } > > output { > if "sidewinder-logs" in [tags] { > gelf { chunksize => 2000 > host => "127.0.0.1" > port => 12205 > } > } > } > > This basically does all the indexing for us no matter how much fields vary > in our log messages. > Logstash seems to handle the field extraction very well without any major > CPU strains. With this setup > I achieved a very steady message I/O in our graylog setup as in buffers > almost never fill up beyond 10% (due to bursts) > which is what I was aiming for. > I will give graylog 2.2 a go once it is out to see if message throughput > will stay as good as it is now. > > Some more insights into my server.conf: > > output_batch_size = 2000 > output_flush_interval = 1 > processbuffer_processors = 6 > outputbuffer_processors = 3 > processor_wait_strategy = blocking > ring_size = 65536 > inputbuffer_ring_size = 65536 > inputbuffer_processors = 2 > inputbuffer_wait_strategy = blocking > message_journal_enabled = false > > Note for future "special" logs: I´m gonna check if logstash has more CPU > friendly ways to index and output with GELF :) > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Graylog Users" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/graylog2/FuoqCzAnuLQ/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/graylog2/d49d0558-45fb-4dcb-b154-aa0077231d3d%40googlegroups.com > <https://groups.google.com/d/msgid/graylog2/d49d0558-45fb-4dcb-b154-aa0077231d3d%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- CONFIDENTIALITY/EMAIL NOTICE: The material in this transmission contains confidential and privileged information intended only for the addressee. If you are not the intended recipient, please be advised that you have received this material in error and that any forwarding, copying, printing, distribution, use or disclosure of the material is strictly prohibited. If you have received this material in error, please (i) do not read it, (ii) reply to the sender that you received the message in error, and (iii) erase or destroy the material. Emails are not secure and can be intercepted, amended, lost or destroyed, or contain viruses. You are deemed to have accepted these risks if you communicate with us by email. Thank you. -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAB85xiyQhu_x8wc0WMh3AVajA8Gw_Qjo4mvouX9PFwJXR8etiw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
