Re: [graylog2] Extractors with Wildcards Cause High CPU/Load Average

2015-06-14 Thread Pete GS
Just wanted to post a follow up to this... I've finally gotten my head around Grok patterns and how to use these with extractors and I have replaced all my extractors with only two to achieve the same set of extractions. Load average on the Graylog servers is now in the 0.08 - 0.12 range while

Re: [graylog2] Extractors with Wildcards Cause High CPU/Load Average

2015-06-05 Thread Pete GS
Ah thanks Kay! I've never looked into Grok patterns, but that sounds like they could help a great deal. As you've pointed out in my extractors, there's only a very small number of specific log lines I need to identify and these contain all the fields I wish to extract relating to the potential

Re: [graylog2] Extractors with Wildcards Cause High CPU/Load Average

2015-06-04 Thread Kay Röpke
Pete, The extractors themselves do not look too bad, but however whenever you use leading wildcards to extract similar data, the work that the extractors have to do is repeated, since they are executed one after the other. If there's no better way to extract that data, you might want to look into

[graylog2] Extractors with Wildcards Cause High CPU/Load Average

2015-06-04 Thread Pete GS
Hi all, I've finally discovered the source of my excess CPU load and high load averages on my Graylog nodes! I've got a bunch of extractors that I use to pull information from my vSphere platform's VMKernel logs. The catch with these is that a lot of items in the message string vary quite a b