Hello Matt, The short answer is that Graylog eats CPU and lot’s of it … and heaps more than competing solutions given the same message rate.
As an example, we’re seeing a performance of roughly 700 messages/sec per physical CPU core across our Graylog cluster(s) when keeping the utilisation across all 128+ CPU cores at ~75%. That’s purely for running our graylog-server nodes - ES runs on dedicated hardware. We do have a crazy amount of extractors and streams which certainly impact the numbers above and a crazy variation in terms of message size, etc so YMMV. Since you’re running on AWS I would not recommend using more than 4 vCPUS per instance - in fact I would recommend 2 vCPUS per instance and a suitable number of those for running multiple graylog-server nodes. There are so many factors involved when it comes to CPU scheduling in virtualised environments and unless you keep track of all the variables (host load, the hardware used, number of instances and their vCPU distribution, etc) you might get substantially lower performance when running with lots of vCPUS … — HenrikJ > On 03 Jun 2015, at 16:19, Matt Hines <[email protected]> wrote: > > Hi BKeep, > > The Elasticsearch node is fine, that's on a separate box. The Graylog JVM > settings are default, as I looked at increasing them but when I looked most > people said to keep it the default. I don't think it's a memory issue as the > memory usage is minimal, even a load times. I'm sure it's due to the fact the > Graylog node only has 2 vCPUs but wanted to confirm that really. > > The Memory usage has stayed below 50% the entire time. > > On Wednesday, 3 June 2015 14:40:30 UTC+1, BKeep wrote: > I would add another vCPU core or two and some more RAM. Also, have you > adjusted your JVM HEAP size for Elasticsearch and Graylog? I have three nodes > setup for our logging cluster. The master node is Graylog, Mongod, and > Elasticsearch setup as master only with a JVM HEAP of 4g for Graylog and 4g > for Elasticsearch. The other two nodes are data storage only. Each of those > has 4vCPU's and 12GB of RAM where Elasticsearch has HEAP of 6g. That > configuration allows us max throughput of about 2500 logs per second give or > take. I should also note these are VMware VM's running on B200 blade hardware > (2.7Ghz Xeon cores, SAN storage, etc.) > > On 06/03/2015 03:21 AM, Matt Hines wrote: >> Hey Henrik, >> >> I'm not. That was the first thing that I found when looking into the issue. >> We have 1 input, a syslog UDP, but the force_rdns is unticked. >> >> Thanks, >> >> On Tuesday, 2 June 2015 18:33:14 UTC+1, Henrik Johansen wrote: >> Hello Matt, >> >> Are you by any chance running inputs that have the force_rdns flag set? >> >> >> >> --- >> HenrikJ >> >> On 2. jun. 2015 kl. 17.37.16 CEST, Matt Hines <[email protected] <>> wrote: >>> Hey guys, >>> >>> We've just updated our Graylog setup by moving the ES to another box to >>> give it space to breathe. >>> >>> But now we're seeing a large increase in msg/s coming into the >>> graylog-server, at peak times, between 600-1200 <>. This amount is only >>> going to get bigger as users increase. >>> >>> My issue is, when the number breaches the 600 msgs/s number, graylog starts >>> to struggle and starts filling up its process buffer and the CPU ramps up >>> to 100% and stays there until the peak period is over and it has cleared >>> the buffer out. >>> The box is an AWS box, 2CPU, 8GB RAM. I have no idea if this hardware >>> should be enough for this number of messages? If so, is there some tweaking >>> I can do to optimise the system? >>> If not, what would you recommend, I know it's mainly CPU based but would it >>> be best to cluster a second box, or increase this box? >>> >>> Also, at what point will I start to lose messages incoming? When the >>> process buffer and journal are both full? >>> >>> We're running Graylog v1.0.2 >>> >>> Thanks for any help! >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "graylog2" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected] <>. >>> For more options, visit https://groups.google.com/d/optout >>> <https://groups.google.com/d/optout>. >> -- >> You received this message because you are subscribed to the Google Groups >> "graylog2" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. > > > -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
