Thanks Henrik, I will keep this in mind when deciding. Might switch the instance out to the CPU optimised one with 4 vCPUs and see how that copes, then scale horizontally out after that.
Thanks all for the input. On Wednesday, 3 June 2015 20:38:25 UTC+1, Henrik Johansen wrote: > > Hello Matt, > > The short answer is that Graylog eats CPU and lot’s > of it … and heaps more than competing solutions given > the same message rate. > > As an example, we’re seeing a performance of roughly > 700 messages/sec per physical CPU core across our > Graylog cluster(s) when keeping the utilisation across > all 128+ CPU cores at ~75%. > > That’s purely for running our graylog-server nodes - ES > runs on dedicated hardware. > > We do have a crazy amount of extractors and streams > which certainly impact the numbers above and a crazy > variation in terms of message size, etc so YMMV. > > Since you’re running on AWS I would not recommend > using more than 4 vCPUS per instance - in fact I would > recommend 2 vCPUS per instance and a suitable number > of those for running multiple graylog-server nodes. > > There are so many factors involved when it comes to CPU > scheduling in virtualised environments and unless you keep > track of all the variables (host load, the hardware used, number > of instances and their vCPU distribution, etc) you might get > substantially lower performance when running with lots of > vCPUS … > > — > HenrikJ > > > On 03 Jun 2015, at 16:19, Matt Hines <[email protected] <javascript:>> > wrote: > > Hi BKeep, > > The Elasticsearch node is fine, that's on a separate box. The Graylog JVM > settings are default, as I looked at increasing them but when I looked most > people said to keep it the default. I don't think it's a memory issue as > the memory usage is minimal, even a load times. I'm sure it's due to the > fact the Graylog node only has 2 vCPUs but wanted to confirm that really. > > The Memory usage has stayed below 50% the entire time. > > On Wednesday, 3 June 2015 14:40:30 UTC+1, BKeep wrote: >> >> I would add another vCPU core or two and some more RAM. Also, have you >> adjusted your JVM HEAP size for Elasticsearch and Graylog? I have three >> nodes setup for our logging cluster. The master node is Graylog, Mongod, >> and Elasticsearch setup as master only with a JVM HEAP of 4g for Graylog >> and 4g for Elasticsearch. The other two nodes are data storage only. Each >> of those has 4vCPU's and 12GB of RAM where Elasticsearch has HEAP of 6g. >> That configuration allows us max throughput of about 2500 logs per second >> give or take. I should also note these are VMware VM's running on B200 >> blade hardware (2.7Ghz Xeon cores, SAN storage, etc.) >> >> On 06/03/2015 03:21 AM, Matt Hines wrote: >> >> Hey Henrik, >> >> I'm not. That was the first thing that I found when looking into the >> issue. We have 1 input, a syslog UDP, but the force_rdns is unticked. >> >> Thanks, >> >> On Tuesday, 2 June 2015 18:33:14 UTC+1, Henrik Johansen wrote: >>> >>> Hello Matt, >>> >>> Are you by any chance running inputs that have the force_rdns flag set? >>> >>> >>> >>> --- >>> HenrikJ >>> >>> On 2. jun. 2015 kl. 17.37.16 CEST, Matt Hines <[email protected]> >>> wrote: >>> >>> Hey guys, >>> >>> We've just updated our Graylog setup by moving the ES to another box >>> to give it space to breathe. >>> >>> But now we're seeing a large increase in msg/s coming into the >>> graylog-server, at peak times, between 600-1200. This amount is only >>> going to get bigger as users increase. >>> >>> My issue is, when the number breaches the 600 msgs/s number, graylog >>> starts to struggle and starts filling up its process buffer and the CPU >>> ramps up to 100% and stays there until the peak period is over and it has >>> cleared the buffer out. >>> The box is an AWS box, 2CPU, 8GB RAM. I have no idea if this hardware >>> should be enough for this number of messages? If so, is there some tweaking >>> I can do to optimise the system? >>> If not, what would you recommend, I know it's mainly CPU based but would >>> it be best to cluster a second box, or increase this box? >>> >>> Also, at what point will I start to lose messages incoming? When the >>> process buffer and journal are both full? >>> >>> We're running Graylog v1.0.2 >>> >>> Thanks for any help! >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "graylog2" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "graylog2" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> >> >> > -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > For more options, visit https://groups.google.com/d/optout. > > > -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
