Hi BKeep, The Elasticsearch node is fine, that's on a separate box. The Graylog JVM settings are default, as I looked at increasing them but when I looked most people said to keep it the default. I don't think it's a memory issue as the memory usage is minimal, even a load times. I'm sure it's due to the fact the Graylog node only has 2 vCPUs but wanted to confirm that really.
The Memory usage has stayed below 50% the entire time. On Wednesday, 3 June 2015 14:40:30 UTC+1, BKeep wrote: > > I would add another vCPU core or two and some more RAM. Also, have you > adjusted your JVM HEAP size for Elasticsearch and Graylog? I have three > nodes setup for our logging cluster. The master node is Graylog, Mongod, > and Elasticsearch setup as master only with a JVM HEAP of 4g for Graylog > and 4g for Elasticsearch. The other two nodes are data storage only. Each > of those has 4vCPU's and 12GB of RAM where Elasticsearch has HEAP of 6g. > That configuration allows us max throughput of about 2500 logs per second > give or take. I should also note these are VMware VM's running on B200 > blade hardware (2.7Ghz Xeon cores, SAN storage, etc.) > > On 06/03/2015 03:21 AM, Matt Hines wrote: > > Hey Henrik, > > I'm not. That was the first thing that I found when looking into the > issue. We have 1 input, a syslog UDP, but the force_rdns is unticked. > > Thanks, > > On Tuesday, 2 June 2015 18:33:14 UTC+1, Henrik Johansen wrote: >> >> Hello Matt, >> >> Are you by any chance running inputs that have the force_rdns flag set? >> >> >> >> --- >> HenrikJ >> >> On 2. jun. 2015 kl. 17.37.16 CEST, Matt Hines <[email protected]> >> wrote: >> >> Hey guys, >> >> We've just updated our Graylog setup by moving the ES to another box to >> give it space to breathe. >> >> But now we're seeing a large increase in msg/s coming into the >> graylog-server, at peak times, between 600-1200. This amount is only >> going to get bigger as users increase. >> >> My issue is, when the number breaches the 600 msgs/s number, graylog >> starts to struggle and starts filling up its process buffer and the CPU >> ramps up to 100% and stays there until the peak period is over and it has >> cleared the buffer out. >> The box is an AWS box, 2CPU, 8GB RAM. I have no idea if this hardware >> should be enough for this number of messages? If so, is there some tweaking >> I can do to optimise the system? >> If not, what would you recommend, I know it's mainly CPU based but would >> it be best to cluster a second box, or increase this box? >> >> Also, at what point will I start to lose messages incoming? When the >> process buffer and journal are both full? >> >> We're running Graylog v1.0.2 >> >> Thanks for any help! >> -- >> You received this message because you are subscribed to the Google Groups >> "graylog2" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> >> -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > For more options, visit https://groups.google.com/d/optout. > > > -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
