Thanks for the information, very useful. Looking at the numbers you provided below, it looks like the changes you have made are doing the trick - load is well under 4 and memory is around 80% though you may need to add more memory in the future.
On Sunday, May 10, 2015 at 6:54:53 PM UTC+2, BKeep wrote: > > Currently, the log volume is about 1200 logs per second. I have an ES > cluster setup with two data storage nodes and an ES node that functions as > master only.. The master only node is installed on the same box as graylog, > graylog web, and mongodb. Adding the additional 2 vCPU's took care of the > problem for now but we plan to add logging for another 200+ devices so am > not sure where that leaves us. The current master node has 4 vCPU's, which > are XEON 2.7Ghz procs and 12 GB of RAM allocated to it. CPU load is -- load > average: 1.92, 2.08, 1.96 and memory usage is -- Mem: 12198260k total, > 9416592k used, 2781668k free, 258876k buffers. I changed the HEAP size > to -Xms4g -Xmx4g from the original setting of 1g. > > On Saturday, May 9, 2015 at 5:42:25 AM UTC-5, Matt Maloney wrote: >> >> How many messages are you ingesting per second? Zi's recommendation is >> right, you should consider running ES separately from Graylog. Moreover, >> you will want a load balancer in front of the Graylog servers. I imagine >> with the firewall logs being added into the mix, things have gotten a lot >> chattier hence the load issue. >> >> On Friday, May 8, 2015 at 3:31:27 AM UTC+2, BKeep wrote: >>> >>> We just setup our ASA to send logs into graylog and It appears the >>> graylog setup is not keeping up with the incoming log data. The server >>> currently has 8 GB of RAM and 2vCPU's. Up until the addition of the ASA, >>> that setup has been keeping up pretty well. However, it looks like i am >>> going to have to make some adjustments. Prior to adding the ASA logs we >>> were logging for about 130 servers. My question is what would be the next >>> steps? Obviously add some Memory resources and another couple of vCPU's but >>> once the ram is added are there any other adjustments to be made on the >>> server itself? >>> >>> Thanks in advance.. >>> >> -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
