I can tell you that we are trying to use the following setup: - 2 'hot' nodes with 32GB RAM, 16-core and 3TB onboard storage RAID 1+0 - 10 'warm' nodes with 16GB RAM and only around 50GB of onboard storage (VMs) - 5 'cold' nodes, with 16GB RAM and nearly limitless NFS attached storage (VMs) - 1 nginx software based load balancer (VM same as the others)
All are running RHEL7. The goal is to make the 'hot' nodes face the brunt of the ingestion. We are looking at around 20k eps. The warm nodes will take the remainder of the ingestion. Hot and warm nodes will store indexes on their internal HD for up to 2 hours, in which they will age over to the cold nodes and move into NFS storage. I have all of the nodes nearly the same. The only thing that is different is the 2 'hot' boxes have more RAM assigned to ES and Graylog. As far as mods go, I'm still tinkering. I had to turn off firewalld, but that's fine because it's in an internal network. I originally had my syslog server sending sources directly to the graylog cluster, but the VMs were getting overwhelmed. So I then added the nginx load balancer to load balance the UDP traffic, and that helped. It just round robins among the sources. Other than that, it's pretty self explanatory. I run everything out of the tarballs (no RPMs) so that I can easily upgrade (RHEL can take a while for new supported RPMs). I install everything in /opt, and put everything in its own log destination in /var/log. Hope this helps some. Ask me more if you are curious about anything. Jas On Friday, December 9, 2016 at 8:49:08 AM UTC-6, BKeep wrote: > > As an interested nerd, would either or both of you be willing to share > some details about your environments and hardware setups? I'm always > curious about what other users of Graylog are doing. A couple of things > that would interest me are your hardware specs, what underlying OS, any OS > level modifications you are making, how many ES/GL nodes, how many log > clients, msg/s etc. > > Regards, > Brandon > > On Saturday, December 3, 2016 at 9:13:51 AM UTC-6, Dustin Tennill wrote: >> >> All, >> >> We just finished implementing >> https://www.elastic.co/blog/hot-warm-architecture >> <https://www.elastic.co/blog/hot-warm-architecture?blade=tw> for our >> Graylog environment. After weeks of troubleshooting elasticsearch >> performance issues with our budget ES nodes, the addition of a two small >> SSD nodes REALLY made a difference. Our output buffers had been filling up >> from time to time, and this appears to have resolved that issue. >> >> If anyone is interested, we will post our config information. >> >> Dustin Tennill >> EKU >> >> -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/7b8e9465-2de9-4e0e-9611-2f77e0c705ff%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
