Asad, I use the indication @ found as a directive (https://www.found.no/pricing/#pricing/availability_zones=2&capacity=16384®ion=us-east-1&ssd=false) So with 16GB of memory you could store 128GB od data on a server. My guess is that you can double that amount in case of replication use.
I see that on one server you use Graylog and ES together what is not very wise in my opinion. You can't configure a HEAP size of 50% for example, set this to 7GB. On my single test server this helped me out. Is is best to run only ES on a node, and nothing else. ES will not be able to automatize it's processes at best possible I think. On Monday, June 9, 2014 2:37:52 AM UTC+2, Asad Mehmood wrote: > > Good day! > > Recently I started implementing log monitoring and analysis system using > graylog2, we will have around 12,000 message / second. Though in staging we > are not even near that number but the cluster is not stable. > > Sometimes ES discovery fails because either the PC is in I/O wait or there > are too many processes in each core. > I tried to tune the settings by one way or another the cluster finds a way > to fail, as for my setup there are some limitation for a a while to use > high speed I/O so I need to either stick with slow disks or divide the > setup in a way that recent logs remain in high speed disks and older are > moved to low performance cluster. I was hoping if someone can help me > formulate or calculate a formula to decide how many nodes I need for ES > cluster, graylog2-server, radio and Kafka. > > There is another problem with KAFKA input if i shutdown Kafka, zookeeper > or radio, the messages stop coming and I need to Terminate Kafka input and > Launch a new input. > Also the message throughput while using KAFKA and Radio is far less than > using direct inputs with graylog2-benchmark tool. > > Current Setup > 2 Nodes for Log Collector and Radio (8 Gb, 2 Core Xeon ) > 1. Graylog2-server + graylog2-web (16 Gb, 4 Core Xeon ) > 1. Graylog2-server + elasticsearch (16 Gb, 4 Core Xeon ) > 3. Elasticsearch + Kafka Node (16 Gb, 4 Core Xeon ) > > The message throughput in peak hours will be 12000 / second and to > implement this system in production, the system needs to withstand stress > test of 20.000 message / second. > > I will appreciate if anyone here can help me with formulating the > performance requirements by quantifying them. > > > regards, > > Asad > > > > -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
