Good day! Recently I started implementing log monitoring and analysis system using graylog2, we will have around 12,000 message / second. Though in staging we are not even near that number but the cluster is not stable.
Sometimes ES discovery fails because either the PC is in I/O wait or there are too many processes in each core. I tried to tune the settings by one way or another the cluster finds a way to fail, as for my setup there are some limitation for a a while to use high speed I/O so I need to either stick with slow disks or divide the setup in a way that recent logs remain in high speed disks and older are moved to low performance cluster. I was hoping if someone can help me formulate or calculate a formula to decide how many nodes I need for ES cluster, graylog2-server, radio and Kafka. There is another problem with KAFKA input if i shutdown Kafka, zookeeper or radio, the messages stop coming and I need to Terminate Kafka input and Launch a new input. Also the message throughput while using KAFKA and Radio is far less than using direct inputs with graylog2-benchmark tool. Current Setup 2 Nodes for Log Collector and Radio (8 Gb, 2 Core Xeon ) 1. Graylog2-server + graylog2-web (16 Gb, 4 Core Xeon ) 1. Graylog2-server + elasticsearch (16 Gb, 4 Core Xeon ) 3. Elasticsearch + Kafka Node (16 Gb, 4 Core Xeon ) The message throughput in peak hours will be 12000 / second and to implement this system in production, the system needs to withstand stress test of 20.000 message / second. I will appreciate if anyone here can help me with formulating the performance requirements by quantifying them. regards, Asad -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
