I found that running graylog2 server and elasticsearch side by side is problematic, try completly seperating duties, have one or more elasticsearch/mongodb nodes then one or more graylog2 nodes with web and server and one or more rabbitmq/kafka graylog radio nodes. I think I will follow the suggestion to try multiple elastic search nodes myself as I am just barely able to keep up with 500-750 mps. I am also using ZFS compression which cant be helping ES performance (though it helps with storage capacity). -=- Joel -=- On Sunday, June 8, 2014 7:37:52 PM UTC-5, Asad Mehmood wrote: > > Good day! > > Recently I started implementing log monitoring and analysis system using > graylog2, we will have around 12,000 message / second. Though in staging we > are not even near that number but the cluster is not stable. > > Sometimes ES discovery fails because either the PC is in I/O wait or there > are too many processes in each core. > I tried to tune the settings by one way or another the cluster finds a way > to fail, as for my setup there are some limitation for a a while to use > high speed I/O so I need to either stick with slow disks or divide the > setup in a way that recent logs remain in high speed disks and older are > moved to low performance cluster. I was hoping if someone can help me > formulate or calculate a formula to decide how many nodes I need for ES > cluster, graylog2-server, radio and Kafka. > > There is another problem with KAFKA input if i shutdown Kafka, zookeeper > or radio, the messages stop coming and I need to Terminate Kafka input and > Launch a new input. > Also the message throughput while using KAFKA and Radio is far less than > using direct inputs with graylog2-benchmark tool. > > Current Setup > 2 Nodes for Log Collector and Radio (8 Gb, 2 Core Xeon ) > 1. Graylog2-server + graylog2-web (16 Gb, 4 Core Xeon ) > 1. Graylog2-server + elasticsearch (16 Gb, 4 Core Xeon ) > 3. Elasticsearch + Kafka Node (16 Gb, 4 Core Xeon ) > > The message throughput in peak hours will be 12000 / second and to > implement this system in production, the system needs to withstand stress > test of 20.000 message / second. > > I will appreciate if anyone here can help me with formulating the > performance requirements by quantifying them. > > > regards, > > Asad > > > >
-- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
