Hello everybody, The recent 1.0 release has made the radio module deprecated. In the documentation we can read that some Kafka mechanism has been put inside the server module (journal on disk) to avoid loosing messages.
We have deployed a testing environment and are able to process about 1000 messages per second (average) using 4 virtual machines per datacenter (1 radio + 1 MQ + 1 server + 1 elasticsearch). The 2 RabbitMQ nodes use the "ha-mode" to synchronize their queues and the elasticsearch cluster is stretched across the sites. The main goal of this architecture is lead by both our disaster recovery plan and maintenance periods : - the log messages are available in both datacentres as fast as possible without being processed. If a site crashes we would be able to read the last messages sent by the servers ; - the MQ cluster can act as a buffer and allow us to shutdown the whole stack (graylog-server + elasticsearch). The message would still be gathered by the radio nodes and stored in the queues. I would like to know your point of view on how to maintain these functionalities without the radio module. Is it possible to replicate the messages across graylog-server instances before being processed ? I planned to test two cases : 1. make Rsyslog / Logstash / Fluentd send their messages directly to RabbitMQ and use the GELF AMQP input : - pros : less virtual machines (no radios) - cons : I need to write the GELF over AMQP outputs / plugins for Logstash, I need to deal with Rsyslog output too. 2. use Logstash instead of Graylog Radio to send messages to RabbitMQ : - pros : same architecture - cons : I still need to write the GELF codec, I am not sure that Logstash can handle the same load than Graylog-Radio. I have just written a GELF codec for Logstash but issued a pending bug (some .toJavaBytes problem)... Do you have an idea about this situation ? Mathieu -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
