For redundancy purposes, our system is split into two datacenters. One of the DCs is considered central where all the backoffice systems reside and the other is edge. Recently we started using Logstash with ElasticSearch and Kibana. The architecture we had is:
- Scribe server on each instance in our cluster forwards logs to a main scribe instance in the DC. - If the DC is the edge, its main scribe instance forwards all logs to the main scribe instance in central. - From the main (central) scribe server we forward message to Logstash, which in turn get written to ES. Because most logs are only stored but never retrieved, to reduce the traffic between DCs, we thought of using custom routing: - Have elastic search node in each DC (currently we have only one). - Tag each log message with the DC it's originated from and route the log messages according to this tag, so each DC's log messages end up in its own ES instance. Will this work? Is this proper use of ElasticSearch's routing? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0017a4a8-80ca-4fcb-97df-032f9d6858c9%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
