"From the main (central) scribe server we forward message to Logstash, which in turn get written to ES."
Hey. I wanted to know how did you infact forward the message from scribe server to logstash? Did you do it by creating a custom logstash input plugin by yourself? Thanks in advance. Regards, Ananya On Thursday, January 9, 2014 4:38:47 PM UTC+5:30, Arik Fraimovich wrote: > > For redundancy purposes, our system is split into two datacenters. One of > the DCs is considered central where all the backoffice systems reside and > the other is edge. Recently we started using Logstash with ElasticSearch > and Kibana. The architecture we had is: > > - Scribe server on each instance in our cluster forwards logs to a > main scribe instance in the DC. > - If the DC is the edge, its main scribe instance forwards all logs to > the main scribe instance in central. > - From the main (central) scribe server we forward message to > Logstash, which in turn get written to ES. > > Because most logs are only stored but never retrieved, to reduce the > traffic between DCs, we thought of using custom routing: > > - Have elastic search node in each DC (currently we have only one). > - Tag each log message with the DC it's originated from and route the > log messages according to this tag, so each DC's log messages end up in > its > own ES instance. > > Will this work? Is this proper use of ElasticSearch's routing? > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/832ee819-de12-437c-b48a-6688330c2625%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
