Hi,

We have a ElasticSearch-Logstash-Kibana deployment in which multiple logstash-forwarders (from multiple service logs) pushes log to logstash which then sends it to Kafka and then one more logstash pulls those logs from Kafka and indexes them in ElasticSearch cluster. Right now we are using single kafka queue in which all the logs from all the services are going.

Can we configure Kafka in such a way that a single logstash-forwarder (of a single service) does not hog the entire set-up? We want to ensure that all services can use the set-up fairly. So basically I want to ensure the Kafka queues are not always filled up by only one type of messages.

Thanks,
Debraj

Reply via email to