Hi list, I currently have a 4 nodes cluster to collect varnish logs of my platform, The cluster is dealing with almost 80Mill of documents (2000 documents per second) and sometime I have some performance problems. Let me explain how i do it. In all frontend server i have a logstash agent sending Varnish logs to a redis instance in one of my elasticsearch machines. In that machine other logstash process pull the events and store it in the elasticsearch cluster. My problem come when the logs begin to queue in redis because elasticsearch cluster is not able to store so fast.
The cluster machines (virtual machines) have 4 CPU and 4GB of RAM and I think are not enough to deal with 2000-3000 events per second. I have been reading a lot about this issue and everybody have machines bigger than mine. What would be the right size for the machines? I have an index per day which size is almost 100GB, and I've read that the size of your RAM should be the same that your indexes size, but that is not possible for me. If I add more nodes to the cluster the performance will be better? Thanks in advance! Jorge -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d21ce426-7612-47a1-9da0-56441d544892%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
