I'm managing a pretty badass 11 node Elasticsearch cluster that is powering a customer facing dashboard reporting platform. 20 cores per node, 64GB RAM, SSDs, Dual 10 GbE of awesome. I evaluated Marvel while we were still in development on the new platform and I found it to be a very valuable tool. At first Marvel was indexing to the same cluster we were monitoring and this was okay while we were in development as there were plenty of extra cycles in the cluster to handle the load but now that we are in production it doesn't make sense to burden the cluster with this. The nature of our reporting system requires us to to have an index for each customer so we're currently at 328 indexes and over 10,000 shards total. The amount of data indexed by Marvel increases dramatically as the number of indices increases so once we got over 300 indices in the system the daily marvel index ended up at around 400 GB replicated and was indexing around 2,000 documents a second by itself.
What I want to do is have Marvel index to a not as awesome 2 node Elasticsearch monitoring cluster. 12 cores, 64 GB RAM and spinning disks. But in practice these 2 nodes are unable to keep up with the load and get completely bogged down. I'm thinking I can sacrifice redundancy and buy myself some cycles by not using any replicas on the Marvel index. My other idea is to set marvel.agent.interval from the default 10s to something like 30s on the assumption that this will cut the amount of data generated by a third. Does this sound sane or do you have anyone have other ideas on what I can try to limited the load? marvel.agent.interval Controls the interval between data samples. Defaults to 10s. Set to -1 to temporarily disable exporting. This setting is update-able via the Cluster Update Settings API. Thanks -Logan- -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5884045a-49f7-48d4-a3cb-93a5f70c53cf%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
