I think it's related to this: https://github.com/elasticsearch/elasticsearch/pull/8270 which I believe was released with 1.4.
We see the same thing, with hot spots on some nodes. You can poke the cluster to rebalance itself, which that #8270 fixes permanently, using "curl -XPOST localhost:9200/_cluster/reroute". That doesn't always sort it out, and this issue (https://github.com/elasticsearch/elasticsearch/issues/8149) is our primary issue. AFAIK it's not just Marvel, but any indice can get into this situation. Right now I have a few nodes with 1TB of free disk and others with 400Gb, and Marvel is in another cluster entirely. cheers mike On Tuesday, November 11, 2014 4:15:33 AM UTC-5, Duncan Innes wrote: > > I now know that Marvel creates a lot of data per day of monitoring - in > our case around 1Gb. > > What I'm just starting to get my head around is the imbalance of disk > usage that this caused on my 5 node cluster. > > I've now removed Marvel and deleted the indexes for now (great tool, but I > don't have the disk space to spare on this proof of concept) and my disk > usage for the 12 months of rsyslog data has equalised across all the nodes > in my cluster. When the Marvel data was sitting there, not only was I > using far too much disk space, but I was also seeing significant > differences between nodes. At least one node would be using nearly all of > the 32Gb, where other nodes would sit at half that or even less. Is there > something intrinsically different about Marvel's indexes that makes them > prone to such wild differences? > > Thanks > > Duncan > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/477f9c6c-b359-4776-83be-e8ac5ac8401a%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
