This gist shows the error in action: https://gist.github.com/nik9000/3acdb38052dba3fbc5a0 Total - free on disk is 479163707392 But used is 238902736642 Meaning about 50% of used space isn't accounted for. But everything on that partition is in elasticsearch's directory: manybubbles@elastic1001:/var/lib/elasticsearch/production-search-eqiad/nodes/0/indices$ du -h | tail -n1 447G .
Its like when we did the upgrade some files weren't deleted when they were no longer in use. On Thu, Aug 21, 2014 at 10:24 AM, Nikolas Everett <[email protected]> wrote: > Hi Lee! Thanks for responding. Ok, here goes: > > Version: 1.2.1->1.3.2 > curl 'localhost:9200/_cat/health?v: > epoch timestamp cluster status node.total node.data > shards pri relo init unassign > 1408630877 14:21:17 production-search-eqiad green 17 17 > 6050 2017 0 0 0 > > curl 'localhost:9200/_cat/shards?v': > https://gist.github.com/nik9000/815f3e39a3673e6f48ac > > I'll keep digging into it now that I'm properly awake as well. > > Nik > > > On Thu, Aug 21, 2014 at 9:36 AM, Lee Hinman <[email protected]> > wrote: > >> On Thursday, August 21, 2014 2:44:19 PM UTC+2, Nikolas Everett wrote: >>> >>> I started a rolling restart yesterday but has add to stop because the >>> disks were filling up oddly. It looks like when the bode comes up it no >>> longer deletes shards it can't use. >>> >>> Elasticsearch reports that the disk is nearly full but that it isn't >>> using most of the space. When I look myself the disk is mostly full and >>> most of the space is taken up by shards. >>> >>> I'm not clear where to go from here though. Find the files elasticsearch >>> doesn't have open and delete them? >>> >> >> Hi Nikolas, >> >> Can you provide the output of `curl 'localhost:9200/_cat/shards?v'` and >> `curl 'localhost:9200/_cat/health?v'`? Also, can you describe your cluster >> topology and what the current disk usages are for all nodes across the >> cluster? >> >> Additionally, what version of ES are you using before and after the >> upgrade? >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/4bcb3a29-7fb8-4f09-80fd-83dbd7d81c47%40googlegroups.com >> <https://groups.google.com/d/msgid/elasticsearch/4bcb3a29-7fb8-4f09-80fd-83dbd7d81c47%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0QkcnQrn%3D%2BSp4EGoNfySfn2%2B7M%3DS8E2EhzPbR7k2wWuA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
