Re: Retrieve disk usage & release disk space after delete
Q1: I'm looking for the disk usage data in Solr admin - Solr Cloud/Nodes. Any way to get the Node data in table thru API call? Q2: Thanks for helpful information about deleting the data. The main issue I have now is for deleting collections, even if I delete by admin UI, this suppose not to hang around in data dirs? not observing specific error message so far, anything could induce? Regards, Chien -- Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Retrieve disk usage & release disk space after delete
We get disk usage on volumes using Telegraf. I’m planning on writing something that gathers size info (docs and bytes) by getting core info from the CLUSTERSTATUS request then using the CoreAdmin API to get the detailed info about cores. It doesn’t look hard, just complicated. Fire up Python and start walking JSON data. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Jun 23, 2020, at 4:27 AM, Erick Erickson wrote: > > Q1: If you’re talking about disk space used up by deleted documents, > then yes, optimize or expungeDeletes will recover it. The former >will recover it all, the latter will rewrite segments with > 10% deleted > documents. HOWEVER: optimize is an expensive operation, and >can have deleterious side-effects, especially before Solr 7.5, see: > > https://lucidworks.com/post/segment-merging-deleted-documents-optimize-may-bad/ > and > https://lucidworks.com/post/solr-and-optimizing-your-index-take-ii/ > > NOTE: if you just ignore it, the deleted data will be merged away as > part of normal indexing so you may have to do nothing. > > Q2: The data if you delete the collections should be removed from > disk, assuming you’re talking about using the Collections API, > DELETE command. Optimize won’t help because the collection is gone. > If you delete the collection and the data dirs are still hanging around, > you should look at your logs to see if there’s any information. > > Best, > Erick > >> On Jun 22, 2020, at 9:04 PM, ChienHuaWang wrote: >> >> Hi Solr users, >> >> Q1: Wondering if there is any way to retrieve disk usage by host? Could we >> get thru metrics API or any other methods? I know the data shows in Solr >> Admin UI, but have other approach for this kind of data. >> >> Q2: >> After delete the collections, it seems not physically removed from the disk. >> Did the research, someone suggest to run an optimize which re-writes the >> index out to disk without the deleted documents, then deletes the original. >> Is there any other way to do clean up without re-writes the index? have to >> manually clean up now, and look for better approach >> >> Appreciate your feedback. >> >> >> Regards, >> Chien >> >> >> >> >> >> -- >> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html >
Re: Retrieve disk usage & release disk space after delete
Q1: If you’re talking about disk space used up by deleted documents, then yes, optimize or expungeDeletes will recover it. The former will recover it all, the latter will rewrite segments with > 10% deleted documents. HOWEVER: optimize is an expensive operation, and can have deleterious side-effects, especially before Solr 7.5, see: https://lucidworks.com/post/segment-merging-deleted-documents-optimize-may-bad/ and https://lucidworks.com/post/solr-and-optimizing-your-index-take-ii/ NOTE: if you just ignore it, the deleted data will be merged away as part of normal indexing so you may have to do nothing. Q2: The data if you delete the collections should be removed from disk, assuming you’re talking about using the Collections API, DELETE command. Optimize won’t help because the collection is gone. If you delete the collection and the data dirs are still hanging around, you should look at your logs to see if there’s any information. Best, Erick > On Jun 22, 2020, at 9:04 PM, ChienHuaWang wrote: > > Hi Solr users, > > Q1: Wondering if there is any way to retrieve disk usage by host? Could we > get thru metrics API or any other methods? I know the data shows in Solr > Admin UI, but have other approach for this kind of data. > > Q2: > After delete the collections, it seems not physically removed from the disk. > Did the research, someone suggest to run an optimize which re-writes the > index out to disk without the deleted documents, then deletes the original. > Is there any other way to do clean up without re-writes the index? have to > manually clean up now, and look for better approach > > Appreciate your feedback. > > > Regards, > Chien > > > > > > -- > Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Retrieve disk usage & release disk space after delete
Hi Solr users, Q1: Wondering if there is any way to retrieve disk usage by host? Could we get thru metrics API or any other methods? I know the data shows in Solr Admin UI, but have other approach for this kind of data. Q2: After delete the collections, it seems not physically removed from the disk. Did the research, someone suggest to run an optimize which re-writes the index out to disk without the deleted documents, then deletes the original. Is there any other way to do clean up without re-writes the index? have to manually clean up now, and look for better approach Appreciate your feedback. Regards, Chien -- Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
