Re: index size growing while deleting

On Tue, Nov 10, 2015 at 7:47 AM, Rob Audenaerde wrote: > Ah yes, that is the way to go. > > It is a bit harder here, because we also use a per-user InMemoryIndex that > is combined in a multi-reader, so it will be a bit more work, but I think > it will be doable. Thanks for all the help. > > That

Re: index size growing while deleting

On Tue, Nov 10, 2015 at 6:19 AM, Jürgen Albert wrote: > In our case we had open index readers, that > blocked the index from merging its segments and thus deleting the marked > segments. Small correction: merging is never blocked because of open NRT readers. The merging happens and finishes eve

Re: index size growing while deleting

Ah yes, that is the way to go. It is a bit harder here, because we also use a per-user InMemoryIndex that is combined in a multi-reader, so it will be a bit more work, but I think it will be doable. Thanks for all the help. That said, I found it not-so-easy to debug this issue; are there methods

Re: index size growing while deleting

Hi Rob, we use a SearcherManager to obtain a fresh Searcher for every Query. From the Searcher we get the Reader. After the query you call searcherManager.release(searcher). The SearcherManager takes care of the rest. Regards, Jürgen. Am 10.11.2015 um 13:27 schrieb Rob Audenaerde: Hi Jürg

Re: index size growing while deleting

Hi Jürgen, Michael Thanks! I seem to be able to reduce the index size by closing and restarting our application. This reduces the index size from 22G tot 4G, with is somewhat the expected size. The infoStream also gives me the 'removed unreferenced file (IFD 0 [2015-11-10T12:21:49.293Z; main]: ini

Re: index size growing while deleting

Hi Rob, we had a similar problem. In our case we had open index readers, that blocked the index from merging its segments and thus deleting the marked segments. Regards, Jürgen. Am 06.11.2015 um 08:59 schrieb Rob Audenaerde: Hi will, others Thanks for you reply, As far as I understand it

Re: index size growing while deleting

On Fri, Nov 6, 2015 at 11:29 AM, Michael McCandless < luc...@mikemccandless.com> wrote: It's also important to IndexWriter.commit (as well as open new NRT > readers) periodically or after doing a large set of updates, as that > lets Lucene remove any old segments referenced by the prior commit > p

Re: index size growing while deleting

Thanks Mike for the reply, I already commit every after every 5000 documents per Thread. I also found out today how to enable the InfoStream through the IndexWriterConfig, so I'll have lots of extra information to work on. Will run it on the production environment to find out what's happening the

Re: index size growing while deleting

It's also important to IndexWriter.commit (as well as open new NRT readers) periodically or after doing a large set of updates, as that lets Lucene remove any old segments referenced by the prior commit point. Mike McCandless http://blog.mikemccandless.com On Fri, Nov 6, 2015 at 2:59 AM, Rob Au

Re: index size growing while deleting

Hi will, others Thanks for you reply, As far as I understand it, deleting a document is just setting the deleted bit, and when segments are merged, then the documents are removed. (not really sure what this means exactly; I guess the document gets removed from the store, the terms will no longer

Re: index size growing while deleting

Hi Rob: Do you understand how deletes work and how an index is compacted? There's some configuration/runtime activities you don't mention And you make testing process sound like a mirror of production? (Including configuration?) -will On 11/5/15 7:33 AM, Rob Audenaerde wrote: Hi all,