Re: lots of .cfs (compound files) in the index directory

2012-03-15 Thread Michael McCandless
On Thu, Mar 15, 2012 at 12:33 PM, Uwe Schindler wrote: > Close calls and always did call commit in 3.x? Right, it does. But in the case when prepareCommit was called... it then only commits the changes as of that prepareCommit and *not* any changes done after that and before close. That's the

RE: lots of .cfs (compound files) in the index directory

2012-03-15 Thread Uwe Schindler
Close calls and always did call commit in 3.x? - Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From: Michael McCandless [mailto:luc...@mikemccandless.com] > Sent: Thursday, March 15, 2012 5:29 PM > To: java-us

Re: lots of .cfs (compound files) in the index directory

2012-03-15 Thread Michael McCandless
On Thu, Mar 15, 2012 at 12:02 PM, Tim Bogaert wrote: > while removing the prepareCommit we noticed we didn't actually called the > IW.commit() method before the IW.close(). > Altough the documentation says the close() method commits all the changes > we tried to add the commit() method before the

RE: lots of .cfs (compound files) in the index directory

2012-03-15 Thread Uwe Schindler
Hi, It will always commit before closing! But the result depends how you call close... but depending on the boolean parameter, the background merges can be interrupted, so it closes as soon as possible, leaving pending merges undone. - Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http:/

Re: lots of .cfs (compound files) in the index directory

2012-03-15 Thread Tim Bogaert
Mike, while removing the prepareCommit we noticed we didn't actually called the IW.commit() method before the IW.close(). Altough the documentation says the close() method commits all the changes we tried to add the commit() method before the IW.close() and this seems to fix our problem. Is this a

Re: TO Mike McCandless : ToParentBlockJoinQuery inconsistent return

2012-03-15 Thread Michael McCandless
You're welcome! Happy searching, Mike McCandless http://blog.mikemccandless.com On Thu, Mar 15, 2012 at 11:40 AM, Jean-Marc MORAS wrote: > Thanks for your two responses. > > Best regards, > > Jean-Marc > > -- > > -> Ok now I have seen the mention of that on  ToParentBlockJoinQuery clas

RE: TO Mike McCandless : ToParentBlockJoinQuery inconsistent return

2012-03-15 Thread Jean-Marc MORAS
Thanks for your two responses. Best regards, Jean-Marc -- -> Ok now I have seen the mention of that on ToParentBlockJoinQuery class java doc ->This java doc specify : "At search time you provide a Filter * identifying the parents, however this Filter must provide * an {@link Fix

Re: lots of .cfs (compound files) in the index directory

2012-03-15 Thread Michael McCandless
Hmm are you calling IW.prepareCommit() before calling IW.close()? Can you try removing the call the IW.prepareCommit() and see if the segments are then contained...? Mike McCandless http://blog.mikemccandless.com On Thu, Mar 15, 2012 at 11:14 AM, Tim Bogaert wrote: > Operating system is Solari

Re: lots of .cfs (compound files) in the index directory

2012-03-15 Thread Tim Bogaert
Operating system is Solaris. Directory is a NAS. Directory implementation is SimpleFSDirectory. I sent you the full log. Thanks, Tim On Thu, Mar 15, 2012 at 4:04 PM, Michael McCandless < luc...@mikemccandless.com> wrote: > Hmm, which OS/filesystem is the index on? Which Directory impl are > you

Re: lots of .cfs (compound files) in the index directory

2012-03-15 Thread Michael McCandless
Hmm, which OS/filesystem is the index on? Which Directory impl are you using...? Indeed CheckIndex sees the 5111 segments, so something seems amiss. The log was edited right? EG I can't see the IW open log entries... and IW 51 is missing (it goes straight from 50 to 52). Can you post the full

Re: lots of .cfs (compound files) in the index directory

2012-03-15 Thread Tim Bogaert
Hi Mike, Segment format: FORMAT_3_1 [Lucene 3.1+] NumBadSegments: 0 NumSegments: 5111 Clean?: true TotLoseDocCount: 0 Partial: false I reindexed and let the feed come in again...The logs of the result are below (I just copied a few) after 10 minutes : Segment format: FORMAT_3_1 [Lucene 3.1+] Num

recent lucene data structures diagrams

2012-03-15 Thread Ioan Eugen Stan
Hello, Does anyone know of a good diagram / overview / description of how lucene works internally and how the segments are persisted to disk. Thanks, -- Ioan Eugen Stan http://ieugen.blogspot.com - To unsubscribe, e-mail: ja

Re: lots of .cfs (compound files) in the index directory

2012-03-15 Thread Michael McCandless
Hmm, that's odd... Can you set IndexWriter's infoStream and then capture the output while doing the small writes every few seconds and post back? If you run CheckIndex on the index does it also report ~3000 segments? Mike McCandless http://blog.mikemccandless.com On Thu, Mar 15, 2012 at 7:14 A

lots of .cfs (compound files) in the index directory

2012-03-15 Thread Tim Bogaert
Hi, We migrated our project to Lucene 3.5 and used the TieredMergePolicy with all its defaults. Our project contains a relatively small index (+-90MB) and gets small writes every few seconds. The behaviour we see is that when we reindex everything works great and performant, but when we keep the i