Re: Bizarre indexing issue where thousands of files get created

2009-08-18 Thread Michael McCandless
Are you .close()ing your IndexReaders when they fall out of the MRU cache? Seems like there are two problems... 1) why are you hitting OOMEs? Seems likely you're just doing too much at once can you ask the JRE to get you a heap dump when it hits OOME? 2) Why is IndexWriter creating zillions o

Re: Problem doing backup using the SnapshotDeletionPolicy class

2009-08-18 Thread Lucas Nazário dos Santos
Thanks, Shai. After thinking a bit I don't believe that the SnapshotDeletionPolicy is the best approach to my problem. Because I do batch indexing, the final solution I came up with is to index documents each time in a temporary folder, copy this temporary folder to a backup directory after index

Re: Is there a way to check for field "uniqueness" when indexing?

2009-08-18 Thread Grant Ingersoll
On Aug 13, 2009, at 10:33 AM, Daniel Shane wrote: Does anyone have an idea on how I could check an index that is in the process of being indexed (things added, things deleted) for the uniquess of a given field *at the time I index a document* ? Solr has de-duplication built-in at indexi

Re: Bizarre indexing issue where thousands of files get created

2009-08-18 Thread Micah Jaffe
Hi, thanks for the response! The (custom) searchers that are falling out of cache are indeed calling close on their IndexReader in finalize(); they are not calling close on themselves as that appears to be a no-op when creating an IndexSearcher with a reader. The searchers are just extend

Re: Bizarre indexing issue where thousands of files get created

2009-08-18 Thread Jason Rutherglen
Micah, If you can post some of your code, it may be easier to identify the problem you're experiencing. -J On Tue, Aug 18, 2009 at 9:55 AM, Micah Jaffe wrote: > Hi, thanks for the response!  The (custom) searchers that are falling out of > cache are indeed calling close on their IndexReader in f

Re: How to normalize Lucene score?

2009-08-18 Thread Chris Hostetter
: Hi, : : How to normalize the Lucene score to the range [0, 1]? http://wiki.apache.org/lucene-java/ScoresAsPercentages -Hoss - To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-ma

Lucene-Core test failures

2009-08-18 Thread Bryan Swift
I was running the tests which lucene-core version 2.4 and I noticed a failure in org.apache.lucene.index.TestIndexInput for testRead at line 89. The assertions in question have to do with reading "Modified UTF-8 null bytes" according to the comments in the file. It seems these modified null

Re: Lucene-Core test failures

2009-08-18 Thread Mark Miller
Bryan Swift wrote: I was running the tests which lucene-core version 2.4 and I noticed a failure in org.apache.lucene.index.TestIndexInput for testRead at line 89. The assertions in question have to do with reading "Modified UTF-8 null bytes" according to the comments in the file. It seems thes