Index: lucene/CHANGES.txt =================================================================== --- lucene/CHANGES.txt (revision 1041013) +++ lucene/CHANGES.txt (working copy) @@ -399,9 +399,15 @@ * LUCENE-2733: Removed public constructors of utility classes with only static methods to prevent instantiation. (Uwe Schindler) -* LUCENE-2753: IndexReader and DirectoryReader .listCommits() now return a List - instead of a Collection, guaranteeing the commits are sorted from oldest to - latest. (Shai Erera) +* LUCENE-2602: The default (LogByteSizeMergePolicy) merge policy now + takes deletions into account by default. You can disable this by + calling setCalibrateSizeByDeletes(false) on the merge policy. (Mike + McCandless) + +* LUCENE-2529, LUCENE-2668: Position increment gap and offset gap of empty + values in multi-valued field has been changed for some cases in index. + If you index empty fields and uses positions/offsets information on that + fields, reindex is recommended. (David Smiley, Koji Sekiguchi) Changes in runtime behavior @@ -434,6 +440,14 @@ affect optimize() as well (as opposed to only regular merges). This means that you can run optimize() and too large segments won't be merged. (Shai Erera) +* LUCENE-2753: IndexReader and DirectoryReader .listCommits() now return a List, + guaranteeing the commits are sorted from oldest to latest. (Shai Erera) + +* LUCENE-2785: TopScoreDocCollector, TopFieldCollector and + the IndexSearcher search methods that take an int nDocs will now + throw IllegalArgumentException if nDocs is 0. Instead, you should + use the newly added TotalHitCountCollector. (Mike McCandless) + API Changes * LUCENE-2076: Rename FSDirectory.getFile -> getDirectory. (George @@ -516,17 +530,32 @@ (such as SnapshotDeletionPolicy), you can call this method to remove those commit points when they are not needed anymore (instead of waiting for the next commit). (Shai Erera) - -* LUCENE-2455: IndexWriter.addIndexesNoOptimize was renamed to addIndexes. - IndexFileNames.segmentFileName now takes another parameter to accommodate - custom file names. You should use this method to name all your files. - (Shai Erera) * LUCENE-2481: SnapshotDeletionPolicy.snapshot() and release() were replaced with equivalent ones that take a String (id) as argument. You can pass whatever ID you want, as long as you use the same one when calling both. (Shai Erera) +* LUCENE-2356: Add IndexWriterConfig.set/getReaderTermIndexDivisor, to + set what IndexWriter passes for termsaIndexDivisor to the readers it + opens internally when apply deletions or creating a near-real-time + reader. (Earwin Burrfoot via Mike McCandless) + +* LUCENE-2167: StandardTokenizer/Analyzer now implement the Word Break + rules from the Unicode Text Segmentation algorithm (UAX#29), as well + as tokenizing URLs and email addresses according to the relevant + RFCs. ClassicTokenizer/Analyzer retains the old StandardTokenizer / + StandardAnalyzer behavior. (Steven Rowe, Robert Muir, Uwe Schindler) + +* LUCENE-2699: Update StandardTokenizer and UAX29Tokenizer to Unicode 6.0.0. + (Steven Rowe) + +* LUCENE-2778: RAMDirectory now exposes newRAMFile() which allows to override + and return a different RAMFile implementation. (Shai Erera) + +* LUCENE-2785: Added TotalHitCountCollector whose sole purpose is to + count the number of hits matching the query. (Mike McCandless) + Bug fixes * LUCENE-2249: ParallelMultiSearcher should shut down thread pool on @@ -547,10 +576,6 @@ a prior (corrupt) index missing its segments_N file. (Mike McCandless) -* LUCENE-2534: fix over-sharing bug in - MultiTermsEnum.docs/AndPositionsEnum. (Robert Muir, Mike - McCandless) - * LUCENE-2458: QueryParser no longer automatically forms phrase queries, assuming whitespace tokenization. Previously all CJK queries, for example, would be turned into phrase queries. The old behavior is preserved with @@ -569,8 +594,11 @@ can cause the same document to score to differently depending on what segment it resides in. (yonik) -* LUCENE-2272: Fix explain in PayloadNearQuery and also fix scoring issue (Peter Keegan via Grant Ingersoll) +* LUCENE-2272: Fix explain in PayloadNearQuery and also fix scoring issue (Peter Keegan via Grant Ingersoll) +* LUCENE-2732: Fix charset problems in XML loading in + HyphenationCompoundWordTokenFilter. (Uwe Schinder) + New features * LUCENE-2128: Parallelized fetching document frequencies during weight @@ -642,6 +670,16 @@ can be used to prevent commits from ever getting deleted from the index. (Shai Erera) +* LUCENE-1585: IndexWriter now accepts a PayloadProcessorProvider which can + return a DirPayloadProcessor for a given Directory, which returns a + PayloadProcessor for a given Term. The PayloadProcessor will be used to + process the payloads of the segments as they are merged (e.g. if one wants to + rewrite payloads of external indexes as they are added, or of local ones). + (Shai Erera, Michael Busch, Mike McCandless) + +* LUCENE-2440: Add support for custom ExecutorService in + ParallelMultiSearcher (Edward Drapkin via Mike McCandless) + * LUCENE-2295: Added a LimitTokenCountAnalyzer / LimitTokenCountFilter to wrap any other Analyzer and provide the same functionality as MaxFieldLength provided on IndexWriter. This patch also fixes a bug @@ -649,10 +687,18 @@ * LUCENE-2526: Don't throw NPE from MultiPhraseQuery.toString when it's empty. (Ross Woolf via Mike McCandless) + +* LUCENE-2559: Added SegmentReader.reopen methods (John Wang via Mike + McCandless) -* LUCENE-2671: Add SortField.setMissingValue( v ) to enable sorting - behavior for documents that do not include the given field. (ryan) +* LUCENE-2590: Added Scorer.visitSubScorers, and Scorer.freq. Along + with a custom Collector these experimental methods make it possible + to gather the hit-count per sub-clause and per document while a + search is running. (Simon Willnauer, Mike McCandless) +* LUCENE-2636: Added MultiCollector which allows running the search with several + Collectors. (Shai Erera) + * LUCENE-2754, LUCENE-2757: Added a wrapper around MultiTermQueries to add span support: SpanMultiTermQueryWrapper. Using this wrapper its easy to add fuzzy/wildcard to e.g. a SpanNearQuery. @@ -660,6 +706,9 @@ Optimizations +* LUCENE-2494: Use CompletionService in ParallelMultiSearcher instead of + simple polling for results. (Edward Drapkin, Simon Willnauer) + * LUCENE-2075: Terms dict cache is now shared across threads instead of being stored separately in thread local storage. Also fixed terms dict so that the cache is used when seeking the thread local @@ -722,6 +771,17 @@ (getStrings, getStringIndex), consume quite a bit less RAM in most cases. (Mike McCandless) +* LUCENE-2410: ~20% speedup on exact (slop=0) PhraseQuery matching. + (Mike McCandless) + +* LUCENE-2531: Fix issue when sorting by a String field that was + causing too many fallbacks to compare-by-value (instead of by-ord). + (Mike McCandless) + +* LUCENE-2574: IndexInput exposes copyBytes(IndexOutput, long) to allow for + efficient copying by sub-classes. Optimized copy is implemented for RAM and FS + streams. (Shai Erera) + * LUCENE-2719: Improved TermsHashPerField's sorting to use a better quick sort algorithm that dereferences the privot element not on every compare call. Also replaced lots of sorting code in Lucene @@ -797,6 +857,15 @@ as Eclipse and IntelliJ. (Paolo Castagna, Steven Rowe via Robert Muir) +Documentation + +* LUCENE-2579: Fix oal.search's package.html description of abstract + methods. (Santiago M. Mola via Mike McCandless) + +* LUCENE-2625: Add a note to IndexReader.termDocs() with additional verbage + that the TermEnum must be seeked since it is unpositioned. + (Adriano Crestani via Robert Muir) + ================== Release 2.9.4 / 3.0.3 2010-12-03 ==================== Changes in runtime behavior