http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net/Search/Similarities/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net/Search/Similarities/package.md b/src/Lucene.Net/Search/Similarities/package.md new file mode 100644 index 0000000..c655791 --- /dev/null +++ b/src/Lucene.Net/Search/Similarities/package.md @@ -0,0 +1,55 @@ + +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +This package contains the various ranking models that can be used in Lucene. The +abstract class [](xref:Lucene.Net.Search.Similarities.Similarity) serves +as the base for ranking functions. For searching, users can employ the models +already implemented or create their own by extending one of the classes in this +package. + +## Table Of Contents + + 1. [Summary of the Ranking Methods](#sims) 2. [Changing the Similarity](#changingSimilarity) + +## Summary of the Ranking Methods + +[](xref:Lucene.Net.Search.Similarities.DefaultSimilarity) is the original Lucene scoring function. It is based on a highly optimized [Vector Space Model](http://en.wikipedia.org/wiki/Vector_Space_Model). For more information, see [](xref:Lucene.Net.Search.Similarities.TFIDFSimilarity). + +[](xref:Lucene.Net.Search.Similarities.BM25Similarity) is an optimized implementation of the successful Okapi BM25 model. + +[](xref:Lucene.Net.Search.Similarities.SimilarityBase) provides a basic implementation of the Similarity contract and exposes a highly simplified interface, which makes it an ideal starting point for new ranking functions. Lucene ships the following methods built on [](xref:Lucene.Net.Search.Similarities.SimilarityBase): * Amati and Rijsbergen's {@linkplain org.apache.lucene.search.similarities.DFRSimilarity DFR} framework; * Clinchant and Gaussier's {@linkplain org.apache.lucene.search.similarities.IBSimilarity Information-based models} for IR; * The implementation of two {@linkplain org.apache.lucene.search.similarities.LMSimilarity language models} from Zhai and Lafferty's paper. Since [](xref:Lucene.Net.Search.Similarities.SimilarityBase) is not optimized to the same extent as [](xref:Lucene.Net.Search.Similarities.DefaultSimilarity) and [](xref:Lucene.Net.Search.Similarities.BM25Similarity), a difference in performance is to be expected when using the methods listed above. Howe ver, optimizations can always be implemented in subclasses; see [below](#changingSimilarity). + +## Changing Similarity + +Chances are the available Similarities are sufficient for all your searching needs. However, in some applications it may be necessary to customize your [Similarity](Similarity.html) implementation. For instance, some applications do not need to distinguish between shorter and longer documents (see [a "fair" similarity](http://www.gossamer-threads.com/lists/lucene/java-user/38967#38967)). + +To change [](xref:Lucene.Net.Search.Similarities.Similarity), one must do so for both indexing and searching, and the changes must happen before either of these actions take place. Although in theory there is nothing stopping you from changing mid-stream, it just isn't well-defined what is going to happen. + +To make this change, implement your own [](xref:Lucene.Net.Search.Similarities.Similarity) (likely you'll want to simply subclass an existing method, be it [](xref:Lucene.Net.Search.Similarities.DefaultSimilarity) or a descendant of [](xref:Lucene.Net.Search.Similarities.SimilarityBase)), and then register the new class by calling [](xref:Lucene.Net.Index.IndexWriterConfig.SetSimilarity(Similarity)) before indexing and [](xref:Lucene.Net.Search.IndexSearcher.SetSimilarity(Similarity)) before searching. + +### Extending {@linkplain org.apache.lucene.search.similarities.SimilarityBase} + + The easiest way to quickly implement a new ranking method is to extend [](xref:Lucene.Net.Search.Similarities.SimilarityBase), which provides basic implementations for the low level . Subclasses are only required to implement the [](xref:Lucene.Net.Search.Similarities.SimilarityBase.Score(BasicStats, float, float)) and [](xref:Lucene.Net.Search.Similarities.SimilarityBase.ToString()) methods. + +Another option is to extend one of the [frameworks](#framework) based on [](xref:Lucene.Net.Search.Similarities.SimilarityBase). These Similarities are implemented modularly, e.g. [](xref:Lucene.Net.Search.Similarities.DFRSimilarity) delegates computation of the three parts of its formula to the classes [](xref:Lucene.Net.Search.Similarities.BasicModel), [](xref:Lucene.Net.Search.Similarities.AfterEffect) and [](xref:Lucene.Net.Search.Similarities.Normalization). Instead of subclassing the Similarity, one can simply introduce a new basic model and tell [](xref:Lucene.Net.Search.Similarities.DFRSimilarity) to use it. + +### Changing {@linkplain org.apache.lucene.search.similarities.DefaultSimilarity} + + If you are interested in use cases for changing your similarity, see the Lucene users's mailing list at [Overriding Similarity](http://www.gossamer-threads.com/lists/lucene/java-user/39125). In summary, here are a few use cases: 1. <p>The `SweetSpotSimilarity` in `org.apache.lucene.misc` gives small increases as the frequency increases a small amount and then greater increases when you hit the "sweet spot", i.e. where you think the frequency of terms is more significant.</p> 2. <p>Overriding tf â In some applications, it doesn't matter what the score of a document is as long as a matching term occurs. In these cases people have overridden Similarity to return 1 from the tf() method.</p> 3. <p>Changing Length Normalization â By overriding [](xref:Lucene.Net.Search.Similarities.Similarity.ComputeNorm(FieldInvertState state)), it is possible to discount how the length of a field contributes to a score. In [](xref:Lucene.Net.Search.Similarities.DefaultSimilarity), lengthNorm = 1 / (numTerms in field)^0.5, but if one changes this to be 1 / (numTerms in field), all fields will be treated ["fairly"](http://www.gossamer-threads.com/lists/lucene/java-user/38967#38967).</p> In general, Chris Hostetter sums it up best in saying (from [the Lucene users's mailing list](http://www.gossamer-threads.com/lists/lucene/java-user/39125#39125)): + +> [One would override the Similarity in] ... any situation where you know more about your data then just that it's "text" is a situation where it *might* make sense to to override your Similarity method. \ No newline at end of file
http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net/Search/Spans/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net/Search/Spans/package.md b/src/Lucene.Net/Search/Spans/package.md new file mode 100644 index 0000000..4f49917 --- /dev/null +++ b/src/Lucene.Net/Search/Spans/package.md @@ -0,0 +1,52 @@ + +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +The calculus of spans. + +A span is a `<doc,startPosition,endPosition>` tuple. + +The following span query operators are implemented: * A [](xref:Lucene.Net.Search.Spans.SpanTermQuery SpanTermQuery) matches all spans containing a particular [](xref:Lucene.Net.Index.Term Term). * A [](xref:Lucene.Net.Search.Spans.SpanNearQuery SpanNearQuery) matches spans which occur near one another, and can be used to implement things like phrase search (when constructed from [](xref:Lucene.Net.Search.Spans.SpanTermQuery)s) and inter-phrase proximity (when constructed from other [](xref:Lucene.Net.Search.Spans.SpanNearQuery)s). * A [](xref:Lucene.Net.Search.Spans.SpanOrQuery SpanOrQuery) merges spans from a number of other [](xref:Lucene.Net.Search.Spans.SpanQuery)s. * A [](xref:Lucene.Net.Search.Spans.SpanNotQuery SpanNotQuery) removes spans matching one [](xref:Lucene.Net.Search.Spans.SpanQuery SpanQuery) which overlap (or comes near) another. This can be used, e.g., to implement within-paragraph search. * A [](xref:Lucene.Net.Search.Spans.SpanFirstQuery SpanFirstQuery) matche s spans matching `q` whose end position is less than `n`. This can be used to constrain matches to the first part of the document. * A [](xref:Lucene.Net.Search.Spans.SpanPositionRangeQuery SpanPositionRangeQuery) is a more general form of SpanFirstQuery that can constrain matches to arbitrary portions of the document. In all cases, output spans are minimally inclusive. In other words, a span formed by matching a span in x and y starts at the lesser of the two starts and ends at the greater of the two ends. + +For example, a span query which matches "John Kerry" within ten +words of "George Bush" within the first 100 words of the document +could be constructed with: + + SpanQuery john = new SpanTermQuery(new Term("content", "john")); + SpanQuery kerry = new SpanTermQuery(new Term("content", "kerry")); + SpanQuery george = new SpanTermQuery(new Term("content", "george")); + SpanQuery bush = new SpanTermQuery(new Term("content", "bush")); + + SpanQuery johnKerry = + new SpanNearQuery(new SpanQuery[] {john, kerry}, 0, true); + + SpanQuery georgeBush = + new SpanNearQuery(new SpanQuery[] {george, bush}, 0, true); + + SpanQuery johnKerryNearGeorgeBush = + new SpanNearQuery(new SpanQuery[] {johnKerry, georgeBush}, 10, false); + + SpanQuery johnKerryNearGeorgeBushAtStart = + new SpanFirstQuery(johnKerryNearGeorgeBush, 100); + +Span queries may be freely intermixed with other Lucene queries. +So, for example, the above query can be restricted to documents which +also use the word "iraq" with: + + Query query = new BooleanQuery(); + query.add(johnKerryNearGeorgeBushAtStart, true, false); + query.add(new TermQuery("content", "iraq"), true, false); \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net/Search/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net/Search/package.md b/src/Lucene.Net/Search/package.md new file mode 100644 index 0000000..7da753e --- /dev/null +++ b/src/Lucene.Net/Search/package.md @@ -0,0 +1,163 @@ + +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Code to search indices. + +## Table Of Contents + + 1. [Search Basics](#search) 2. [The Query Classes](#query) 3. [Scoring: Introduction](#scoring) 4. [Scoring: Basics](#scoringBasics) 5. [Changing the Scoring](#changingScoring) 6. [Appendix: Search Algorithm](#algorithm) + +## Search Basics + + Lucene offers a wide variety of [](xref:Lucene.Net.Search.Query) implementations, most of which are in this package, its subpackages ([](xref:Lucene.Net.Search.Spans spans), [](xref:Lucene.Net.Search.Payloads payloads)), or the [queries module]({@docRoot}/../queries/overview-summary.html). These implementations can be combined in a wide variety of ways to provide complex querying capabilities along with information about where matches took place in the document collection. The [Query Classes](#query) section below highlights some of the more important Query classes. For details on implementing your own Query class, see [Custom Queries -- Expert Level](#customQueriesExpert) below. + + To perform a search, applications usually call [](xref:Lucene.Net.Search.IndexSearcher.Search(Query,int)) or [](xref:Lucene.Net.Search.IndexSearcher.Search(Query,Filter,int)). + + Once a Query has been created and submitted to the [](xref:Lucene.Net.Search.IndexSearcher IndexSearcher), the scoring process begins. After some infrastructure setup, control finally passes to the [](xref:Lucene.Net.Search.Weight Weight) implementation and its [](xref:Lucene.Net.Search.Scorer Scorer) or [](xref:Lucene.Net.Search.BulkScorer BulkScore) instances. See the [Algorithm](#algorithm) section for more notes on the process. + + <!-- TODO: this page over-links the same things too many times --> + +## Query Classes + +#### + [](xref:Lucene.Net.Search.TermQuery TermQuery) + +Of the various implementations of [](xref:Lucene.Net.Search.Query Query), the [](xref:Lucene.Net.Search.TermQuery TermQuery) is the easiest to understand and the most often used in applications. A [](xref:Lucene.Net.Search.TermQuery TermQuery) matches all the documents that contain the specified [](xref:Lucene.Net.Index.Term Term), which is a word that occurs in a certain [](xref:Lucene.Net.Documents.Field Field). Thus, a [](xref:Lucene.Net.Search.TermQuery TermQuery) identifies and scores all [](xref:Lucene.Net.Documents.Document Document)s that have a [](xref:Lucene.Net.Documents.Field Field) with the specified string in it. Constructing a [](xref:Lucene.Net.Search.TermQuery TermQuery) is as simple as: TermQuery tq = new TermQuery(new Term("fieldName", "term")); In this example, the [](xref:Lucene.Net.Search.Query Query) identifies all [](xref:Lucene.Net.Documents.Document Document)s that have the [](xref:Lucene.Net.Documents.Field Field) named <tt>"fieldName"</tt> containing the word <tt>"term"</tt>. + +#### + [](xref:Lucene.Net.Search.BooleanQuery BooleanQuery) + +Things start to get interesting when one combines multiple [](xref:Lucene.Net.Search.TermQuery TermQuery) instances into a [](xref:Lucene.Net.Search.BooleanQuery BooleanQuery). A [](xref:Lucene.Net.Search.BooleanQuery BooleanQuery) contains multiple [](xref:Lucene.Net.Search.BooleanClause BooleanClause)s, where each clause contains a sub-query ([](xref:Lucene.Net.Search.Query Query) instance) and an operator (from [](xref:Lucene.Net.Search.BooleanClause.Occur BooleanClause.Occur)) describing how that sub-query is combined with the other clauses: 1. <p>[](xref:Lucene.Net.Search.BooleanClause.Occur.SHOULD SHOULD) â Use this operator when a clause can occur in the result set, but is not required. If a query is made up of all SHOULD clauses, then every document in the result set matches at least one of these clauses.</p> 2. <p>[](xref:Lucene.Net.Search.BooleanClause.Occur.MUST MUST) â Use this operator when a clause is required to occur in the result set. Every document in the resul t set will match all such clauses.</p> 3. <p>[](xref:Lucene.Net.Search.BooleanClause.Occur.MUST_NOT MUST NOT) â Use this operator when a clause must not occur in the result set. No document in the result set will match any such clauses.</p> Boolean queries are constructed by adding two or more [](xref:Lucene.Net.Search.BooleanClause BooleanClause) instances. If too many clauses are added, a [](xref:Lucene.Net.Search.BooleanQuery.TooManyClauses TooManyClauses) exception will be thrown during searching. This most often occurs when a [](xref:Lucene.Net.Search.Query Query) is rewritten into a [](xref:Lucene.Net.Search.BooleanQuery BooleanQuery) with many [](xref:Lucene.Net.Search.TermQuery TermQuery) clauses, for example by [](xref:Lucene.Net.Search.WildcardQuery WildcardQuery). The default setting for the maximum number of clauses 1024, but this can be changed via the static method [](xref:Lucene.Net.Search.BooleanQuery.SetMaxClauseCount(int)). + +#### Phrases + +Another common search is to find documents containing certain phrases. This + is handled three different ways: + +1. + +[](xref:Lucene.Net.Search.PhraseQuery PhraseQuery) â Matches a sequence of [](xref:Lucene.Net.Index.Term Term)s. [](xref:Lucene.Net.Search.PhraseQuery PhraseQuery) uses a slop factor to determine how many positions may occur between any two terms in the phrase and still be considered a match. The slop is 0 by default, meaning the phrase must match exactly. + +2. + +[](xref:Lucene.Net.Search.MultiPhraseQuery MultiPhraseQuery) â A more general form of PhraseQuery that accepts multiple Terms for a position in the phrase. For example, this can be used to perform phrase queries that also incorporate synonyms. 3. <p>[](xref:Lucene.Net.Search.Spans.SpanNearQuery SpanNearQuery) â Matches a sequence of other [](xref:Lucene.Net.Search.Spans.SpanQuery SpanQuery) instances. [](xref:Lucene.Net.Search.Spans.SpanNearQuery SpanNearQuery) allows for much more complicated phrase queries since it is constructed from other [](xref:Lucene.Net.Search.Spans.SpanQuery SpanQuery) instances, instead of only [](xref:Lucene.Net.Search.TermQuery TermQuery) instances.</p> + +#### + [](xref:Lucene.Net.Search.TermRangeQuery TermRangeQuery) + +The [](xref:Lucene.Net.Search.TermRangeQuery TermRangeQuery) matches all documents that occur in the exclusive range of a lower [](xref:Lucene.Net.Index.Term Term) and an upper [](xref:Lucene.Net.Index.Term Term) according to [](xref:Lucene.Net.Index.TermsEnum.GetComparator TermsEnum.GetComparator()). It is not intended for numerical ranges; use [](xref:Lucene.Net.Search.NumericRangeQuery NumericRangeQuery) instead. For example, one could find all documents that have terms beginning with the letters <tt>a</tt> through <tt>c</tt>. + +#### + [](xref:Lucene.Net.Search.NumericRangeQuery NumericRangeQuery) + +The [](xref:Lucene.Net.Search.NumericRangeQuery NumericRangeQuery) matches all documents that occur in a numeric range. For NumericRangeQuery to work, you must index the values using a one of the numeric fields ([](xref:Lucene.Net.Documents.IntField IntField), [](xref:Lucene.Net.Documents.LongField LongField), [](xref:Lucene.Net.Documents.FloatField FloatField), or [](xref:Lucene.Net.Documents.DoubleField DoubleField)). + +#### + [](xref:Lucene.Net.Search.PrefixQuery PrefixQuery), + [](xref:Lucene.Net.Search.WildcardQuery WildcardQuery), + [](xref:Lucene.Net.Search.RegexpQuery RegexpQuery) + +While the [](xref:Lucene.Net.Search.PrefixQuery PrefixQuery) has a different implementation, it is essentially a special case of the [](xref:Lucene.Net.Search.WildcardQuery WildcardQuery). The [](xref:Lucene.Net.Search.PrefixQuery PrefixQuery) allows an application to identify all documents with terms that begin with a certain string. The [](xref:Lucene.Net.Search.WildcardQuery WildcardQuery) generalizes this by allowing for the use of <tt>*</tt> (matches 0 or more characters) and <tt>?</tt> (matches exactly one character) wildcards. Note that the [](xref:Lucene.Net.Search.WildcardQuery WildcardQuery) can be quite slow. Also note that [](xref:Lucene.Net.Search.WildcardQuery WildcardQuery) should not start with <tt>*</tt> and <tt>?</tt>, as these are extremely slow. Some QueryParsers may not allow this by default, but provide a `setAllowLeadingWildcard` method to remove that protection. The [](xref:Lucene.Net.Search.RegexpQuery RegexpQuery) is even more general than WildcardQuery, al lowing an application to identify all documents with terms that match a regular expression pattern. + +#### + [](xref:Lucene.Net.Search.FuzzyQuery FuzzyQuery) + +A [](xref:Lucene.Net.Search.FuzzyQuery FuzzyQuery) matches documents that contain terms similar to the specified term. Similarity is determined using [Levenshtein (edit) distance](http://en.wikipedia.org/wiki/Levenshtein). This type of query can be useful when accounting for spelling variations in the collection. + +## Scoring â Introduction + +Lucene scoring is the heart of why we all love Lucene. It is blazingly fast and it hides almost all of the complexity from the user. In a nutshell, it works. At least, that is, until it doesn't work, or doesn't work as one would expect it to work. Then we are left digging into Lucene internals or asking for help on [[email protected]](mailto:[email protected]) to figure out why a document with five of our query terms scores lower than a different document with only one of the query terms. + +While this document won't answer your specific scoring issues, it will, hopefully, point you to the places that can help you figure out the *what* and *why* of Lucene scoring. + +Lucene scoring supports a number of pluggable information retrieval [models](http://en.wikipedia.org/wiki/Information_retrieval#Model_types), including: * [Vector Space Model (VSM)](http://en.wikipedia.org/wiki/Vector_Space_Model) * [Probablistic Models](http://en.wikipedia.org/wiki/Probabilistic_relevance_model) such as [Okapi BM25](http://en.wikipedia.org/wiki/Probabilistic_relevance_model_(BM25)) and [DFR](http://en.wikipedia.org/wiki/Divergence-from-randomness_model) * [Language models](http://en.wikipedia.org/wiki/Language_model) These models can be plugged in via the [](xref:Lucene.Net.Search.Similarities Similarity API), and offer extension hooks and parameters for tuning. In general, Lucene first finds the documents that need to be scored based on boolean logic in the Query specification, and then ranks this subset of matching documents via the retrieval model. For some valuable references on VSM and IR in general refer to [Lucene Wiki IR references](http://wiki.apache.org/l ucene-java/InformationRetrieval). + +The rest of this document will cover [Scoring basics](#scoringBasics) and explain how to change your [](xref:Lucene.Net.Search.Similarities.Similarity Similarity). Next, it will cover ways you can customize the lucene internals in [Custom Queries -- Expert Level](#customQueriesExpert), which gives details on implementing your own [](xref:Lucene.Net.Search.Query Query) class and related functionality. Finally, we will finish up with some reference material in the [Appendix](#algorithm). + +## Scoring â Basics + +Scoring is very much dependent on the way documents are indexed, so it is important to understand + indexing. (see [Lucene overview]({@docRoot}/overview-summary.html#overview_description) + before continuing on with this section) Be sure to use the useful + [](xref:Lucene.Net.Search.IndexSearcher.Explain(Lucene.Net.Search.Query, int) IndexSearcher.Explain(Query, doc)) + to understand how the score for a certain matching document was + computed. + +Generally, the Query determines which documents match (a binary decision), while the Similarity determines how to assign scores to the matching documents. + +#### Fields and Documents + +In Lucene, the objects we are scoring are [](xref:Lucene.Net.Documents.Document Document)s. A Document is a collection of [](xref:Lucene.Net.Documents.Field Field)s. Each Field has [](xref:Lucene.Net.Documents.FieldType semantics) about how it is created and stored ([](xref:Lucene.Net.Documents.FieldType.Tokenized() tokenized), [](xref:Lucene.Net.Documents.FieldType.Stored() stored), etc). It is important to note that Lucene scoring works on Fields and then combines the results to return Documents. This is important because two Documents with the exact same content, but one having the content in two Fields and the other in one Field may return different scores for the same query due to length normalization. + +#### Score Boosting + +Lucene allows influencing search results by "boosting" at different times: * **Index-time boost** by calling [](xref:Lucene.Net.Documents.Field.SetBoost(float) Field.SetBoost()) before a document is added to the index. * **Query-time boost** by setting a boost on a query clause, calling [](xref:Lucene.Net.Search.Query.SetBoost(float) Query.SetBoost()). + +Indexing time boosts are pre-processed for storage efficiency and written to storage for a field as follows: * All boosts of that field (i.e. all boosts under the same field name in that doc) are multiplied. * The boost is then encoded into a normalization value by the Similarity object at index-time: [](xref:Lucene.Net.Search.Similarities.Similarity.ComputeNorm computeNorm()). The actual encoding depends upon the Similarity implementation, but note that most use a lossy encoding (such as multiplying the boost with document length or similar, packed into a single byte!). * Decoding of any index-time normalization values and integration into the document's score is also performed at search time by the Similarity. + +## Changing Scoring â Similarity + + Changing [](xref:Lucene.Net.Search.Similarities.Similarity Similarity) is an easy way to influence scoring, this is done at index-time with [](xref:Lucene.Net.Index.IndexWriterConfig.SetSimilarity(Lucene.Net.Search.Similarities.Similarity) IndexWriterConfig.SetSimilarity(Similarity)) and at query-time with [](xref:Lucene.Net.Search.IndexSearcher.SetSimilarity(Lucene.Net.Search.Similarities.Similarity) IndexSearcher.SetSimilarity(Similarity)). Be sure to use the same Similarity at query-time as at index-time (so that norms are encoded/decoded correctly); Lucene makes no effort to verify this. + + You can influence scoring by configuring a different built-in Similarity implementation, or by tweaking its parameters, subclassing it to override behavior. Some implementations also offer a modular API which you can extend by plugging in a different component (e.g. term frequency normalizer). + + Finally, you can extend the low level [](xref:Lucene.Net.Search.Similarities.Similarity Similarity) directly to implement a new retrieval model, or to use external scoring factors particular to your application. For example, a custom Similarity can access per-document values via [](xref:Lucene.Net.Search.FieldCache FieldCache) or [](xref:Lucene.Net.Index.NumericDocValues) and integrate them into the score. + + See the [](xref:Lucene.Net.Search.Similarities) package documentation for information on the built-in available scoring models and extending or changing Similarity. + +## Custom Queries â Expert Level + +Custom queries are an expert level task, so tread carefully and be prepared to share your code if you want help. + +With the warning out of the way, it is possible to change a lot more than just the Similarity when it comes to matching and scoring in Lucene. Lucene's search is a complex mechanism that is grounded by <span>three main classes</span>: 1. [](xref:Lucene.Net.Search.Query Query) â The abstract object representation of the user's information need. 2. [](xref:Lucene.Net.Search.Weight Weight) â The internal interface representation of the user's Query, so that Query objects may be reused. This is global (across all segments of the index) and generally will require global statistics (such as docFreq for a given term across all segments). 3. [](xref:Lucene.Net.Search.Scorer Scorer) â An abstract class containing common functionality for scoring. Provides both scoring and explanation capabilities. This is created per-segment. 4. [](xref:Lucene.Net.Search.BulkScorer BulkScorer) â An abstract class that scores a range of documents. A default implementation simply iterates through the h its from [](xref:Lucene.Net.Search.Scorer Scorer), but some queries such as [](xref:Lucene.Net.Search.BooleanQuery BooleanQuery) have more efficient implementations. Details on each of these classes, and their children, can be found in the subsections below. + +#### The Query Class + +In some sense, the [](xref:Lucene.Net.Search.Query Query) class is where it all begins. Without a Query, there would be nothing to score. Furthermore, the Query class is the catalyst for the other scoring classes as it is often responsible for creating them or coordinating the functionality between them. The [](xref:Lucene.Net.Search.Query Query) class has several methods that are important for derived classes: 1. [](xref:Lucene.Net.Search.Query.CreateWeight(IndexSearcher) createWeight(IndexSearcher searcher)) â A [](xref:Lucene.Net.Search.Weight Weight) is the internal representation of the Query, so each Query implementation must provide an implementation of Weight. See the subsection on [The Weight Interface](#weightClass) below for details on implementing the Weight interface. 2. [](xref:Lucene.Net.Search.Query.Rewrite(IndexReader) rewrite(IndexReader reader)) â Rewrites queries into primitive queries. Primitive queries are: [](xref:Lucene.Net.Search.TermQuery TermQuery), [] (xref:Lucene.Net.Search.BooleanQuery BooleanQuery), <span>and other queries that implement [](xref:Lucene.Net.Search.Query.CreateWeight(IndexSearcher) createWeight(IndexSearcher searcher))</span> + +#### The Weight Interface + +The [](xref:Lucene.Net.Search.Weight Weight) interface provides an internal representation of the Query so that it can be reused. Any [](xref:Lucene.Net.Search.IndexSearcher IndexSearcher) dependent state should be stored in the Weight implementation, not in the Query class. The interface defines five methods that must be implemented: 1. [](xref:Lucene.Net.Search.Weight.GetQuery getQuery()) â Pointer to the Query that this Weight represents. 2. [](xref:Lucene.Net.Search.Weight.GetValueForNormalization() getValueForNormalization()) â A weight can return a floating point value to indicate its magnitude for query normalization. Typically a weight such as TermWeight that scores via a [](xref:Lucene.Net.Search.Similarities.Similarity Similarity) will just defer to the Similarity's implementation: [](xref:Lucene.Net.Search.Similarities.Similarity.SimWeight.GetValueForNormalization SimWeight.getValueForNormalization()). For example, with [](xref:Lucene.Net.Search.Similarities.TFIDFSimi larity Lucene's classic vector-space formula), this is implemented as the sum of squared weights: `` 3. [](xref:Lucene.Net.Search.Weight.Normalize(float,float) normalize(float norm, float topLevelBoost)) â Performs query normalization: * `topLevelBoost`: A query-boost factor from any wrapping queries that should be multiplied into every document's score. For example, a TermQuery that is wrapped within a BooleanQuery with a boost of `5` would receive this value at this time. This allows the TermQuery (the leaf node in this case) to compute this up-front a single time (e.g. by multiplying into the IDF), rather than for every document. * `norm`: Passes in a a normalization factor which may allow for comparing scores between queries. Typically a weight such as TermWeight that scores via a [](xref:Lucene.Net.Search.Similarities.Similarity Similarity) will just defer to the Similarity's implementation: [](xref:Lucene.Net.Search.Similarities.Similarity.SimWeight.Normalize SimWeight.norma lize(float,float)). 4. [](xref:Lucene.Net.Search.Weight.Scorer(Lucene.Net.Index.AtomicReaderContext, Lucene.Net.Util.Bits) scorer(AtomicReaderContext context, Bits acceptDocs)) â Construct a new [](xref:Lucene.Net.Search.Scorer Scorer) for this Weight. See [The Scorer Class](#scorerClass) below for help defining a Scorer. As the name implies, the Scorer is responsible for doing the actual scoring of documents given the Query. 5. [](xref:Lucene.Net.Search.Weight.BulkScorer(Lucene.Net.Index.AtomicReaderContext, boolean, Lucene.Net.Util.Bits) scorer(AtomicReaderContext context, boolean scoreDocsInOrder, Bits acceptDocs)) â Construct a new [](xref:Lucene.Net.Search.BulkScorer BulkScorer) for this Weight. See [The BulkScorer Class](#bulkScorerClass) below for help defining a BulkScorer. This is an optional method, and most queries do not implement it. 6. [](xref:Lucene.Net.Search.Weight.Explain(Lucene.Net.Index.AtomicReaderContext, int) explain(AtomicReaderContext context, int doc)) â Provide a means for explaining why a given document was scored the way it was. Typically a weight such as TermWeight that scores via a [](xref:Lucene.Net.Search.Similarities.Similarity Similarity) will make use of the Similarity's implementation: [](xref:Lucene.Net.Search.Similarities.Similarity.SimScorer.Explain(int, Explanation) SimScorer.explain(int doc, Explanation freq)). + +#### The Scorer Class + +The [](xref:Lucene.Net.Search.Scorer Scorer) abstract class provides common scoring functionality for all Scorer implementations and is the heart of the Lucene scoring process. The Scorer defines the following abstract (some of them are not yet abstract, but will be in future versions and should be considered as such now) methods which must be implemented (some of them inherited from [](xref:Lucene.Net.Search.DocIdSetIterator DocIdSetIterator)): 1. [](xref:Lucene.Net.Search.Scorer.NextDoc nextDoc()) â Advances to the next document that matches this Query, returning true if and only if there is another document that matches. 2. [](xref:Lucene.Net.Search.Scorer.DocID docID()) â Returns the id of the [](xref:Lucene.Net.Documents.Document Document) that contains the match. 3. [](xref:Lucene.Net.Search.Scorer.Score score()) â Return the score of the current document. This value can be determined in any appropriate way for an application. For instance, the [](xref:Lucene.Net.Search. TermScorer TermScorer) simply defers to the configured Similarity: [](xref:Lucene.Net.Search.Similarities.Similarity.SimScorer.Score(int, float) SimScorer.Score(int doc, float freq)). 4. [](xref:Lucene.Net.Search.Scorer.Freq freq()) â Returns the number of matches for the current document. This value can be determined in any appropriate way for an application. For instance, the [](xref:Lucene.Net.Search.TermScorer TermScorer) simply defers to the term frequency from the inverted index: [](xref:Lucene.Net.Index.DocsEnum.Freq DocsEnum.Freq()). 5. [](xref:Lucene.Net.Search.Scorer.Advance advance()) â Skip ahead in the document matches to the document whose id is greater than or equal to the passed in value. In many instances, advance can be implemented more efficiently than simply looping through all the matching documents until the target document is identified. 6. [](xref:Lucene.Net.Search.Scorer.GetChildren getChildren()) â Returns any child subscorers underneath this scorer. This allows for users to navigate the scorer hierarchy and receive more fine-grained details on the scoring process. + +#### The BulkScorer Class + +The [](xref:Lucene.Net.Search.BulkScorer BulkScorer) scores a range of documents. There is only one abstract method: 1. [](xref:Lucene.Net.Search.BulkScorer.Score(Lucene.Net.Search.Collector,int) score(Collector,int)) â Score all documents up to but not including the specified max document. + +#### Why would I want to add my own Query? + +In a nutshell, you want to add your own custom Query implementation when you think that Lucene's aren't appropriate for the task that you want to do. You might be doing some cutting edge research or you need more information back out of Lucene (similar to Doug adding SpanQuery functionality). + +## Appendix: Search Algorithm + +This section is mostly notes on stepping through the Scoring process and serves as fertilizer for the earlier sections. + +In the typical search application, a [](xref:Lucene.Net.Search.Query Query) is passed to the [](xref:Lucene.Net.Search.IndexSearcher IndexSearcher), beginning the scoring process. + +Once inside the IndexSearcher, a [](xref:Lucene.Net.Search.Collector Collector) is used for the scoring and sorting of the search results. These important objects are involved in a search: 1. The [](xref:Lucene.Net.Search.Weight Weight) object of the Query. The Weight object is an internal representation of the Query that allows the Query to be reused by the IndexSearcher. 2. The IndexSearcher that initiated the call. 3. A [](xref:Lucene.Net.Search.Filter Filter) for limiting the result set. Note, the Filter may be null. 4. A [](xref:Lucene.Net.Search.Sort Sort) object for specifying how to sort the results if the standard score-based sort method is not desired. + +Assuming we are not sorting (since sorting doesn't affect the raw Lucene score), we call one of the search methods of the IndexSearcher, passing in the [](xref:Lucene.Net.Search.Weight Weight) object created by [](xref:Lucene.Net.Search.IndexSearcher.CreateNormalizedWeight(Lucene.Net.Search.Query) IndexSearcher.CreateNormalizedWeight(Query)), [](xref:Lucene.Net.Search.Filter Filter) and the number of results we want. This method returns a [](xref:Lucene.Net.Search.TopDocs TopDocs) object, which is an internal collection of search results. The IndexSearcher creates a [](xref:Lucene.Net.Search.TopScoreDocCollector TopScoreDocCollector) and passes it along with the Weight, Filter to another expert search method (for more on the [](xref:Lucene.Net.Search.Collector Collector) mechanism, see [](xref:Lucene.Net.Search.IndexSearcher IndexSearcher)). The TopScoreDocCollector uses a [](xref:Lucene.Net.Util.PriorityQueue PriorityQueue) to collect the top results for the search. + +If a Filter is being used, some initial setup is done to determine which docs to include. Otherwise, we ask the Weight for a [](xref:Lucene.Net.Search.Scorer Scorer) for each [](xref:Lucene.Net.Index.IndexReader IndexReader) segment and proceed by calling [](xref:Lucene.Net.Search.BulkScorer.Score(Lucene.Net.Search.Collector) BulkScorer.Score(Collector)). + +At last, we are actually going to score some documents. The score method takes in the Collector (most likely the TopScoreDocCollector or TopFieldCollector) and does its business.Of course, here is where things get involved. The [](xref:Lucene.Net.Search.Scorer Scorer) that is returned by the [](xref:Lucene.Net.Search.Weight Weight) object depends on what type of Query was submitted. In most real world applications with multiple query terms, the [](xref:Lucene.Net.Search.Scorer Scorer) is going to be a `BooleanScorer2` created from [](xref:Lucene.Net.Search.BooleanQuery.BooleanWeight BooleanWeight) (see the section on [custom queries](#customQueriesExpert) for info on changing this). + +Assuming a BooleanScorer2, we first initialize the Coordinator, which is used to apply the coord() factor. We then get a internal Scorer based on the required, optional and prohibited parts of the query. Using this internal Scorer, the BooleanScorer2 then proceeds into a while loop based on the [](xref:Lucene.Net.Search.Scorer.NextDoc Scorer.NextDoc()) method. The nextDoc() method advances to the next document matching the query. This is an abstract method in the Scorer class and is thus overridden by all derived implementations. If you have a simple OR query your internal Scorer is most likely a DisjunctionSumScorer, which essentially combines the scorers from the sub scorers of the OR'd terms. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net/Store/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net/Store/package.md b/src/Lucene.Net/Store/package.md new file mode 100644 index 0000000..e8d2ba6 --- /dev/null +++ b/src/Lucene.Net/Store/package.md @@ -0,0 +1,19 @@ + +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Binary i/o API, used for all index data. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net/Util/Automaton/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net/Util/Automaton/package.md b/src/Lucene.Net/Util/Automaton/package.md new file mode 100644 index 0000000..e0243c0 --- /dev/null +++ b/src/Lucene.Net/Util/Automaton/package.md @@ -0,0 +1,43 @@ + +<!-- + dk.brics.automaton + + Copyright (c) 2001-2009 Anders Moeller + All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions + are met: + 1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + 2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + 3. The name of the author may not be used to endorse or promote products + derived from this software without specific prior written permission. + + THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR + IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES + OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. + IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, + INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT + NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF + THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +--> + +Finite-state automaton for regular expressions. + +This package contains a full DFA/NFA implementation with Unicode +alphabet and support for all standard (and a number of non-standard) +regular expression operations. + +The most commonly used functionality is located in the classes +<tt>[](xref:Lucene.Net.Util.Automaton.Automaton)</tt> and +<tt>[](xref:Lucene.Net.Util.Automaton.RegExp)</tt>. + +For more information, go to the package home page at +<tt>[http://www.brics.dk/automaton/](http://www.brics.dk/automaton/)</tt>. [email protected] \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net/Util/Fst/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net/Util/Fst/package.md b/src/Lucene.Net/Util/Fst/package.md new file mode 100644 index 0000000..3f58a80 --- /dev/null +++ b/src/Lucene.Net/Util/Fst/package.md @@ -0,0 +1,84 @@ + +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Finite state transducers + +This package implements [ +Finite State Transducers](http://en.wikipedia.org/wiki/Finite_state_transducer) with the following characteristics: + +* Fast and low memory overhead construction of the minimal FST + (but inputs must be provided in sorted order) +* Low object overhead and quick deserialization (byte[] representation) +* Optional two-pass compression: [](xref:Lucene.Net.Util.Fst.FST.Pack FST.Pack()) +* [](xref:Lucene.Net.Util.Fst.Util.GetByOutput Lookup-by-output) when the + outputs are in sorted order (e.g., ordinals or file pointers) +* Pluggable [](xref:Lucene.Net.Util.Fst.Outputs Outputs) representation +* [](xref:Lucene.Net.Util.Fst.Util.ShortestPaths N-shortest-paths) search by + weight +* Enumerators ([](xref:Lucene.Net.Util.Fst.IntsRefFSTEnum IntsRef) and [](xref:Lucene.Net.Util.Fst.BytesRefFSTEnum BytesRef)) that behave like {@link java.util.SortedMap SortedMap} iterators + +FST Construction example: + + // Input values (keys). These must be provided to Builder in Unicode sorted order! + String inputValues[] = {"cat", "dog", "dogs"}; + long outputValues[] = {5, 7, 12}; + + PositiveIntOutputs outputs = PositiveIntOutputs.getSingleton(); + Builder<Long> builder = new Builder<Long>(INPUT_TYPE.BYTE1, outputs); + BytesRef scratchBytes = new BytesRef(); + IntsRef scratchInts = new IntsRef(); + for (int i = 0; i < inputValues.length; i++) { + scratchBytes.copyChars(inputValues[i]); + builder.add(Util.toIntsRef(scratchBytes, scratchInts), outputValues[i]); + } + FST<Long> fst = builder.finish(); + +Retrieval by key: + + Long value = Util.get(fst, new BytesRef("dog")); + System.out.println(value); // 7 + +Retrieval by value: + + // Only works because outputs are also in sorted order + IntsRef key = Util.getByOutput(fst, 12); + System.out.println(Util.toBytesRef(key, scratchBytes).utf8ToString()); // dogs + +Iterate over key-value pairs in sorted order: + + // Like TermsEnum, this also supports seeking (advance) + BytesRefFSTEnum<Long> iterator = new BytesRefFSTEnum<Long>(fst); + while (iterator.next() != null) { + InputOutput<Long> mapEntry = iterator.current(); + System.out.println(mapEntry.input.utf8ToString()); + System.out.println(mapEntry.output); + } + +N-shortest paths by weight: + + Comparator<Long> comparator = new Comparator<Long>() { + public int compare(Long left, Long right) { + return left.compareTo(right); + } + }; + Arc<Long> firstArc = fst.getFirstArc(new Arc<Long>()); + MinResult<Long> paths[] = Util.shortestPaths(fst, firstArc, comparator, 2); + System.out.println(Util.toBytesRef(paths[0].input, scratchBytes).utf8ToString()); // cat + System.out.println(paths[0].output); // 5 + System.out.println(Util.toBytesRef(paths[1].input, scratchBytes).utf8ToString()); // dog + System.out.println(paths[1].output); // 7 \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net/Util/Mutable/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net/Util/Mutable/package.md b/src/Lucene.Net/Util/Mutable/package.md new file mode 100644 index 0000000..eda976e --- /dev/null +++ b/src/Lucene.Net/Util/Mutable/package.md @@ -0,0 +1,19 @@ + +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Comparable object wrappers \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net/Util/Packed/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net/Util/Packed/package.md b/src/Lucene.Net/Util/Packed/package.md new file mode 100644 index 0000000..24ac142 --- /dev/null +++ b/src/Lucene.Net/Util/Packed/package.md @@ -0,0 +1,71 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +## Packed integer arrays and streams. + + The packed package provides * sequential and random access capable arrays of positive longs, * routines for efficient serialization and deserialization of streams of packed integers. The implementations provide different trade-offs between memory usage and access speed. The standard usage scenario is replacing large int or long arrays in order to reduce the memory footprint. + + The main access point is the [](xref:Lucene.Net.Util.Packed.PackedInts) factory. + +### In-memory structures + +* **[](xref:Lucene.Net.Util.Packed.PackedInts.Mutable)** + + * Only supports positive longs. + * Requires the number of bits per value to be known in advance. + * Random-access for both writing and reading. +* **[](xref:Lucene.Net.Util.Packed.GrowableWriter)** + + * Same as PackedInts.Mutable but grows the number of bits per values when needed. + * Useful to build a PackedInts.Mutable from a read-once stream of longs. +* **[](xref:Lucene.Net.Util.Packed.PagedGrowableWriter)** + + * Slices data into fixed-size blocks stored in GrowableWriters. + * Supports more than 2B values. + * You should use Appending(Delta)PackedLongBuffer instead if you don't need random write access. +* **[](xref:Lucene.Net.Util.Packed.AppendingDeltaPackedLongBuffer)** + + * Can store any sequence of longs. + * Compression is good when values are close to each other. + * Supports random reads, but only sequential writes. + * Can address up to 2^42 values. +* **[](xref:Lucene.Net.Util.Packed.AppendingPackedLongBuffer)** + + * Same as AppendingDeltaPackedLongBuffer but assumes values are 0-based. +* **[](xref:Lucene.Net.Util.Packed.MonotonicAppendingLongBuffer)** + + * Same as AppendingDeltaPackedLongBuffer except that compression is good when the stream is a succession of affine functions. + +### Disk-based structures + +* **[](xref:Lucene.Net.Util.Packed.PackedInts.Writer), [](xref:Lucene.Net.Util.Packed.PackedInts.Reader), [](xref:Lucene.Net.Util.Packed.PackedInts.ReaderIterator)** + + * Only supports positive longs. + * Requires the number of bits per value to be known in advance. + * Supports both fast sequential access with low memory footprint with ReaderIterator and random-access by either loading values in memory or leaving them on disk with Reader. +* **[](xref:Lucene.Net.Util.Packed.BlockPackedWriter), [](xref:Lucene.Net.Util.Packed.BlockPackedReader), [](xref:Lucene.Net.Util.Packed.BlockPackedReaderIterator)** + + * Splits the stream into fixed-size blocks. + * Compression is good when values are close to each other. + * Can address up to 2B * blockSize values. +* **[](xref:Lucene.Net.Util.Packed.MonotonicBlockPackedWriter), [](xref:Lucene.Net.Util.Packed.MonotonicBlockPackedReader)** + + * Same as the non-monotonic variants except that compression is good when the stream is a succession of affine functions. + * The reason why there is no sequential access is that if you need sequential access, you should rather delta-encode and use BlockPackedWriter. +* **[](xref:Lucene.Net.Util.Packed.PackedDataOutput), [](xref:Lucene.Net.Util.Packed.PackedDataInput)** + + * Writes sequences of longs where each long can use any number of bits. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net/Util/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net/Util/package.md b/src/Lucene.Net/Util/package.md new file mode 100644 index 0000000..dcbfc19 --- /dev/null +++ b/src/Lucene.Net/Util/package.md @@ -0,0 +1,19 @@ + +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Some utility classes. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net/overview.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net/overview.md b/src/Lucene.Net/overview.md new file mode 100644 index 0000000..4237a1c --- /dev/null +++ b/src/Lucene.Net/overview.md @@ -0,0 +1,141 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Apache Lucene is a high-performance, full-featured text search engine library. Here's a simple example how to use Lucene for indexing and searching (using JUnit to check if the results are what we expect): + +<!-- = Java2Html Converter 5.0 [2006-03-04] by Markus Gebhard [email protected] = --> + + Analyzer analyzer = new StandardAnalyzer(Version.LUCENE_CURRENT); + + // Store the index in memory: + Directory directory = new RAMDirectory(); + // To store an index on disk, use this instead: + //Directory directory = FSDirectory.open("/tmp/testindex"); + IndexWriterConfig config = new IndexWriterConfig(Version.LUCENE_CURRENT, analyzer); + IndexWriter iwriter = new IndexWriter(directory, config); + Document doc = new Document(); + String text = "This is the text to be indexed."; + doc.add(new Field("fieldname", text, TextField.TYPE_STORED)); + iwriter.addDocument(doc); + iwriter.close(); + + // Now search the index: + DirectoryReader ireader = DirectoryReader.open(directory); + IndexSearcher isearcher = new IndexSearcher(ireader); + // Parse a simple query that searches for "text": + QueryParser parser = new QueryParser(Version.LUCENE_CURRENT, "fieldname", analyzer); + Query query = parser.parse("text"); + ScoreDoc[] hits = isearcher.search(query, null, 1000).scoreDocs; + assertEquals(1, hits.length); + // Iterate through the results: + for (int i = 0; i < hits.length;="" i++)="" {="" document="" hitdoc="isearcher.doc(hits[i].doc);" assertequals("this="" is="" the="" text="" to="" be="" indexed.",="" hitdoc.get("fieldname"));="" }="" ireader.close();=""> + +The Lucene API is divided into several packages: + +* **[](xref:Lucene.Net.Analysis)** +defines an abstract [](xref:Lucene.Net.Analysis.Analyzer Analyzer) +API for converting text from a {@link java.io.Reader} +into a [](xref:Lucene.Net.Analysis.TokenStream TokenStream), +an enumeration of token [](xref:Lucene.Net.Util.Attribute Attribute)s. +A TokenStream can be composed by applying [](xref:Lucene.Net.Analysis.TokenFilter TokenFilter)s +to the output of a [](xref:Lucene.Net.Analysis.Tokenizer Tokenizer). +Tokenizers and TokenFilters are strung together and applied with an [](xref:Lucene.Net.Analysis.Analyzer Analyzer). +[analyzers-common](../analyzers-common/overview-summary.html) provides a number of Analyzer implementations, including +[StopAnalyzer](../analyzers-common/org/apache/lucene/analysis/core/StopAnalyzer.html) +and the grammar-based [StandardAnalyzer](../analyzers-common/org/apache/lucene/analysis/standard/StandardAnalyzer.html). +* **[](xref:Lucene.Net.Codecs)** +provides an abstraction over the encoding and decoding of the inverted index structure, +as well as different implementations that can be chosen depending upon application needs. + + **[](xref:Lucene.Net.Documents)** +provides a simple [](xref:Lucene.Net.Documents.Document Document) +class. A Document is simply a set of named [](xref:Lucene.Net.Documents.Field Field)s, +whose values may be strings or instances of {@link java.io.Reader}. +* **[](xref:Lucene.Net.Index)** +provides two primary classes: [](xref:Lucene.Net.Index.IndexWriter IndexWriter), +which creates and adds documents to indices; and [](xref:Lucene.Net.Index.IndexReader), +which accesses the data in the index. +* **[](xref:Lucene.Net.Search)** +provides data structures to represent queries (ie [](xref:Lucene.Net.Search.TermQuery TermQuery) +for individual words, [](xref:Lucene.Net.Search.PhraseQuery PhraseQuery) +for phrases, and [](xref:Lucene.Net.Search.BooleanQuery BooleanQuery) +for boolean combinations of queries) and the [](xref:Lucene.Net.Search.IndexSearcher IndexSearcher) +which turns queries into [](xref:Lucene.Net.Search.TopDocs TopDocs). +A number of [QueryParser](../queryparser/overview-summary.html)s are provided for producing +query structures from strings or xml. + + **[](xref:Lucene.Net.Store)** +defines an abstract class for storing persistent data, the [](xref:Lucene.Net.Store.Directory Directory), +which is a collection of named files written by an [](xref:Lucene.Net.Store.IndexOutput IndexOutput) +and read by an [](xref:Lucene.Net.Store.IndexInput IndexInput). +Multiple implementations are provided, including [](xref:Lucene.Net.Store.FSDirectory FSDirectory), +which uses a file system directory to store files, and [](xref:Lucene.Net.Store.RAMDirectory RAMDirectory) +which implements files as memory-resident data structures. +* **[](xref:Lucene.Net.Util)** +contains a few handy data structures and util classes, ie [](xref:Lucene.Net.Util.OpenBitSet OpenBitSet) +and [](xref:Lucene.Net.Util.PriorityQueue PriorityQueue). +To use Lucene, an application should: + +1. Create [](xref:Lucene.Net.Documents.Document Document)s by +adding +[](xref:Lucene.Net.Documents.Field Field)s; +2. Create an [](xref:Lucene.Net.Index.IndexWriter IndexWriter) +and add documents to it with [](xref:Lucene.Net.Index.IndexWriter.AddDocument(Iterable) addDocument()); +3. Call [QueryParser.parse()](../queryparser/org/apache/lucene/queryparser/classic/QueryParserBase.html#parse(java.lang.String)) +to build a query from a string; and +4. Create an [](xref:Lucene.Net.Search.IndexSearcher IndexSearcher) +and pass the query to its [](xref:Lucene.Net.Search.IndexSearcher.Search(Lucene.Net.Search.Query, int) search()) +method. +Some simple examples of code which does this are: + +*  [IndexFiles.java](../demo/src-html/org/apache/lucene/demo/IndexFiles.html) creates an +index for all the files contained in a directory. +*  [SearchFiles.java](../demo/src-html/org/apache/lucene/demo/SearchFiles.html) prompts for +queries and searches an index. +To demonstrate these, try something like: + +> <tt>> **java -cp lucene-core.jar:lucene-demo.jar:lucene-analyzers-common.jar org.apache.lucene.demo.IndexFiles -index index -docs rec.food.recipes/soups**</tt> +> +> <tt>adding rec.food.recipes/soups/abalone-chowder</tt> +> +> <tt> </tt>[ ... ] +> +> <tt>> **java -cp lucene-core.jar:lucene-demo.jar:lucene-queryparser.jar:lucene-analyzers-common.jar org.apache.lucene.demo.SearchFiles**</tt> +> +> <tt>Query: **chowder**</tt> +> +> <tt>Searching for: chowder</tt> +> +> <tt>34 total matching documents</tt> +> +> <tt>1. rec.food.recipes/soups/spam-chowder</tt> +> +> <tt> </tt>[ ... thirty-four documents contain the word "chowder" ... ] +> +> <tt>Query: **"clam chowder" AND Manhattan**</tt> +> +> <tt>Searching for: +"clam chowder" +manhattan</tt> +> +> <tt>2 total matching documents</tt> +> +> <tt>1. rec.food.recipes/soups/clam-chowder</tt> +> +> <tt> </tt>[ ... two documents contain the phrase "clam chowder" +> and the word "manhattan" ... ] +> +>    [ Note: "+" and "-" are canonical, but "AND", "OR" +> and "NOT" may be used. ] \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net/package.md b/src/Lucene.Net/package.md new file mode 100644 index 0000000..bb71718 --- /dev/null +++ b/src/Lucene.Net/package.md @@ -0,0 +1,17 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +Top-level package. \ No newline at end of file
