http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.Facet/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.Facet/package.md b/src/Lucene.Net.Facet/package.md new file mode 100644 index 0000000..e3caf3e --- /dev/null +++ b/src/Lucene.Net.Facet/package.md @@ -0,0 +1,24 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# faceted search + + This module provides multiple methods for computing facet counts and value aggregations: * Taxonomy-based methods rely on a separate taxonomy index to map hierarchical facet paths to global int ordinals for fast counting at search time; these methods can compute counts (([](xref:Lucene.Net.Facet.Taxonomy.FastTaxonomyFacetCounts), [](xref:Lucene.Net.Facet.Taxonomy.TaxonomyFacetCounts)) aggregate long or double values [](xref:Lucene.Net.Facet.Taxonomy.TaxonomyFacetSumIntAssociations), [](xref:Lucene.Net.Facet.Taxonomy.TaxonomyFacetSumFloatAssociations), [](xref:Lucene.Net.Facet.Taxonomy.TaxonomyFacetSumValueSource). Add [](xref:Lucene.Net.Facet.FacetField) or [](xref:Lucene.Net.Facet.Taxonomy.AssociationFacetField) to your documents at index time to use taxonomy-based methods. Sorted-set doc values method does not require a separate taxonomy index, and computes counts based on sorted set doc values fields ([](xref:Lucene.Net.Facet.Sortedset.SortedSetDocValuesFacetCounts)). Add [](xre f:Lucene.Net.Facet.Sortedset.SortedSetDocValuesFacetField) to your documents at index time to use sorted set facet counts. Range faceting [](xref:Lucene.Net.Facet.Range.LongRangeFacetCounts), [](xref:Lucene.Net.Facet.Range.DoubleRangeFacetCounts) compute counts for a dynamic numeric range from a provided [](xref:Lucene.Net.Queries.Function.ValueSource) (previously indexed numeric field, or a dynamic expression such as distance). + + At search time you first run your search, but pass a [](xref:Lucene.Net.Facet.FacetsCollector) to gather all hits (and optionally, scores for each hit). Then, instantiate whichever facet methods you'd like to use to compute aggregates. Finally, all methods implement a common [](xref:Lucene.Net.Facet.Facets) base API that you use to obtain specific facet counts. + + The various [](xref:Lucene.Net.Facet.FacetsCollector.Search) utility methods are useful for doing an "ordinary" search (sorting by score, or by a specified Sort) but also collecting into a [](xref:Lucene.Net.Facet.FacetsCollector) for subsequent faceting. \ No newline at end of file
http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.Grouping/Function/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.Grouping/Function/package.md b/src/Lucene.Net.Grouping/Function/package.md new file mode 100644 index 0000000..f6cb8a3 --- /dev/null +++ b/src/Lucene.Net.Grouping/Function/package.md @@ -0,0 +1,18 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Support for grouping by [](xref:Lucene.Net.Queries.Function.ValueSource). \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.Grouping/Term/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.Grouping/Term/package.md b/src/Lucene.Net.Grouping/Term/package.md new file mode 100644 index 0000000..f7dbcef --- /dev/null +++ b/src/Lucene.Net.Grouping/Term/package.md @@ -0,0 +1,18 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Support for grouping by indexed terms via [](xref:Lucene.Net.Search.FieldCache). \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.Grouping/overview.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.Grouping/overview.md b/src/Lucene.Net.Grouping/overview.md new file mode 100644 index 0000000..ca6edde --- /dev/null +++ b/src/Lucene.Net.Grouping/overview.md @@ -0,0 +1,18 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Lucene's grouping module \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.Grouping/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.Grouping/package.md b/src/Lucene.Net.Grouping/package.md new file mode 100644 index 0000000..b5668ef --- /dev/null +++ b/src/Lucene.Net.Grouping/package.md @@ -0,0 +1,147 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +This module enables search result grouping with Lucene, where hits with the same value in the specified single-valued group field are grouped together. For example, if you group by the `author` field, then all documents with the same value in the `author` field fall into a single group. + +Grouping requires a number of inputs: + +* `groupField`: this is the field used for grouping. + For example, if you use the `author` field then each + group has all books by the same author. Documents that don't + have this field are grouped under a single group with + a `null` group value. + + `groupSort`: how the groups are sorted. For sorting + purposes, each group is "represented" by the highest-sorted + document according to the `groupSort` within it. For + example, if you specify "price" (ascending) then the first group + is the one with the lowest price book within it. Or if you + specify relevance group sort, then the first group is the one + containing the highest scoring book. + + `topNGroups`: how many top groups to keep. For + example, 10 means the top 10 groups are computed. + + `groupOffset`: which "slice" of top groups you want to + retrieve. For example, 3 means you'll get 7 groups back + (assuming `topNGroups` is 10). This is useful for + paging, where you might show 5 groups per page. + + `withinGroupSort`: how the documents within each group + are sorted. This can be different from the group sort. + + `maxDocsPerGroup`: how many top documents within each + group to keep. + + `withinGroupOffset`: which "slice" of top + documents you want to retrieve from each group. + +The implementation is two-pass: the first pass ([](xref:Lucene.Net.Search.Grouping.Term.TermFirstPassGroupingCollector)) gathers the top groups, and the second pass ([](xref:Lucene.Net.Search.Grouping.Term.TermSecondPassGroupingCollector)) gathers documents within those groups. If the search is costly to run you may want to use the [](xref:Lucene.Net.Search.CachingCollector) class, which caches hits and can (quickly) replay them for the second pass. This way you only run the query once, but you pay a RAM cost to (briefly) hold all hits. Results are returned as a [](xref:Lucene.Net.Search.Grouping.TopGroups) instance. + + This module abstracts away what defines group and how it is collected. All grouping collectors are abstract and have currently term based implementations. One can implement collectors that for example group on multiple fields. + +Known limitations: + +* For the two-pass grouping search, the group field must be a + single-valued indexed field (or indexed as a [](xref:Lucene.Net.Documents.SortedDocValuesField)). + [](xref:Lucene.Net.Search.FieldCache) is used to load the [](xref:Lucene.Net.Index.SortedDocValues) for this field. + Although Solr support grouping by function and this module has abstraction of what a group is, there are currently only + implementations for grouping based on terms. + Sharding is not directly supported, though is not too + difficult, if you can merge the top groups and top documents per + group yourself. + +Typical usage for the generic two-pass grouping search looks like this using the grouping convenience utility (optionally using caching for the second pass search): + + GroupingSearch groupingSearch = new GroupingSearch("author"); + groupingSearch.setGroupSort(groupSort); + groupingSearch.setFillSortFields(fillFields); + + if (useCache) { + // Sets cache in MB + groupingSearch.setCachingInMB(4.0, true); + } + + if (requiredTotalGroupCount) { + groupingSearch.setAllGroups(true); + } + + TermQuery query = new TermQuery(new Term("content", searchTerm)); + TopGroups<BytesRef> result = groupingSearch.search(indexSearcher, query, groupOffset, groupLimit); + + // Render groupsResult... + if (requiredTotalGroupCount) { + int totalGroupCount = result.totalGroupCount; + } + +To use the single-pass `BlockGroupingCollector`, first, at indexing time, you must ensure all docs in each group are added as a block, and you have some way to find the last document of each group. One simple way to do this is to add a marker binary field: + + // Create Documents from your source: + List<Document> oneGroup = ...; + + Field groupEndField = new Field("groupEnd", "x", Field.Store.NO, Field.Index.NOT_ANALYZED); + groupEndField.setIndexOptions(IndexOptions.DOCS_ONLY); + groupEndField.setOmitNorms(true); + oneGroup.get(oneGroup.size()-1).add(groupEndField); + + // You can also use writer.updateDocuments(); just be sure you + // replace an entire previous doc block with this new one. For + // example, each group could have a "groupID" field, with the same + // value for all docs in this group: + writer.addDocuments(oneGroup); + +Then, at search time, do this up front: + + // Set this once in your app & save away for reusing across all queries: + Filter groupEndDocs = new CachingWrapperFilter(new QueryWrapperFilter(new TermQuery(new Term("groupEnd", "x")))); + +Finally, do this per search: + + // Per search: + BlockGroupingCollector c = new BlockGroupingCollector(groupSort, groupOffset+topNGroups, needsScores, groupEndDocs); + s.search(new TermQuery(new Term("content", searchTerm)), c); + TopGroups groupsResult = c.getTopGroups(withinGroupSort, groupOffset, docOffset, docOffset+docsPerGroup, fillFields); + + // Render groupsResult... + +Or alternatively use the `GroupingSearch` convenience utility: + + // Per search: + GroupingSearch groupingSearch = new GroupingSearch(groupEndDocs); + groupingSearch.setGroupSort(groupSort); + groupingSearch.setIncludeScores(needsScores); + TermQuery query = new TermQuery(new Term("content", searchTerm)); + TopGroups groupsResult = groupingSearch.search(indexSearcher, query, groupOffset, groupLimit); + + // Render groupsResult... + +Note that the `groupValue` of each `GroupDocs` +will be `null`, so if you need to present this value you'll +have to separately retrieve it (for example using stored +fields, `FieldCache`, etc.). + +Another collector is the `TermAllGroupHeadsCollector` that can be used to retrieve all most relevant documents per group. Also known as group heads. This can be useful in situations when one wants to compute group based facets / statistics on the complete query result. The collector can be executed during the first or second phase. This collector can also be used with the `GroupingSearch` convenience utility, but when if one only wants to compute the most relevant documents per group it is better to just use the collector as done here below. + + AbstractAllGroupHeadsCollector c = TermAllGroupHeadsCollector.create(groupField, sortWithinGroup); + s.search(new TermQuery(new Term("content", searchTerm)), c); + // Return all group heads as int array + int[] groupHeadsArray = c.retrieveGroupHeads() + // Return all group heads as FixedBitSet. + int maxDoc = s.maxDoc(); + FixedBitSet groupHeadsBitSet = c.retrieveGroupHeads(maxDoc) + +For each of the above collector types there is also a variant that works with `ValueSource` instead of of fields. Concretely this means that these variants can work with functions. These variants are slower than there term based counter parts. These implementations are located in the `org.apache.lucene.search.grouping.function` package, but can also be used with the `GroupingSearch` convenience utility \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.Highlighter/Highlight/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.Highlighter/Highlight/package.md b/src/Lucene.Net.Highlighter/Highlight/package.md new file mode 100644 index 0000000..3b5f033 --- /dev/null +++ b/src/Lucene.Net.Highlighter/Highlight/package.md @@ -0,0 +1,71 @@ + +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +The highlight package contains classes to provide "keyword in context" features +typically used to highlight search terms in the text of results pages. +The Highlighter class is the central component and can be used to extract the +most interesting sections of a piece of text and highlight them, with the help of +Fragmenter, fragment Scorer, and Formatter classes. + +## Example Usage + + //... Above, create documents with two fields, one with term vectors (tv) and one without (notv) + IndexSearcher searcher = new IndexSearcher(directory); + QueryParser parser = new QueryParser("notv", analyzer); + Query query = parser.parse("million"); + + TopDocs hits = searcher.search(query, 10); + + SimpleHTMLFormatter htmlFormatter = new SimpleHTMLFormatter(); + Highlighter highlighter = new Highlighter(htmlFormatter, new QueryScorer(query)); + for (int i = 0; i < 10;="" i++)="" {="" int="" id="hits.scoreDocs[i].doc;" document="" doc="searcher.doc(id);" string="" text="doc.get(" notv");"="" tokenstream="" tokenstream="TokenSources.getAnyTokenStream(searcher.getIndexReader()," id,="" "notv",="" analyzer);="" textfragment[]="" frag="highlighter.getBestTextFragments(tokenStream," text,="" false,="" 10);//highlighter.getbestfragments(tokenstream,="" text,="" 3,="" "...");="" for="" (int="" j="0;" j="">< frag.length;="" j++)="" {="" if="" ((frag[j]="" !="null)" &&="" (frag[j].getscore()=""> 0)) { + System.out.println((frag[j].toString())); + } + } + //Term vector + text = doc.get("tv"); + tokenStream = TokenSources.getAnyTokenStream(searcher.getIndexReader(), hits.scoreDocs[i].doc, "tv", analyzer); + frag = highlighter.getBestTextFragments(tokenStream, text, false, 10); + for (int j = 0; j < frag.length;="" j++)="" {="" if="" ((frag[j]="" !="null)" &&="" (frag[j].getscore()=""> 0)) { + System.out.println((frag[j].toString())); + } + } + System.out.println("-------------"); + } + +## New features 06/02/2005 + +This release adds options for encoding (thanks to Nicko Cadell). +An "Encoder" implementation such as the new SimpleHTMLEncoder class can be passed to the highlighter to encode +all those non-xhtml standard characters such as & into legal values. This simple class may not suffice for +some languages - Commons Lang has an implementation that could be used: escapeHtml(String) in +http://svn.apache.org/viewcvs.cgi/jakarta/commons/proper/lang/trunk/src/java/org/apache/commons/lang/StringEscapeUtils.java?rev=137958&view=markup + +## New features 22/12/2004 + +This release adds some new capabilities: + +1. Faster highlighting using Term vector support +2. New formatting options to use color intensity to show informational value +3. Options for better summarization by using term IDF scores to influence fragment selection + + The highlighter takes a TokenStream as input. Until now these streams have typically been produced using an Analyzer but the new class TokenSources provides helper methods for obtaining TokenStreams from the new TermVector position support (see latest CVS version). + +The new class GradientFormatter can use a scale of colors to highlight terms according to their score. A subtle use of color can help emphasise the reasons for matching (useful when doing "MoreLikeThis" queries and you want to see what the basis of the similarities are). + +The QueryScorer class has a new constructor which can use an IndexReader to derive the IDF (inverse document frequency) for each term in order to influence the score. This is useful for helping to extracting the most significant sections of a document and in supplying scores used by the new GradientFormatter to color significant words more strongly. The QueryScorer.getMaxWeight method is useful when passed to the GradientFormatter constructor to define the top score which is associated with the top color. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.Highlighter/PostingsHighlight/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.Highlighter/PostingsHighlight/package.md b/src/Lucene.Net.Highlighter/PostingsHighlight/package.md new file mode 100644 index 0000000..a3fac20 --- /dev/null +++ b/src/Lucene.Net.Highlighter/PostingsHighlight/package.md @@ -0,0 +1,19 @@ + +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Highlighter implementation that uses offsets from postings lists. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.Highlighter/VectorHighlight/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.Highlighter/VectorHighlight/package.md b/src/Lucene.Net.Highlighter/VectorHighlight/package.md new file mode 100644 index 0000000..75f50d9 --- /dev/null +++ b/src/Lucene.Net.Highlighter/VectorHighlight/package.md @@ -0,0 +1,175 @@ + +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +This is an another highlighter implementation. + +## Features + +* fast for large docs +* support N-gram fields +* support phrase-unit highlighting with slops +* support multi-term (includes wildcard, range, regexp, etc) queries +* need Java 1.5 +* highlight fields need to be stored with Positions and Offsets +* take into account query boost and/or IDF-weight to score fragments +* support colored highlight tags +* pluggable FragListBuilder / FieldFragList +* pluggable FragmentsBuilder + +## Algorithm + +To explain the algorithm, let's use the following sample text (to be highlighted) and user query: + +<table border="1"> +<tr> +<td>**Sample Text**</td> +<td>Lucene is a search engine library.</td> +</tr> +<tr> +<td>**User Query**</td> +<td>Lucene^2 OR "search library"~1</td> +</tr> +</table> + +The user query is a BooleanQuery that consists of TermQuery("Lucene") with boost of 2 and PhraseQuery("search library") with slop of 1. + +For your convenience, here is the offsets and positions info of the sample text. + + +--------+-----------------------------------+ + | | 1111111111222222222233333| + | offset|01234567890123456789012345678901234| + +--------+-----------------------------------+ + |document|Lucene is a search engine library. | + +--------*-----------------------------------+ + |position|0 1 2 3 4 5 | + +--------*-----------------------------------+ + +### Step 1. + +In Step 1, Fast Vector Highlighter generates [](xref:Lucene.Net.Search.VectorHighlight.FieldQuery.QueryPhraseMap) from the user query. `QueryPhraseMap` consists of the following members: + + public class QueryPhraseMap { + boolean terminal; + int slop; // valid if terminal == true and phraseHighlight == true + float boost; // valid if terminal == true + Map<String, QueryPhraseMap> subMap; + } + +`QueryPhraseMap` has subMap. The key of the subMap is a term text in the user query and the value is a subsequent `QueryPhraseMap`. If the query is a term (not phrase), then the subsequent `QueryPhraseMap` is marked as terminal. If the query is a phrase, then the subsequent `QueryPhraseMap` is not a terminal and it has the next term text in the phrase. + +From the sample user query, the following `QueryPhraseMap` will be generated: + + QueryPhraseMap + +--------+-+ +-------+-+ + |"Lucene"|o+->|boost=2|*| * : terminal + +--------+-+ +-------+-+ + + +--------+-+ +---------+-+ +-------+------+-+ + |"search"|o+->|"library"|o+->|boost=1|slop=1|*| + +--------+-+ +---------+-+ +-------+------+-+ + +### Step 2. + +In Step 2, Fast Vector Highlighter generates [](xref:Lucene.Net.Search.VectorHighlight.FieldTermStack). Fast Vector Highlighter uses term vector data (must be stored [](xref:Lucene.Net.Documents.FieldType.SetStoreTermVectorOffsets(boolean)) and [](xref:Lucene.Net.Documents.FieldType.SetStoreTermVectorPositions(boolean))) to generate it. `FieldTermStack` keeps the terms in the user query. Therefore, in this sample case, Fast Vector Highlighter generates the following `FieldTermStack`: + + FieldTermStack + +------------------+ + |"Lucene"(0,6,0) | + +------------------+ + |"search"(12,18,3) | + +------------------+ + |"library"(26,33,5)| + +------------------+ + where : "termText"(startOffset,endOffset,position) + +### Step 3. + +In Step 3, Fast Vector Highlighter generates [](xref:Lucene.Net.Search.VectorHighlight.FieldPhraseList) by reference to `QueryPhraseMap` and `FieldTermStack`. + + FieldPhraseList + +----------------+-----------------+---+ + |"Lucene" |[(0,6)] |w=2| + +----------------+-----------------+---+ + |"search library"|[(12,18),(26,33)]|w=1| + +----------------+-----------------+---+ + +The type of each entry is `WeightedPhraseInfo` that consists of an array of terms offsets and weight. + +### Step 4. + +In Step 4, Fast Vector Highlighter creates `FieldFragList` by reference to `FieldPhraseList`. In this sample case, the following `FieldFragList` will be generated: + + FieldFragList + +---------------------------------+ + |"Lucene"[(0,6)] | + |"search library"[(12,18),(26,33)]| + |totalBoost=3 | + +---------------------------------+ + +The calculation for each `FieldFragList.WeightedFragInfo.totalBoost` (weight) +depends on the implementation of `FieldFragList.add( ... )`: + + public void add( int startOffset, int endOffset, List<WeightedPhraseInfo> phraseInfoList ) { + float totalBoost = 0; + List<SubInfo> subInfos = new ArrayList<SubInfo>(); + for( WeightedPhraseInfo phraseInfo : phraseInfoList ){ + subInfos.add( new SubInfo( phraseInfo.getText(), phraseInfo.getTermsOffsets(), phraseInfo.getSeqnum() ) ); + totalBoost += phraseInfo.getBoost(); + } + getFragInfos().add( new WeightedFragInfo( startOffset, endOffset, subInfos, totalBoost ) ); + } + +The used implementation of `FieldFragList` is noted in `BaseFragListBuilder.createFieldFragList( ... )`: + + public FieldFragList createFieldFragList( FieldPhraseList fieldPhraseList, int fragCharSize ){ + return createFieldFragList( fieldPhraseList, new SimpleFieldFragList( fragCharSize ), fragCharSize ); + } + + Currently there are basically to approaches available: + +* `SimpleFragListBuilder using SimpleFieldFragList`: *sum-of-boosts*-approach. The totalBoost is calculated by summarizing the query-boosts per term. Per default a term is boosted by 1.0 +* `WeightedFragListBuilder using WeightedFieldFragList`: *sum-of-distinct-weights*-approach. The totalBoost is calculated by summarizing the IDF-weights of distinct terms. + +Comparison of the two approaches: + +<table border="1"> +<caption> + query = das alte testament (The Old Testament) +</caption> +<tr><th>Terms in fragment</th><th>sum-of-distinct-weights</th><th>sum-of-boosts</th></tr> +<tr><td>das alte testament</td><td>5.339621</td><td>3.0</td></tr> +<tr><td>das alte testament</td><td>5.339621</td><td>3.0</td></tr> +<tr><td>das testament alte</td><td>5.339621</td><td>3.0</td></tr> +<tr><td>das alte testament</td><td>5.339621</td><td>3.0</td></tr> +<tr><td>das testament</td><td>2.9455688</td><td>2.0</td></tr> +<tr><td>das alte</td><td>2.4759595</td><td>2.0</td></tr> +<tr><td>das das das das</td><td>1.5015357</td><td>4.0</td></tr> +<tr><td>das das das</td><td>1.3003681</td><td>3.0</td></tr> +<tr><td>das das</td><td>1.061746</td><td>2.0</td></tr> +<tr><td>alte</td><td>1.0</td><td>1.0</td></tr> +<tr><td>alte</td><td>1.0</td><td>1.0</td></tr> +<tr><td>das</td><td>0.7507678</td><td>1.0</td></tr> +<tr><td>das</td><td>0.7507678</td><td>1.0</td></tr> +<tr><td>das</td><td>0.7507678</td><td>1.0</td></tr> +<tr><td>das</td><td>0.7507678</td><td>1.0</td></tr> +<tr><td>das</td><td>0.7507678</td><td>1.0</td></tr> +</table> + +### Step 5. + +In Step 5, by using `FieldFragList` and the field stored data, Fast Vector Highlighter creates highlighted snippets! \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.Highlighter/overview.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.Highlighter/overview.md b/src/Lucene.Net.Highlighter/overview.md new file mode 100644 index 0000000..29046cb --- /dev/null +++ b/src/Lucene.Net.Highlighter/overview.md @@ -0,0 +1,23 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + + <title> + Highlighter + </title> + + The highlight package contains classes to provide "keyword in context" features + typically used to highlight search terms in the text of results pages. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.Join/overview.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.Join/overview.md b/src/Lucene.Net.Join/overview.md new file mode 100644 index 0000000..b86cae6 --- /dev/null +++ b/src/Lucene.Net.Join/overview.md @@ -0,0 +1,18 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Lucene's join module \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.Join/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.Join/package.md b/src/Lucene.Net.Join/package.md new file mode 100644 index 0000000..c21b566 --- /dev/null +++ b/src/Lucene.Net.Join/package.md @@ -0,0 +1,58 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +This modules support index-time and query-time joins. + +## Index-time joins + +The index-time joining support joins while searching, where joined documents are indexed as a single document block using [](xref:Lucene.Net.Index.IndexWriter.AddDocuments IndexWriter.AddDocuments()). This is useful for any normalized content (XML documents or database tables). In database terms, all rows for all joined tables matching a single row of the primary table must be indexed as a single document block, with the parent document being last in the group. + +When you index in this way, the documents in your index are divided into parent documents (the last document of each block) and child documents (all others). You provide a [](xref:Lucene.Net.Search.Filter) that identifies the parent documents, as Lucene does not currently record any information about doc blocks. + +At search time, use [](xref:Lucene.Net.Search.Join.ToParentBlockJoinQuery) to remap/join matches from any child [](xref:Lucene.Net.Search.Query) (ie, a query that matches only child documents) up to the parent document space. The resulting query can then be used as a clause in any query that matches parent. + +If you only care about the parent documents matching the query, you can use any collector to collect the parent hits, but if you'd also like to see which child documents match for each parent document, use the [](xref:Lucene.Net.Search.Join.ToParentBlockJoinCollector) to collect the hits. Once the search is done, you retrieve a [](xref:Lucene.Net.Search.Grouping.TopGroups) instance from the [](xref:Lucene.Net.Search.Join.ToParentBlockJoinCollector.GetTopGroups ToParentBlockJoinCollector.GetTopGroups()) method. + +To map/join in the opposite direction, use [](xref:Lucene.Net.Search.Join.ToChildBlockJoinQuery). This wraps + any query matching parent documents, creating the joined query + matching only child documents. + +## Query-time joins + + The query time joining is index term based and implemented as two pass search. The first pass collects all the terms from a fromField that match the fromQuery. The second pass returns all documents that have matching terms in a toField to the terms collected in the first pass. + +Query time joining has the following input: + +* `fromField`: The from field to join from. + `fromQuery`: The query executed to collect the from terms. This is usually the user specified query. + `multipleValuesPerDocument`: Whether the fromField contains more than one value per document + `scoreMode`: Defines how scores are translated to the other join side. If you don't care about scoring + use [](xref:Lucene.Net.Search.Join.ScoreMode.None) mode. This will disable scoring and is therefore more + efficient (requires less memory and is faster). + `toField`: The to field to join to + + Basically the query-time joining is accessible from one static method. The user of this method supplies the method with the described input and a `IndexSearcher` where the from terms need to be collected from. The returned query can be executed with the same `IndexSearcher`, but also with another `IndexSearcher`. Example usage of the [](xref:Lucene.Net.Search.Join.JoinUtil.CreateJoinQuery(String, boolean, String, Lucene.Net.Search.Query, Lucene.Net.Search.IndexSearcher, Lucene.Net.Search.Join.ScoreMode) JoinUtil.CreateJoinQuery()) : + + String fromField = "from"; // Name of the from field + boolean multipleValuesPerDocument = false; // Set only yo true in the case when your fromField has multiple values per document in your index + String toField = "to"; // Name of the to field + ScoreMode scoreMode = ScoreMode.Max // Defines how the scores are translated into the other side of the join. + Query fromQuery = new TermQuery(new Term("content", searchTerm)); // Query executed to collect from values to join to the to values + + Query joinQuery = JoinUtil.createJoinQuery(fromField, multipleValuesPerDocument, toField, fromQuery, fromSearcher, scoreMode); + TopDocs topDocs = toSearcher.search(joinQuery, 10); // Note: toSearcher can be the same as the fromSearcher + // Render topDocs... \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.Memory/overview.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.Memory/overview.md b/src/Lucene.Net.Memory/overview.md new file mode 100644 index 0000000..8c82483 --- /dev/null +++ b/src/Lucene.Net.Memory/overview.md @@ -0,0 +1,22 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + + <title> + memory + </title> + + memory \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.Memory/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.Memory/package.md b/src/Lucene.Net.Memory/package.md new file mode 100644 index 0000000..f0b262c --- /dev/null +++ b/src/Lucene.Net.Memory/package.md @@ -0,0 +1,22 @@ + +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<HTML> +<BODY> +High-performance single-document main memory Apache Lucene fulltext search index. +</BODY> +</HTML> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.Misc/Document/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.Misc/Document/package.md b/src/Lucene.Net.Misc/Document/package.md new file mode 100644 index 0000000..f741951 --- /dev/null +++ b/src/Lucene.Net.Misc/Document/package.md @@ -0,0 +1,18 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Misc extensions of the Document/Field API. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.Misc/Index/Sorter/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.Misc/Index/Sorter/package.md b/src/Lucene.Net.Misc/Index/Sorter/package.md new file mode 100644 index 0000000..b489a6e --- /dev/null +++ b/src/Lucene.Net.Misc/Index/Sorter/package.md @@ -0,0 +1,34 @@ + +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Provides index sorting capablities. The application can use any +Sort specification, e.g. to sort by fields using DocValues or FieldCache, or to +reverse the order of the documents (by using SortField.Type.DOC in reverse). +Multi-level sorts can be specified the same way you would when searching, by +building Sort from multiple SortFields. + +[](xref:Lucene.Net.Index.Sorter.SortingMergePolicy) can be used to +make Lucene sort segments before merging them. This will ensure that every +segment resulting from a merge will be sorted according to the provided +[](xref:Lucene.Net.Search.Sort). This however makes merging and +thus indexing slower. + +Sorted segments allow for early query termination when the sort order +matches index order. This makes query execution faster since not all documents +need to be visited. Please note that this is an expert feature and should not +be used without a deep understanding of Lucene merging and document collection. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.Misc/Index/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.Misc/Index/package.md b/src/Lucene.Net.Misc/Index/package.md new file mode 100644 index 0000000..c13a47e --- /dev/null +++ b/src/Lucene.Net.Misc/Index/package.md @@ -0,0 +1,18 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Misc index tools and index support. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.Misc/Misc/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.Misc/Misc/package.md b/src/Lucene.Net.Misc/Misc/package.md new file mode 100644 index 0000000..b5ef487 --- /dev/null +++ b/src/Lucene.Net.Misc/Misc/package.md @@ -0,0 +1,19 @@ + +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Miscellaneous index tools. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.Misc/Store/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.Misc/Store/package.md b/src/Lucene.Net.Misc/Store/package.md new file mode 100644 index 0000000..e11f11c --- /dev/null +++ b/src/Lucene.Net.Misc/Store/package.md @@ -0,0 +1,18 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Misc Directory implementations. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.Misc/Util/Fst/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.Misc/Util/Fst/package.md b/src/Lucene.Net.Misc/Util/Fst/package.md new file mode 100644 index 0000000..465bdc1 --- /dev/null +++ b/src/Lucene.Net.Misc/Util/Fst/package.md @@ -0,0 +1,18 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Misc FST classes. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.Misc/overview.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.Misc/overview.md b/src/Lucene.Net.Misc/overview.md new file mode 100644 index 0000000..c47d765 --- /dev/null +++ b/src/Lucene.Net.Misc/overview.md @@ -0,0 +1,56 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + + <title> + miscellaneous + </title> + +## Misc Tools + +The misc package has various tools for splitting/merging indices, +changing norms, finding high freq terms, and others. + +## NativeUnixDirectory + +**NOTE**: This uses C++ sources (accessible via JNI), which you'll +have to compile on your platform. + +[](xref:Lucene.Net.Store.NativeUnixDirectory) is a Directory implementation that bypasses the +OS's buffer cache (using direct IO) for any IndexInput and IndexOutput +used during merging of segments larger than a specified size (default +10 MB). This avoids evicting hot pages that are still in-use for +searching, keeping search more responsive while large merges run. + +See [this blog post](http://blog.mikemccandless.com/2010/06/lucene-and-fadvisemadvise.html) +for details. + +Steps to build: + +* <tt>cd lucene/misc/</tt> + + To compile NativePosixUtil.cpp -> libNativePosixUtil.so, run<tt> ant build-native-unix</tt>. + + <tt>libNativePosixUtil.so</tt> will be located in the <tt>lucene/build/native/</tt> folder + + Make sure libNativePosixUtil.so is on your LD_LIBRARY_PATH so java can find it (something like <tt>export LD_LIBRARY_PATH=/path/to/dir:$LD_LIBRARY_PATH</tt>, where /path/to/dir contains libNativePosixUtil.so) + + <tt>ant jar</tt> to compile the java source and put that JAR on your CLASSPATH + +NativePosixUtil.cpp/java also expose access to the posix_madvise, +madvise, posix_fadvise functions, which are somewhat more cross +platform than O_DIRECT, however, in testing (see above link), these +APIs did not seem to help prevent buffer cache eviction. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.Queries/Function/DocValues/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.Queries/Function/DocValues/package.md b/src/Lucene.Net.Queries/Function/DocValues/package.md new file mode 100644 index 0000000..5fade84 --- /dev/null +++ b/src/Lucene.Net.Queries/Function/DocValues/package.md @@ -0,0 +1,18 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +FunctionValues for different data types. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.Queries/Function/ValueSources/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.Queries/Function/ValueSources/package.md b/src/Lucene.Net.Queries/Function/ValueSources/package.md new file mode 100644 index 0000000..3527dc7 --- /dev/null +++ b/src/Lucene.Net.Queries/Function/ValueSources/package.md @@ -0,0 +1,18 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +A variety of functions to use with FunctionQuery. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.Queries/Function/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.Queries/Function/package.md b/src/Lucene.Net.Queries/Function/package.md new file mode 100644 index 0000000..64dd300 --- /dev/null +++ b/src/Lucene.Net.Queries/Function/package.md @@ -0,0 +1,18 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Queries that compute score based upon a function \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.Queries/Mlt/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.Queries/Mlt/package.md b/src/Lucene.Net.Queries/Mlt/package.md new file mode 100644 index 0000000..46a2f33 --- /dev/null +++ b/src/Lucene.Net.Queries/Mlt/package.md @@ -0,0 +1,19 @@ + +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Document similarity query generators. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.Queries/overview.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.Queries/overview.md b/src/Lucene.Net.Queries/overview.md new file mode 100644 index 0000000..ba3f288 --- /dev/null +++ b/src/Lucene.Net.Queries/overview.md @@ -0,0 +1,22 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + --> + + <title> + Queries + </title> + + Queries \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.Queries/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.Queries/package.md b/src/Lucene.Net.Queries/package.md new file mode 100644 index 0000000..f24268f --- /dev/null +++ b/src/Lucene.Net.Queries/package.md @@ -0,0 +1,18 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Filters and Queries that add to core Lucene. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.QueryParser/Analyzing/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.QueryParser/Analyzing/package.md b/src/Lucene.Net.QueryParser/Analyzing/package.md new file mode 100644 index 0000000..390b079 --- /dev/null +++ b/src/Lucene.Net.QueryParser/Analyzing/package.md @@ -0,0 +1,19 @@ + +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +QueryParser that passes Fuzzy-, Prefix-, Range-, and WildcardQuerys through the given analyzer. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.QueryParser/Classic/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.QueryParser/Classic/package.md b/src/Lucene.Net.QueryParser/Classic/package.md new file mode 100644 index 0000000..ee90202 --- /dev/null +++ b/src/Lucene.Net.QueryParser/Classic/package.md @@ -0,0 +1,313 @@ + +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +A simple query parser implemented with JavaCC. + +Note that JavaCC defines lots of public classes, methods and fields +that do not need to be public. These clutter the documentation. +Sorry. + +Note that because JavaCC defines a class named <tt>Token</tt>, <tt>org.apache.lucene.analysis.Token</tt> +must always be fully qualified in source code in this package. + +**NOTE**: [](xref:Lucene.Net.QueryParsers.Flexible.Standard) has an alternative queryparser that matches the syntax of this one, but is more modular, +enabling substantial customization to how a query is created. + +## Query Parser Syntax + +<div id="minitoc-area"> + +* [Overview](#Overview) + +* [Terms](#Terms) + +* [Fields](#Fields) + +* [Term Modifiers](#Term_Modifiers) + + * [Wildcard Searches](#Wildcard_Searches) + + * [Regular expression Searches](#Regexp_Searches) + + * [Fuzzy Searches](#Fuzzy_Searches) + + * [Proximity Searches](#Proximity_Searches) + + * [Range Searches](#Range_Searches) + + * [Boosting a Term](#Boosting_a_Term) + +* [Boolean Operators](#Boolean_operators) + + * [OR](#OR) + + * [AND](#AND) + + * [+](#+) + + * [NOT](#NOT) + + * [-](#-) + +* [Grouping](#Grouping) + +* [Field Grouping](#Field_Grouping) + +* [Escaping Special Characters](#Escaping_Special_Characters) + +</div> + +## Overview + +<div class="section"> + +Although Lucene provides the ability to create your own queries through its API, it also provides a rich query language through the Query Parser, a lexer which interprets a string into a Lucene Query using JavaCC. + +Generally, the query parser syntax may change from release to release. This page describes the syntax as of the current release. If you are using a different version of Lucene, please consult the copy of <span class="codefrag">docs/queryparsersyntax.html</span> that was distributed with the version you are using. + + Before choosing to use the provided Query Parser, please consider the following: 1. If you are programmatically generating a query string and then parsing it with the query parser then you should seriously consider building your queries directly with the query API. In other words, the query parser is designed for human-entered text, not for program-generated text. 2. Untokenized fields are best added directly to queries, and not through the query parser. If a field's values are generated programmatically by the application, then so should query clauses for this field. An analyzer, which the query parser uses, is designed to convert human-entered text to terms. Program-generated values, like dates, keywords, etc., should be consistently program-generated. 3. In a query form, fields which are general text should use the query parser. All others, such as date ranges, keywords, etc. are better added directly through the query API. A field with a limit set of values, that can be specifi ed with a pull-down menu should not be added to a query string which is subsequently parsed, but rather added as a TermQuery clause. + +</div> + +## Terms + +<div class="section"> + +A query is broken up into terms and operators. There are two types of terms: Single Terms and Phrases. + +A Single Term is a single word such as "test" or "hello". + +A Phrase is a group of words surrounded by double quotes such as "hello dolly". + +Multiple terms can be combined together with Boolean operators to form a more complex query (see below). + +Note: The analyzer used to create the index will be used on the terms and phrases in the query string. So it is important to choose an analyzer that will not interfere with the terms used in the query string. + +</div> + +## Fields + +<div class="section"> + +Lucene supports fielded data. When performing a search you can either specify a field, or use the default field. The field names and default field is implementation specific. + +You can search any field by typing the field name followed by a colon ":" and then the term you are looking for. + +As an example, let's assume a Lucene index contains two fields, title and text and text is the default field. If you want to find the document entitled "The Right Way" which contains the text "don't go this way", you can enter: + +title:"The Right Way" AND text:go + +or + +title:"The Right Way" AND go + +Since text is the default field, the field indicator is not required. + +Note: The field is only valid for the term that it directly precedes, so the query + +title:The Right Way + +Will only find "The" in the title field. It will find "Right" and "Way" in the default field (in this case the text field). + +</div> + +## Term Modifiers + +<div class="section"> + +Lucene supports modifying query terms to provide a wide range of searching options. + +### Wildcard Searches + +Lucene supports single and multiple character wildcard searches within single terms (not within phrase queries). + +To perform a single character wildcard search use the "?" symbol. + +To perform a multiple character wildcard search use the "*" symbol. + +The single character wildcard search looks for terms that match that with the single character replaced. For example, to search for "text" or "test" you can use the search: + +te?t + +Multiple character wildcard searches looks for 0 or more characters. For example, to search for test, tests or tester, you can use the search: + +test* + +You can also use the wildcard searches in the middle of a term. + +te*t + +Note: You cannot use a * or ? symbol as the first character of a search. + +### Regular Expression Searches + +Lucene supports regular expression searches matching a pattern between forward slashes "/". The syntax may change across releases, but the current supported syntax is documented in the [](xref:Lucene.Net.Util.Automaton.RegExp RegExp) class. For example to find documents containing "moat" or "boat": + +/[mb]oat/ + +### Fuzzy Searches + +Lucene supports fuzzy searches based on Damerau-Levenshtein Distance. To do a fuzzy search use the tilde, "~", symbol at the end of a Single word Term. For example to search for a term similar in spelling to "roam" use the fuzzy search: + +roam~ + +This search will find terms like foam and roams. + +An additional (optional) parameter can specify the maximum number of edits allowed. The value is between 0 and 2, For example: + +roam~1 + +The default that is used if the parameter is not given is 2 edit distances. + +Previously, a floating point value was allowed here. This syntax is considered deprecated and will be removed in Lucene 5.0 + +### Proximity Searches + +Lucene supports finding words are a within a specific distance away. To do a proximity search use the tilde, "~", symbol at the end of a Phrase. For example to search for a "apache" and "jakarta" within 10 words of each other in a document use the search: + +"jakarta apache"~10 + +### Range Searches + +Range Queries allow one to match documents whose field(s) values are between the lower and upper bound specified by the Range Query. Range Queries can be inclusive or exclusive of the upper and lower bounds. Sorting is done lexicographically. + +mod_date:[20020101 TO 20030101] + +This will find documents whose mod_date fields have values between 20020101 and 20030101, inclusive. Note that Range Queries are not reserved for date fields. You could also use range queries with non-date fields: + +title:{Aida TO Carmen} + +This will find all documents whose titles are between Aida and Carmen, but not including Aida and Carmen. + +Inclusive range queries are denoted by square brackets. Exclusive range queries are denoted by curly brackets. + +### Boosting a Term + +Lucene provides the relevance level of matching documents based on the terms found. To boost a term use the caret, "^", symbol with a boost factor (a number) at the end of the term you are searching. The higher the boost factor, the more relevant the term will be. + +Boosting allows you to control the relevance of a document by boosting its term. For example, if you are searching for + +jakarta apache + +and you want the term "jakarta" to be more relevant boost it using the ^ symbol along with the boost factor next to the term. You would type: + +jakarta^4 apache + +This will make documents with the term jakarta appear more relevant. You can also boost Phrase Terms as in the example: + +"jakarta apache"^4 "Apache Lucene" + +By default, the boost factor is 1. Although the boost factor must be positive, it can be less than 1 (e.g. 0.2) + +</div> + +## Boolean Operators + +<div class="section"> + +Boolean operators allow terms to be combined through logic operators. Lucene supports AND, "+", OR, NOT and "-" as Boolean operators(Note: Boolean operators must be ALL CAPS). + +### OR + +The OR operator is the default conjunction operator. This means that if there is no Boolean operator between two terms, the OR operator is used. The OR operator links two terms and finds a matching document if either of the terms exist in a document. This is equivalent to a union using sets. The symbol || can be used in place of the word OR. + +To search for documents that contain either "jakarta apache" or just "jakarta" use the query: + +"jakarta apache" jakarta + +or + +"jakarta apache" OR jakarta + +### AND + +The AND operator matches documents where both terms exist anywhere in the text of a single document. This is equivalent to an intersection using sets. The symbol && can be used in place of the word AND. + +To search for documents that contain "jakarta apache" and "Apache Lucene" use the query: + +"jakarta apache" AND "Apache Lucene" + +### + + +The "+" or required operator requires that the term after the "+" symbol exist somewhere in a the field of a single document. + +To search for documents that must contain "jakarta" and may contain "lucene" use the query: + ++jakarta lucene + +### NOT + +The NOT operator excludes documents that contain the term after NOT. This is equivalent to a difference using sets. The symbol ! can be used in place of the word NOT. + +To search for documents that contain "jakarta apache" but not "Apache Lucene" use the query: + +"jakarta apache" NOT "Apache Lucene" + +Note: The NOT operator cannot be used with just one term. For example, the following search will return no results: + +NOT "jakarta apache" + +### - + +The "-" or prohibit operator excludes documents that contain the term after the "-" symbol. + +To search for documents that contain "jakarta apache" but not "Apache Lucene" use the query: + +"jakarta apache" -"Apache Lucene" + +</div> + +## Grouping + +<div class="section"> + +Lucene supports using parentheses to group clauses to form sub queries. This can be very useful if you want to control the boolean logic for a query. + +To search for either "jakarta" or "apache" and "website" use the query: + +(jakarta OR apache) AND website + +This eliminates any confusion and makes sure you that website must exist and either term jakarta or apache may exist. + +</div> + +## Field Grouping + +<div class="section"> + +Lucene supports using parentheses to group multiple clauses to a single field. + +To search for a title that contains both the word "return" and the phrase "pink panther" use the query: + +title:(+return +"pink panther") + +</div> + +## Escaping Special Characters + +<div class="section"> + +Lucene supports escaping special characters that are part of the query syntax. The current list special characters are + ++ - && || ! ( ) { } [ ] ^ " ~ * ? : \ / + +To escape these character use the \ before the character. For example to search for (1+1):2 use the query: + +\(1\+1\)\:2 + +</div> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.QueryParser/ComplexPhrase/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.QueryParser/ComplexPhrase/package.md b/src/Lucene.Net.QueryParser/ComplexPhrase/package.md new file mode 100644 index 0000000..9944cf2 --- /dev/null +++ b/src/Lucene.Net.QueryParser/ComplexPhrase/package.md @@ -0,0 +1,19 @@ + +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +QueryParser which permits complex phrase query syntax eg "(john jon jonathan~) peters*" \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.QueryParser/Ext/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.QueryParser/Ext/package.md b/src/Lucene.Net.QueryParser/Ext/package.md new file mode 100644 index 0000000..9b602b7 --- /dev/null +++ b/src/Lucene.Net.QueryParser/Ext/package.md @@ -0,0 +1,19 @@ + +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Extendable QueryParser provides a simple and flexible extension mechanism by overloading query field names. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.QueryParser/Flexible/Core/Builders/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.QueryParser/Flexible/Core/Builders/package.md b/src/Lucene.Net.QueryParser/Flexible/Core/Builders/package.md new file mode 100644 index 0000000..e61d138 --- /dev/null +++ b/src/Lucene.Net.QueryParser/Flexible/Core/Builders/package.md @@ -0,0 +1,23 @@ + +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Necessary classes to implement query builders. + +## Query Parser Builders + + The package <tt>org.apache.lucene.queryParser.builders</tt> contains the interface that builders must implement, it also contain a utility [](xref:Lucene.Net.QueryParsers.Flexible.Core.Builders.QueryTreeBuilder), which walks the tree and call the Builder for each node in the tree. Builder normally convert QueryNode Object into a Lucene Query Object, and normally it's a one-to-one mapping class. But other builders implementations can by written to convert QueryNode objects to other non lucene objects. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.QueryParser/Flexible/Core/Config/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.QueryParser/Flexible/Core/Config/package.md b/src/Lucene.Net.QueryParser/Flexible/Core/Config/package.md new file mode 100644 index 0000000..09b1859 --- /dev/null +++ b/src/Lucene.Net.QueryParser/Flexible/Core/Config/package.md @@ -0,0 +1,27 @@ + +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Base classes used to configure the query processing. + +## Query Configuration Interfaces + + The package <tt>org.apache.lucene.queryparser.flexible.config</tt> contains query configuration handler abstract class that all config handlers should extend. + + See [](xref:Lucene.Net.QueryParsers.Flexible.Standard.Config.StandardQueryConfigHandler) for a reference implementation. + + The [](xref:Lucene.Net.QueryParsers.Flexible.Core.Config.QueryConfigHandler) and [](xref:Lucene.Net.QueryParsers.Flexible.Core.Config.FieldConfig) are used in the processors to access config information in a flexible and independent way. See [](xref:Lucene.Net.QueryParsers.Flexible.Standard.Processors.TermRangeQueryNodeProcessor) for a reference implementation. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.QueryParser/Flexible/Core/Messages/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.QueryParser/Flexible/Core/Messages/package.md b/src/Lucene.Net.QueryParser/Flexible/Core/Messages/package.md new file mode 100644 index 0000000..baad13b --- /dev/null +++ b/src/Lucene.Net.QueryParser/Flexible/Core/Messages/package.md @@ -0,0 +1,23 @@ + +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Messages usually used by query parser implementations. + +## Query Parser Messages + +Messages for the Flexible Query Parser, they use <tt>org.apache.lucene.messages.NLS</tt> API. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.QueryParser/Flexible/Core/Nodes/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.QueryParser/Flexible/Core/Nodes/package.md b/src/Lucene.Net.QueryParser/Flexible/Core/Nodes/package.md new file mode 100644 index 0000000..ba4712e --- /dev/null +++ b/src/Lucene.Net.QueryParser/Flexible/Core/Nodes/package.md @@ -0,0 +1,37 @@ + +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Query nodes commonly used by query parser implementations. + +## Query Nodes + + The package <tt>org.apache.lucene.queryParser.nodes</tt> contains all the basic query nodes. The interface that represents a query node is [](xref:Lucene.Net.QueryParsers.Flexible.Core.Nodes.QueryNode). + + [](xref:Lucene.Net.QueryParsers.Flexible.Core.Nodes.QueryNode)s are used by the text parser to create a syntax tree. These nodes are designed to be used by UI or other text parsers. The default Lucene text parser is [](xref:Lucene.Net.QueryParsers.Flexible.Standard.Parser.StandardSyntaxParser), it implements Lucene's standard syntax. + + [](xref:Lucene.Net.QueryParsers.Flexible.Core.Nodes.QueryNode) interface should be implemented by all query nodes, the class [](xref:Lucene.Net.QueryParsers.Flexible.Core.Nodes.QueryNodeImpl) implements [](xref:Lucene.Net.QueryParsers.Flexible.Core.Nodes.QueryNode) and is extended by all current query node implementations. + + A query node tree can be printed to the a stream, and it generates a pseudo XML representation with all the nodes. + + A query node tree can also generate a query string that can be parsed back by the original text parser, at this point only the standard lucene syntax is supported. + + Grouping nodes: * AndQueryNode - used for AND operator * AnyQueryNode - used for ANY operator * OrQueryNode - used for OR operator * BooleanQueryNode - used when no operator is specified * ModifierQueryNode - used for modifier operator * GroupQueryNode - used for parenthesis * BoostQueryNode - used for boost operator * SlopQueryNode - phrase slop * FuzzyQueryNode - fuzzy node * TermRangeQueryNode - used for parametric field:[low_value TO high_value] * ProximityQueryNode - used for proximity search * NumericRangeQueryNode - used for numeric range search * TokenizedPhraseQueryNode - used by tokenizers/lemmatizers/analyzers for phrases/autophrases + + Leaf Nodes: * FieldQueryNode - field/value node * NumericQueryNode - used for numeric search * PathQueryNode - [](xref:Lucene.Net.QueryParsers.Flexible.Core.Nodes.QueryNode) object used with path-like queries * OpaqueQueryNode - Used as for part of the query that can be parsed by other parsers. schema/value * PrefixWildcardQueryNode - non-phrase wildcard query * QuotedFieldQUeryNode - regular phrase node * WildcardQueryNode - non-phrase wildcard query + + Utility Nodes: * DeletedQueryNode - used by processors on optimizations * MatchAllDocsQueryNode - used by processors on optimizations * MatchNoDocsQueryNode - used by processors on optimizations * NoTokenFoundQueryNode - used by tokenizers/lemmatizers/analyzers \ No newline at end of file http://git-wip-us.apache.org/repos/asf/lucenenet/blob/6a95ad43/src/Lucene.Net.QueryParser/Flexible/Core/Parser/package.md ---------------------------------------------------------------------- diff --git a/src/Lucene.Net.QueryParser/Flexible/Core/Parser/package.md b/src/Lucene.Net.QueryParser/Flexible/Core/Parser/package.md new file mode 100644 index 0000000..220a8b9 --- /dev/null +++ b/src/Lucene.Net.QueryParser/Flexible/Core/Parser/package.md @@ -0,0 +1,23 @@ + +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Necessary interfaces to implement text parsers. + +## Parser + + The package <tt>org.apache.lucene.queryparser.flexible.parser</tt> contains interfaces that should be implemented by the parsers. Parsers produce QueryNode Trees from a string object. These package still needs some work to add support to for multiple parsers. Features that should be supported for the future, related with the parser: - QueryNode tree should be able convertible to any parser syntax. - The query syntax should support calling other parsers. - QueryNode tree created by multiple parsers. \ No newline at end of file
