Chad has submitted this change and it was merged. Change subject: More documentation ......................................................................
More documentation Change-Id: I87b4aed8dee4ba71c7d696ce1a8ce1dfa97ed7a0 --- M README.md A docs/how_does_it_work.md 2 files changed, 73 insertions(+), 0 deletions(-) Approvals: Chad: Looks good to me, approved jenkins-bot: Verified diff --git a/README.md b/README.md index 076f284..a5d5244 100644 --- a/README.md +++ b/README.md @@ -11,6 +11,8 @@ * Lucene: A jar containing a bridge between the core and lucene * Elasticsearch: An Elasticsearch plugin +You can read more on how it works [here](docs/how_it_works.md). + Elasticsearch value proposition ------------------------------- diff --git a/docs/how_does_it_work.md b/docs/how_does_it_work.md new file mode 100644 index 0000000..15e7cb8 --- /dev/null +++ b/docs/how_does_it_work.md @@ -0,0 +1,71 @@ +How does it work? +================= + +The highlighter has a few key types of components: <dl> +<dt>HitEnum</dt><dd>Enumerates hits (aka matches)</dd> +<dt>SnippetChooser</dt><dd>Uses HitEnum to pick snippets</dd> +<dt>Segmenter</dt><dd>Plugged into AbstractBasicSnippetChooser implementations +to pick where snippets (aka fragments) begin and end.</dt> +</dl> + + +HitEnum +------- +Comes in two flavors:<dl> +<dt>Plain HitEnums</dt> +<dd>Actually pulls hits from the source document. Examples are + DocsAndPositionsHitEnum, TokenStreamHitEnum, and BreakIteratorHitEnum.</dd> +<dt>Transforming HitEnums</dt> +<dd>Wraps and transforms one or more HitEnums. Some transforms are simple like + WeightFilterHitEnumWrapper or PositionBoostingHitEnumWrapper. Some are + much more involved like PhraseHitEnumWrapper and + MergingHitEnum. HitEnums that wrap a single HitEnum should be named + FooHitEnumWrapper. I don't have a consisten naming scheme for those that + wrap more then one HitEnum.</dd> +</dl> + + +SnippetChooser +-------------- +There are two implementations:<dl> +<dt>BasicScoreBasedSnippetChooser</dt> +<dd>Score ordered and score cutoff snippets. Keeps snippets in a priority + queue then sorts them in document order to pick bounds so there is no + overlap then optionally sorts them in score order. Worst case performance + is ```O(n*log(m) + m*log(m))``` where n is number of snippets found and m + is number of snippets requested. The first term is scanning all the + segments and the second is the sorts. You can put an upper bound on n by + setting ```maxSnippetsChecked``` which is piped through to Elasticsearch as + ```max_fragments_scored```.</dd> +<dt>BasicSourceOrderSnippetChooser</dt> +<dd>Source order fragments. Can be much faster because it can exit after + hitting the first snippet.</dd> +</dl> + + +Segmenter +--------- +Four major implementations:<dl> +<dt>CharScanningSegmenter</dt> +<dd>FastVectorHighlighter like character scanning. Usually the fastest choice + for large text.</dd> +<dt>BreakIteratorSegmenter</dt> +<dd>PostingHighlighter like sentence breaks. Slower but sometimes prettier. + Suffers if text isn't 100% prose and/or some sentences are hugely + long.</dd> +<dt>WholeSourceSegmenter</dt> +<dd>Doesn't break the source at all.</dd> +<dt>MultiSegmenter</dt> +<dd>Wraps many segmenters adding hard stops between them. The life blood of + multi valued fields.</dd> +</dl> + +The Elasticsearch plugin adds a DelayedSegmenter which constructs one of the +first three segmenters lazily to prevent loading the field until the first hit +is found. + + +Others +--------------- +SourceExtracters are responsible for extracting the snippets from the source +once they are identified. -- To view, visit https://gerrit.wikimedia.org/r/131702 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I87b4aed8dee4ba71c7d696ce1a8ce1dfa97ed7a0 Gerrit-PatchSet: 1 Gerrit-Project: search/highlighter Gerrit-Branch: master Gerrit-Owner: Manybubbles <[email protected]> Gerrit-Reviewer: Chad <[email protected]> Gerrit-Reviewer: jenkins-bot <> _______________________________________________ MediaWiki-commits mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
