Modified:
websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/Cookbook/FastUpdates.html
==============================================================================
---
websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/Cookbook/FastUpdates.html
(original)
+++
websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/Cookbook/FastUpdates.html
Fri Feb 26 12:52:42 2016
@@ -71,24 +71,24 @@
<div>
<a name='___top' class='dummyTopAnchor' ></a>
-<h1><a class='u' href='#___top' title='click to go to top of document'
+<h2><a class='u'
name="NAME"
->NAME</a></h1>
+>NAME</a></h2>
-<p>Lucy::Docs::Cookbook::FastUpdates - Near real-time index updates.</p>
+<p>Lucy::Docs::Cookbook::FastUpdates - Near real-time index updates</p>
-<h1><a class='u' href='#___top' title='click to go to top of document'
-name="ABSTRACT"
->ABSTRACT</a></h1>
+<h2><a class='u'
+name="DESCRIPTION"
+>DESCRIPTION</a></h2>
<p>While index updates are fast on average,
worst-case update performance may be significantly slower.
To make index updates consistently quick,
we must manually intervene to control the process of index segment
consolidation.</p>
-<h1><a class='u' href='#___top' title='click to go to top of document'
+<h3><a class='u'
name="The_problem"
->The problem</a></h1>
+>The problem</a></h3>
<p>Ordinarily,
modifying an index is cheap.
@@ -96,7 +96,7 @@ New data is added to new segments,
and the time to write a new segment scales more or less linearly with the
number of documents added during the indexing session.</p>
<p>Deletions are also cheap most of the time,
-because we don't remove documents immediately but instead mark them as
deleted,
+because we don’t remove documents immediately but instead mark them as
deleted,
and adding the deletion mark is cheap.</p>
<p>However,
@@ -112,15 +112,16 @@ Every once in a while,
though,
a large amount of data must be rewritten.</p>
-<h1><a class='u' href='#___top' title='click to go to top of document'
+<h3><a class='u'
name="Procrastinating_and_playing_catch-up"
->Procrastinating and playing catch-up</a></h1>
+>Procrastinating and playing catch-up</a></h3>
<p>The simplest way to force fast index updates is to avoid rewriting
anything.</p>
<p>Indexer relies upon <a href="../../../Lucy/Index/IndexManager.html"
class="podlinkpod"
->IndexManager</a>'s recycle() method to tell it which segments should be
consolidated.
-If we subclass IndexManager and override recycle() so that it always returns
an empty array,
+>IndexManager</a>’s <a
href="../../../Lucy/Index/IndexManager.html#recycle" class="podlinkpod"
+>recycle()</a> method to tell it which segments should be consolidated.
+If we subclass IndexManager and override the method so that it always returns
an empty array,
we get consistently quick performance:</p>
<pre> package NoMergeManager;
@@ -135,7 +136,7 @@ we get consistently quick performance:</
...
$indexer->commit;</pre>
-<p>However, we can't procrastinate forever. Eventually, we'll have to
run an ordinary, uncontrolled indexing session, potentially triggering a large
rewrite of lots of small and/or degraded segments:</p>
+<p>However, we can’t procrastinate forever. Eventually, we’ll have
to run an ordinary, uncontrolled indexing session, potentially triggering a
large rewrite of lots of small and/or degraded segments:</p>
<pre> my $indexer = Lucy::Index::Indexer->new(
index => '/path/to/index',
@@ -144,9 +145,9 @@ we get consistently quick performance:</
...
$indexer->commit;</pre>
-<h1><a class='u' href='#___top' title='click to go to top of document'
+<h3><a class='u'
name="Acceptable_worst-case_update_time,_slower_degradation"
->Acceptable worst-case update time, slower degradation</a></h1>
+>Acceptable worst-case update time, slower degradation</a></h3>
<p>Never merging anything at all in the main indexing process is probably
overkill. Small segments are relatively cheap to merge; we just need to guard
against the big rewrites.</p>
@@ -164,13 +165,14 @@ name="Acceptable_worst-case_update_time,
<p>However, we still have to consolidate every once in a while, and while that
happens content updates will be locked out.</p>
-<h1><a class='u' href='#___top' title='click to go to top of document'
+<h3><a class='u'
name="Background_merging"
->Background merging</a></h1>
+>Background merging</a></h3>
-<p>If it's not acceptable to lock out updates while the index
consolidation process runs, the alternative is to move the consolidation
process out of band, using Lucy::Index::BackgroundMerger.</p>
+<p>If it’s not acceptable to lock out updates while the index
consolidation process runs, the alternative is to move the consolidation
process out of band, using <a href="../../../Lucy/Index/BackgroundMerger.html"
class="podlinkpod"
+>BackgroundMerger</a>.</p>
-<p>It's never safe to have more than one Indexer attempting to modify the
content of an index at the same time, but a BackgroundMerger and an Indexer can
operate simultaneously:</p>
+<p>It’s never safe to have more than one Indexer attempting to modify
the content of an index at the same time, but a BackgroundMerger and an Indexer
can operate simultaneously:</p>
<pre> # Indexing process.
use Scalar::Util qw( blessed );
@@ -204,7 +206,9 @@ name="Background_merging"
);
$bg_merger->commit;</pre>
-<p>The exception handling code becomes useful once you have more than one
index modification process happening simultaneously. By default, Indexer tries
several times to acquire a write lock over the span of one second, then holds
it until commit() completes. BackgroundMerger handles most of its work without
the write lock, but it does need it briefly once at the beginning and once
again near the end. Under normal loads, the internal retry logic will resolve
conflicts, but if it's not acceptable to miss an insert, you probably want
to catch LockErr exceptions thrown by Indexer. In contrast, a LockErr from
BackgroundMerger probably just needs to be logged.</p>
+<p>The exception handling code becomes useful once you have more than one
index modification process happening simultaneously. By default, Indexer tries
several times to acquire a write lock over the span of one second, then holds
it until <a href="../../../Lucy/Index/Indexer.html#commit" class="podlinkpod"
+>commit()</a> completes. BackgroundMerger handles most of its work without the
write lock, but it does need it briefly once at the beginning and once again
near the end. Under normal loads, the internal retry logic will resolve
conflicts, but if it’s not acceptable to miss an insert, you probably
want to catch <a href="../../../Lucy/Store/LockErr.html" class="podlinkpod"
+>LockErr</a> exceptions thrown by Indexer. In contrast, a LockErr from
BackgroundMerger probably just needs to be logged.</p>
</div>
Modified: websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/DevGuide.html
==============================================================================
--- websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/DevGuide.html
(original)
+++ websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/DevGuide.html Fri
Feb 26 12:52:42 2016
@@ -71,33 +71,47 @@
<div>
<a name='___top' class='dummyTopAnchor' ></a>
-<h1><a class='u' href='#___top' title='click to go to top of document'
+<h2><a class='u'
name="NAME"
->NAME</a></h1>
+>NAME</a></h2>
<p>Lucy::Docs::DevGuide - Quick-start guide to hacking on Apache Lucy.</p>
-<h1><a class='u' href='#___top' title='click to go to top of document'
+<h2><a class='u'
name="DESCRIPTION"
->DESCRIPTION</a></h1>
+>DESCRIPTION</a></h2>
<p>The Apache Lucy code base is organized into roughly four layers:</p>
-<pre> * Charmonizer - compiler and OS configuration probing.
- * Clownfish - header files.
- * C - implementation files.
- * Host - binding language.</pre>
+<ul>
+<li>Charmonizer - compiler and OS configuration probing.</li>
-<p>Charmonizer is a configuration prober which writes a single header file,
"charmony.h", describing the build environment and facilitating
cross-platform development. It's similar to Autoconf or Metaconfig, but
written in pure C.</p>
+<li>Clownfish - header files.</li>
-<p>The ".cfh" files within the Lucy core are Clownfish header files.
Clownfish is a purpose-built, declaration-only language which superimposes a
single-inheritance object model on top of C which is specifically designed to
co-exist happily with variety of "host" languages and to allow limited
run-time dynamic subclassing. For more information see the Clownfish docs, but
if there's one thing you should know about Clownfish OO before you start
hacking, it's that method calls are differentiated from functions by
capitalization:</p>
+<li>C - implementation files.</li>
+
+<li>Host - binding language.</li>
+</ul>
+
+<p>Charmonizer is a configuration prober which writes a single header file,
+“charmony.h”,
+describing the build environment and facilitating cross-platform development.
+It’s similar to Autoconf or Metaconfig,
+but written in pure C.</p>
+
+<p>The “.cfh” files within the Lucy core are Clownfish header
files.
+Clownfish is a purpose-built,
+declaration-only language which superimposes a single-inheritance object model
on top of C which is specifically designed to co-exist happily with variety of
“host” languages and to allow limited run-time dynamic subclassing.
+For more information see the Clownfish docs,
+but if there’s one thing you should know about Clownfish OO before you
start hacking,
+it’s that method calls are differentiated from functions by
capitalization:</p>
<pre> Indexer_Add_Doc <-- Method, typically uses dynamic dispatch.
Indexer_add_doc <-- Function, always a direct invocation.</pre>
-<p>The C files within the Lucy core are where most of Lucy's low-level
functionality lies. They implement the interface defined by the Clownfish
header files.</p>
+<p>The C files within the Lucy core are where most of Lucy’s low-level
functionality lies. They implement the interface defined by the Clownfish
header files.</p>
-<p>The C core is intentionally left incomplete, however; to be usable, it must
be bound to a "host" language. (In this context, even C is considered a
"host" which must implement the missing pieces and be "bound"
to the core.) Some of the binding code is autogenerated by Clownfish on a spec
customized for each language. Other pieces are hand-coded in either C (using
the host's C API) or the host language itself.</p>
+<p>The C core is intentionally left incomplete, however; to be usable, it must
be bound to a “host” language. (In this context, even C is
considered a “host” which must implement the missing pieces and be
“bound” to the core.) Some of the binding code is autogenerated by
Clownfish on a spec customized for each language. Other pieces are hand-coded
in either C (using the host’s C API) or the host language itself.</p>
</div>
Modified: websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/DocIDs.html
==============================================================================
--- websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/DocIDs.html
(original)
+++ websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/DocIDs.html Fri Feb
26 12:52:42 2016
@@ -71,19 +71,19 @@
<div>
<a name='___top' class='dummyTopAnchor' ></a>
-<h1><a class='u' href='#___top' title='click to go to top of document'
+<h2><a class='u'
name="NAME"
->NAME</a></h1>
+>NAME</a></h2>
<p>Lucy::Docs::DocIDs - Characteristics of Apache Lucy document ids.</p>
-<h1><a class='u' href='#___top' title='click to go to top of document'
+<h2><a class='u'
name="DESCRIPTION"
->DESCRIPTION</a></h1>
+>DESCRIPTION</a></h2>
-<h2><a class='u' href='#___top' title='click to go to top of document'
+<h3><a class='u'
name="Document_ids_are_signed_32-bit_integers"
->Document ids are signed 32-bit integers</a></h2>
+>Document ids are signed 32-bit integers</a></h3>
<p>Document ids in Apache Lucy start at 1.
Because 0 is never a valid doc id,
@@ -93,9 +93,9 @@ we can use it as a sentinel value:</p>
...
}</pre>
-<h2><a class='u' href='#___top' title='click to go to top of document'
+<h3><a class='u'
name="Document_ids_are_ephemeral"
->Document ids are ephemeral</a></h2>
+>Document ids are ephemeral</a></h3>
<p>The document ids used by Lucy are associated with a single index snapshot.
The moment an index is updated, the mapping of document ids to documents is
subject to change.</p>
Modified:
websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/FileFormat.html
==============================================================================
--- websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/FileFormat.html
(original)
+++ websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/FileFormat.html Fri
Feb 26 12:52:42 2016
@@ -71,15 +71,15 @@
<div>
<a name='___top' class='dummyTopAnchor' ></a>
-<h1><a class='u' href='#___top' title='click to go to top of document'
+<h2><a class='u'
name="NAME"
->NAME</a></h1>
+>NAME</a></h2>
-<p>Lucy::Docs::FileFormat - Overview of index file format.</p>
+<p>Lucy::Docs::FileFormat - Overview of index file format</p>
-<h1><a class='u' href='#___top' title='click to go to top of document'
-name="OVERVIEW"
->OVERVIEW</a></h1>
+<h2><a class='u'
+name="DESCRIPTION"
+>DESCRIPTION</a></h2>
<p>It is not necessary to understand the current implementation details of the
index file format in order to use Apache Lucy effectively,
but it may be helpful if you are interested in tweaking for high performance,
@@ -88,7 +88,7 @@ or debugging and development.</p>
<p>On a file system,
an index is a directory.
-The files inside have a hierarchical relationship: an index is made up of
"segments",
+The files inside have a hierarchical relationship: an index is made up of
“segments”,
each of which is an independent inverted index with its own subdirectory; each
segment is made up of several component parts.</p>
<pre> [index]--|
@@ -118,72 +118,72 @@ each of which is an independent inverted
|
|--[...]--| </pre>
-<h1><a class='u' href='#___top' title='click to go to top of document'
+<h3><a class='u'
name="Write-once_philosophy"
->Write-once philosophy</a></h1>
+>Write-once philosophy</a></h3>
-<p>All segment directory names consist of the string "seg_" followed
by a number in base 36: seg_1, seg_5m, seg_p9s2 and so on, with higher numbers
indicating more recent segments. Once a segment is finished and committed, its
name is never re-used and its files are never modified.</p>
+<p>All segment directory names consist of the string “seg_”
followed by a number in base 36: seg_1, seg_5m, seg_p9s2 and so on, with higher
numbers indicating more recent segments. Once a segment is finished and
committed, its name is never re-used and its files are never modified.</p>
<p>Old segments become obsolete and can be removed when their data has been
consolidated into new segments during the process of segment merging and
optimization. A fully-optimized index has only one segment.</p>
-<h1><a class='u' href='#___top' title='click to go to top of document'
+<h3><a class='u'
name="Top-level_entries"
->Top-level entries</a></h1>
+>Top-level entries</a></h3>
-<p>There are a handful of "top-level" files and directories which
belong to the entire index rather than to a particular segment.</p>
+<p>There are a handful of “top-level” files and directories which
belong to the entire index rather than to a particular segment.</p>
-<h2><a class='u' href='#___top' title='click to go to top of document'
+<h4><a class='u'
name="snapshot_XXX.json"
->snapshot_XXX.json</a></h2>
+>snapshot_XXX.json</a></h4>
-<p>A "snapshot" file, e.g. <code>snapshot_m7p.json</code>, is list of
index files and directories. Because index files, once written, are never
modified, the list of entries in a snapshot defines a point-in-time view of the
data in an index.</p>
+<p>A “snapshot” file, e.g. <code>snapshot_m7p.json</code>, is list
of index files and directories. Because index files, once written, are never
modified, the list of entries in a snapshot defines a point-in-time view of the
data in an index.</p>
-<p>Like segment directories, snapshot files also utilize the
unique-base-36-number naming convention; the higher the number, the more recent
the file. The appearance of a new snapshot file within the index directory
constitutes an index update. While a new segment is being written new files may
be added to the index directory, but until a new snapshot file gets written, a
Searcher opening the index for reading won't know about them.</p>
+<p>Like segment directories, snapshot files also utilize the
unique-base-36-number naming convention; the higher the number, the more recent
the file. The appearance of a new snapshot file within the index directory
constitutes an index update. While a new segment is being written new files may
be added to the index directory, but until a new snapshot file gets written, a
Searcher opening the index for reading won’t know about them.</p>
-<h2><a class='u' href='#___top' title='click to go to top of document'
+<h4><a class='u'
name="schema_XXX.json"
->schema_XXX.json</a></h2>
+>schema_XXX.json</a></h4>
-<p>The schema file is a Schema object describing the index's format,
serialized as JSON. It, too, is versioned, and a given snapshot file will
reference one and only one schema file.</p>
+<p>The schema file is a Schema object describing the index’s format,
serialized as JSON. It, too, is versioned, and a given snapshot file will
reference one and only one schema file.</p>
-<h2><a class='u' href='#___top' title='click to go to top of document'
+<h4><a class='u'
name="locks"
->locks</a></h2>
+>locks</a></h4>
<p>By default, only one indexing process may safely modify the index at any
given time. Processes reserve an index by laying claim to the
<code>write.lock</code> file within the <code>locks/</code> directory. A
smattering of other lock files may be used from time to time, as well.</p>
-<h1><a class='u' href='#___top' title='click to go to top of document'
-name="A_segment's_component_parts"
->A segment's component parts</a></h1>
+<h3><a class='u'
+name="A_segment(8217)s_component_parts"
+>A segment’s component parts</a></h3>
-<p>By default, each segment has up to five logical components: lexicon,
postings, document storage, highlight data, and deletions. Binary data from
these components gets stored in virtual files within the "cf.dat"
compound file; metadata is stored in a shared "segmeta.json" file.</p>
+<p>By default, each segment has up to five logical components: lexicon,
postings, document storage, highlight data, and deletions. Binary data from
these components gets stored in virtual files within the “cf.dat”
compound file; metadata is stored in a shared “segmeta.json”
file.</p>
-<h2><a class='u' href='#___top' title='click to go to top of document'
+<h4><a class='u'
name="segmeta.json"
->segmeta.json</a></h2>
+>segmeta.json</a></h4>
<p>The segmeta.json file is a central repository for segment metadata. In
addition to information such as document counts and field numbers, it also
warehouses arbitrary metadata on behalf of individual index components.</p>
-<h2><a class='u' href='#___top' title='click to go to top of document'
+<h4><a class='u'
name="Lexicon"
->Lexicon</a></h2>
+>Lexicon</a></h4>
-<p>Each indexed field gets its own lexicon in each segment. The exact files
involved depend on the field's type, but generally speaking there will be
two parts. First, there's a primary <code>lexicon-XXX.dat</code> file which
houses a complete term list associating terms with corpus frequency statistics,
postings file locations, etc. Second, one or more "lexicon index" files
may be present which contain periodic samples from the primary lexicon file to
facilitate fast lookups.</p>
+<p>Each indexed field gets its own lexicon in each segment. The exact files
involved depend on the field’s type, but generally speaking there will be
two parts. First, there’s a primary <code>lexicon-XXX.dat</code> file
which houses a complete term list associating terms with corpus frequency
statistics, postings file locations, etc. Second, one or more “lexicon
index” files may be present which contain periodic samples from the
primary lexicon file to facilitate fast lookups.</p>
-<h2><a class='u' href='#___top' title='click to go to top of document'
+<h4><a class='u'
name="Postings"
->Postings</a></h2>
+>Postings</a></h4>
-<p>"Posting" is a technical term from the field of <a
href="../../Lucy/Docs/IRTheory.html" class="podlinkpod"
->information retrieval</a>, defined as a single instance of a one term
indexing one document. If you are looking at the index in the back of a book,
and you see that "freedom" is referenced on pages 8, 86, and 240, that
would be three postings, which taken together form a "posting list".
The same terminology applies to an index in electronic form.</p>
+<p>“Posting” is a technical term from the field of <a
href="../../Lucy/Docs/IRTheory.html" class="podlinkpod"
+>information retrieval</a>, defined as a single instance of a one term
indexing one document. If you are looking at the index in the back of a book,
and you see that “freedom” is referenced on pages 8, 86, and 240,
that would be three postings, which taken together form a “posting
list”. The same terminology applies to an index in electronic form.</p>
<p>Each segment has one postings file per indexed field. When a search is
performed for a single term, first that term is looked up in the lexicon. If
the term exists in the segment, the record in the lexicon will contain
information about which postings file to look at and where to look.</p>
-<p>The first thing any posting record tells you is a document id. By iterating
over all the postings associated with a term, you can find all the documents
that match that term, a process which is analogous to looking up page numbers
in a book's index. However, each posting record typically contains other
information in addition to document id, e.g. the positions at which the term
occurs within the field.</p>
+<p>The first thing any posting record tells you is a document id. By iterating
over all the postings associated with a term, you can find all the documents
that match that term, a process which is analogous to looking up page numbers
in a book’s index. However, each posting record typically contains other
information in addition to document id, e.g. the positions at which the term
occurs within the field.</p>
-<h2><a class='u' href='#___top' title='click to go to top of document'
+<h4><a class='u'
name="Documents"
->Documents</a></h2>
+>Documents</a></h4>
<p>The document storage section is a simple database, organized into two
files:</p>
@@ -193,49 +193,49 @@ name="Documents"
<li><b>documents.ix</b> - Document storage index, a solid array of 64-bit
integers where each integer location corresponds to a document id, and the
value at that location points at a file position in the documents.dat file.</li>
</ul>
-<h2><a class='u' href='#___top' title='click to go to top of document'
+<h4><a class='u'
name="Highlight_data"
->Highlight data</a></h2>
+>Highlight data</a></h4>
<p>The files which store data used for excerpting and highlighting are
organized similarly to the files used to store documents.</p>
<ul>
<li><b>highlight.dat</b> - Chunks of serialized highlight data, one per doc
id.</li>
-<li><b>highlight.ix</b> - Highlight data index -- as with the
<code>documents.ix</code> file, a solid array of 64-bit file pointers.</li>
+<li><b>highlight.ix</b> - Highlight data index – as with the
<code>documents.ix</code> file, a solid array of 64-bit file pointers.</li>
</ul>
-<h2><a class='u' href='#___top' title='click to go to top of document'
+<h4><a class='u'
name="Deletions"
->Deletions</a></h2>
+>Deletions</a></h4>
-<p>When a document is "deleted" from a segment, it is not actually
purged right away; it is merely marked as "deleted" via a deletions
file. Deletions files contains bit vectors with one bit for each document in
the segment; if bit #254 is set then document 254 is deleted, and if that
document turns up in a search it will be masked out.</p>
+<p>When a document is “deleted” from a segment, it is not actually
purged right away; it is merely marked as “deleted” via a deletions
file. Deletions files contains bit vectors with one bit for each document in
the segment; if bit #254 is set then document 254 is deleted, and if that
document turns up in a search it will be masked out.</p>
-<p>It is only when a segment's contents are rewritten to a new segment
during the segment-merging process that deleted documents truly go away.</p>
+<p>It is only when a segment’s contents are rewritten to a new segment
during the segment-merging process that deleted documents truly go away.</p>
-<h1><a class='u' href='#___top' title='click to go to top of document'
+<h3><a class='u'
name="Compound_Files"
->Compound Files</a></h1>
+>Compound Files</a></h3>
-<p>If you peer inside an index directory, you won't actually find any
files named "documents.dat", "highlight.ix", etc. unless there
is an indexing process underway. What you will find instead is one
"cf.dat" and one "cfmeta.json" file per segment.</p>
+<p>If you peer inside an index directory, you won’t actually find any
files named “documents.dat”, “highlight.ix”, etc.
unless there is an indexing process underway. What you will find instead is one
“cf.dat” and one “cfmeta.json” file per segment.</p>
-<p>To minimize the need for file descriptors at search-time, all per-segment
binary data files are concatenated together in "cf.dat" at the close of
each indexing session. Information about where each file begins and ends is
stored in <code>cfmeta.json</code>. When the segment is opened for reading, a
single file descriptor per "cf.dat" file can be shared among several
readers.</p>
+<p>To minimize the need for file descriptors at search-time, all per-segment
binary data files are concatenated together in “cf.dat” at the
close of each indexing session. Information about where each file begins and
ends is stored in <code>cfmeta.json</code>. When the segment is opened for
reading, a single file descriptor per “cf.dat” file can be shared
among several readers.</p>
-<h1><a class='u' href='#___top' title='click to go to top of document'
+<h3><a class='u'
name="A_Typical_Search"
->A Typical Search</a></h1>
+>A Typical Search</a></h3>
-<p>Here's a simplified narrative, dramatizing how a search for
"freedom" against a given segment plays out:</p>
+<p>Here’s a simplified narrative, dramatizing how a search for
“freedom” against a given segment plays out:</p>
-<ol>
-<li>The searcher asks the relevant Lexicon Index, "Do you know anything
about 'freedom'?" Lexicon Index replies, "Can't say for
sure, but if the main Lexicon file does, 'freedom' is probably
somewhere around byte 21008".</li>
+<ul>
+<li>The searcher asks the relevant Lexicon Index, “Do you know anything
about ‘freedom’?” Lexicon Index replies, “Can’t
say for sure, but if the main Lexicon file does, ‘freedom’ is
probably somewhere around byte 21008”.</li>
-<li>The main Lexicon tells the searcher "One moment, let me scan our
records... Yes, we have 2 documents which contain 'freedom'. You'll
find them in seg_6/postings-4.dat starting at byte 66991."</li>
+<li>The main Lexicon tells the searcher “One moment, let me scan our
records… Yes, we have 2 documents which contain ‘freedom’.
You’ll find them in seg_6/postings-4.dat starting at byte
66991.”</li>
-<li>The Postings file says "Yep, we have 'freedom', all right!
Document id 40 has 1 'freedom', and document 44 has 8. If you need to
know more, like if any 'freedom' is part of the phrase 'freedom of
speech', ask me about positions!</li>
+<li>The Postings file says “Yep, we have ‘freedom’, all
right! Document id 40 has 1 ‘freedom’, and document 44 has 8. If
you need to know more, like if any ‘freedom’ is part of the phrase
‘freedom of speech’, ask me about positions!</li>
-<li>If the searcher is only looking for 'freedom' in isolation,
that's where it stops. It now knows enough to assign the documents scores
against "freedom", with the 8-freedom document likely ranking higher
than the single-freedom document.</li>
-</ol>
+<li>If the searcher is only looking for ‘freedom’ in isolation,
that’s where it stops. It now knows enough to assign the documents scores
against “freedom”, with the 8-freedom document likely ranking
higher than the single-freedom document.</li>
+</ul>
</div>
Modified:
websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/FileLocking.html
==============================================================================
--- websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/FileLocking.html
(original)
+++ websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/FileLocking.html
Fri Feb 26 12:52:42 2016
@@ -71,15 +71,67 @@
<div>
<a name='___top' class='dummyTopAnchor' ></a>
-<h1><a class='u' href='#___top' title='click to go to top of document'
+<h2><a class='u'
name="NAME"
->NAME</a></h1>
+>NAME</a></h2>
<p>Lucy::Docs::FileLocking - Manage indexes on shared volumes.</p>
-<h1><a class='u' href='#___top' title='click to go to top of document'
-name="SYNOPSIS"
->SYNOPSIS</a></h1>
+<h2><a class='u'
+name="DESCRIPTION"
+>DESCRIPTION</a></h2>
+
+<p>Normally,
+index locking is an invisible process.
+Exclusive write access is controlled via lockfiles within the index directory
and problems only arise if multiple processes attempt to acquire the write lock
simultaneously; search-time processes do not ordinarily require locking at
all.</p>
+
+<p>On shared volumes,
+however,
+the default locking mechanism fails,
+and manual intervention becomes necessary.</p>
+
+<p>Both read and write applications accessing an index on a shared volume need
to identify themselves with a unique <code>host</code> id,
+e.g.
+hostname or ip address.
+Knowing the host id makes it possible to tell which lockfiles belong to other
machines and therefore must not be removed when the lockfile’s pid number
appears not to correspond to an active process.</p>
+
+<p>At index-time,
+the danger is that multiple indexing processes from different machines which
fail to specify a unique <code>host</code> id can delete each others’
lockfiles and then attempt to modify the index at the same time,
+causing index corruption.
+The search-time problem is more complex.</p>
+
+<p>Once an index file is no longer listed in the most recent snapshot,
+Indexer attempts to delete it as part of a post-<a href="lucy:Indexer.Commit"
class="podlinkurl"
+>lucy:Indexer.Commit</a> cleanup routine.
+It is possible that at the moment an Indexer is deleting files which it
believes no longer needed,
+a Searcher referencing an earlier snapshot is in fact using them.
+The more often that an index is either updated or searched,
+the more likely it is that this conflict will arise from time to time.</p>
+
+<p>Ordinarily,
+the deletion attempts are not a problem.
+On a typical unix volume,
+the files will be deleted in name only: any process which holds an open
filehandle against a given file will continue to have access,
+and the file won’t actually get vaporized until the last filehandle is
cleared.
+Thanks to “delete on last close semantics”,
+an Indexer can’t truly delete the file out from underneath an active
Searcher.
+On Windows,
+where file deletion fails whenever any process holds an open handle,
+the situation is different but still workable: Indexer just keeps retrying
after each commit until deletion finally succeeds.</p>
+
+<p>On NFS,
+however,
+the system breaks,
+because NFS allows files to be deleted out from underneath active processes.
+Should this happen,
+the unlucky read process will crash with a “Stale NFS filehandle”
exception.</p>
+
+<p>Under normal circumstances,
+it is neither necessary nor desirable for IndexReaders to secure read locks
against an index,
+but for NFS we have to make an exception.
+LockFactory’s <a href="lucy:LockFactory.Make_Shared_Lock"
class="podlinkurl"
+>lucy:LockFactory.Make_Shared_Lock</a> method exists for this reason;
supplying an IndexManager instance to IndexReader’s constructor activates
an internal locking mechanism using <a href="lucy:LockFactory.Make_Shared_Lock"
class="podlinkurl"
+>lucy:LockFactory.Make_Shared_Lock</a> which prevents concurrent indexing
processes from deleting files that are needed by active readers.</p>
<pre> use Sys::Hostname qw( hostname );
my $hostname = hostname() or die "Can't get unique hostname";
@@ -98,27 +150,7 @@ name="SYNOPSIS"
);
my $searcher = Lucy::Search::IndexSearcher->new( index => $reader
);</pre>
-<h1><a class='u' href='#___top' title='click to go to top of document'
-name="DESCRIPTION"
->DESCRIPTION</a></h1>
-
-<p>Normally, index locking is an invisible process. Exclusive write access is
controlled via lockfiles within the index directory and problems only arise if
multiple processes attempt to acquire the write lock simultaneously;
search-time processes do not ordinarily require locking at all.</p>
-
-<p>On shared volumes, however, the default locking mechanism fails, and manual
intervention becomes necessary.</p>
-
-<p>Both read and write applications accessing an index on a shared volume need
to identify themselves with a unique <code>host</code> id, e.g. hostname or ip
address. Knowing the host id makes it possible to tell which lockfiles belong
to other machines and therefore must not be removed when the lockfile's pid
number appears not to correspond to an active process.</p>
-
-<p>At index-time, the danger is that multiple indexing processes from
different machines which fail to specify a unique <code>host</code> id can
delete each others' lockfiles and then attempt to modify the index at the
same time, causing index corruption. The search-time problem is more
complex.</p>
-
-<p>Once an index file is no longer listed in the most recent snapshot, Indexer
attempts to delete it as part of a post-commit() cleanup routine. It is
possible that at the moment an Indexer is deleting files which it believes no
longer needed, a Searcher referencing an earlier snapshot is in fact using
them. The more often that an index is either updated or searched, the more
likely it is that this conflict will arise from time to time.</p>
-
-<p>Ordinarily, the deletion attempts are not a problem. On a typical unix
volume, the files will be deleted in name only: any process which holds an open
filehandle against a given file will continue to have access, and the file
won't actually get vaporized until the last filehandle is cleared. Thanks
to "delete on last close semantics", an Indexer can't truly delete
the file out from underneath an active Searcher. On Windows, where file
deletion fails whenever any process holds an open handle, the situation is
different but still workable: Indexer just keeps retrying after each commit
until deletion finally succeeds.</p>
-
-<p>On NFS, however, the system breaks, because NFS allows files to be deleted
out from underneath active processes. Should this happen, the unlucky read
process will crash with a "Stale NFS filehandle" exception.</p>
-
-<p>Under normal circumstances, it is neither necessary nor desirable for
IndexReaders to secure read locks against an index, but for NFS we have to make
an exception. LockFactory's make_shared_lock() method exists for this
reason; supplying an IndexManager instance to IndexReader's constructor
activates an internal locking mechanism using make_shared_lock() which prevents
concurrent indexing processes from deleting files that are needed by active
readers.</p>
-
-<p>Since shared locks are implemented using lockfiles located in the index
directory (as are exclusive locks), reader applications must have write access
for read locking to work. Stale lock files from crashed processes are
ordinarily cleared away the next time the same machine -- as identified by the
<code>host</code> parameter -- opens another IndexReader. (The classic
technique of timing out lock files is not feasible because search processes may
lie dormant indefinitely.) However, please be aware that if the last thing a
given machine does is crash, lock files belonging to it may persist, preventing
deletion of obsolete index data.</p>
+<p>Since shared locks are implemented using lockfiles located in the index
directory (as are exclusive locks), reader applications must have write access
for read locking to work. Stale lock files from crashed processes are
ordinarily cleared away the next time the same machine – as identified by
the <code>host</code> parameter – opens another IndexReader. (The classic
technique of timing out lock files is not feasible because search processes may
lie dormant indefinitely.) However, please be aware that if the last thing a
given machine does is crash, lock files belonging to it may persist, preventing
deletion of obsolete index data.</p>
</div>
Modified: websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/IRTheory.html
==============================================================================
--- websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/IRTheory.html
(original)
+++ websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/IRTheory.html Fri
Feb 26 12:52:42 2016
@@ -71,25 +71,25 @@
<div>
<a name='___top' class='dummyTopAnchor' ></a>
-<h1><a class='u' href='#___top' title='click to go to top of document'
+<h2><a class='u'
name="NAME"
->NAME</a></h1>
+>NAME</a></h2>
-<p>Lucy::Docs::IRTheory - Crash course in information retrieval.</p>
+<p>Lucy::Docs::IRTheory - Crash course in information retrieval</p>
-<h1><a class='u' href='#___top' title='click to go to top of document'
-name="ABSTRACT"
->ABSTRACT</a></h1>
+<h2><a class='u'
+name="DESCRIPTION"
+>DESCRIPTION</a></h2>
<p>Just enough Information Retrieval theory to find your way around Apache
Lucy.</p>
-<h1><a class='u' href='#___top' title='click to go to top of document'
+<h3><a class='u'
name="Terminology"
->Terminology</a></h1>
+>Terminology</a></h3>
<p>Lucy uses some terminology from the field of information retrieval which
may be unfamiliar to many users.
-"Document" and "term" mean pretty much what you'd expect
them to,
-but others such as "posting" and "inverted index" need a
formal introduction:</p>
+“Document” and “term” mean pretty much what
you’d expect them to,
+but others such as “posting” and “inverted index” need
a formal introduction:</p>
<ul>
<li><i>document</i> - An atomic unit of retrieval.</li>
@@ -109,23 +109,21 @@ but others such as "posting" and
it loads these abstract,
distilled definitions down with useful traits.
For instance,
-a "posting" in its most rarefied form is simply a term-document
pairing; in Lucy,
-the class <a href="../../Lucy/Index/Posting/MatchPosting.html"
class="podlinkpod"
->Lucy::Index::Posting::MatchPosting</a> fills this role.
+a “posting” in its most rarefied form is simply a term-document
pairing; in Lucy,
+the class MatchPosting fills this role.
However,
by associating additional information with a posting like the number of times
the term occurs in the document,
-we can turn it into a <a href="../../Lucy/Index/Posting/ScorePosting.html"
class="podlinkpod"
->ScorePosting</a>,
+we can turn it into a ScorePosting,
making it possible to rank documents by relevance rather than just list
documents which happen to match in no particular order.</p>
-<h1><a class='u' href='#___top' title='click to go to top of document'
+<h3><a class='u'
name="TF/IDF_ranking_algorithm"
->TF/IDF ranking algorithm</a></h1>
+>TF/IDF ranking algorithm</a></h3>
-<p>Lucy uses a variant of the well-established "Term Frequency / Inverse
Document Frequency" weighting scheme.
+<p>Lucy uses a variant of the well-established “Term Frequency / Inverse
Document Frequency” weighting scheme.
A thorough treatment of TF/IDF is too ambitious for our present purposes,
but in a nutshell,
-it means that...</p>
+it means that…</p>
<ul>
<li>in a search for <code>skate park</code>,
@@ -134,7 +132,7 @@ documents which score well for the compa
<li>a 10-word text which has one occurrence each of both <code>skate</code>
and <code>park</code> will rank higher than a 1000-word text which also
contains one occurrence of each.</li>
</ul>
-<p>A web search for "tf idf" will turn up many excellent explanations
of the algorithm.</p>
+<p>A web search for “tf idf” will turn up many excellent
explanations of the algorithm.</p>
</div>
Modified: websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/Tutorial.html
==============================================================================
--- websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/Tutorial.html
(original)
+++ websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/Tutorial.html Fri
Feb 26 12:52:42 2016
@@ -71,81 +71,75 @@
<div>
<a name='___top' class='dummyTopAnchor' ></a>
-<h1><a class='u' href='#___top' title='click to go to top of document'
+<h2><a class='u'
name="NAME"
->NAME</a></h1>
+>NAME</a></h2>
<p>Lucy::Docs::Tutorial - Step-by-step introduction to Apache Lucy.</p>
-<h1><a class='u' href='#___top' title='click to go to top of document'
-name="ABSTRACT"
->ABSTRACT</a></h1>
+<h2><a class='u'
+name="DESCRIPTION"
+>DESCRIPTION</a></h2>
-<p>Explore Apache Lucy's basic functionality by starting with a minimalist
CGI search app based on <a href="../../Lucy/Simple.html" class="podlinkpod"
->Lucy::Simple</a> and transforming it,
+<p>Explore Apache Lucy’s basic functionality by starting with a
minimalist CGI search app based on Lucy::Simple and transforming it,
step by step,
-into an "advanced search" interface utilizing more flexible core
modules like <a href="../../Lucy/Index/Indexer.html" class="podlinkpod"
->Lucy::Index::Indexer</a> and <a href="../../Lucy/Search/IndexSearcher.html"
class="podlinkpod"
->Lucy::Search::IndexSearcher</a>.</p>
-
-<h1><a class='u' href='#___top' title='click to go to top of document'
-name="DESCRIPTION"
->DESCRIPTION</a></h1>
+into an “advanced search” interface utilizing more flexible core
modules like <a href="../../Lucy/Index/Indexer.html" class="podlinkpod"
+>Indexer</a> and <a href="../../Lucy/Search/IndexSearcher.html"
class="podlinkpod"
+>IndexSearcher</a>.</p>
-<h2><a class='u' href='#___top' title='click to go to top of document'
+<h3><a class='u'
name="Chapters"
->Chapters</a></h2>
+>Chapters</a></h3>
<ul>
-<li><a href="../../Lucy/Docs/Tutorial/Simple.html" class="podlinkpod"
->Lucy::Docs::Tutorial::Simple</a> - Build a bare-bones search app using <a
href="../../Lucy/Simple.html" class="podlinkpod"
->Lucy::Simple</a>.</li>
+<li><a href="../../Lucy/Docs/Tutorial/SimpleTutorial.html" class="podlinkpod"
+>SimpleTutorial</a> - Build a bare-bones search app using Lucy::Simple.</li>
-<li><a href="../../Lucy/Docs/Tutorial/BeyondSimple.html" class="podlinkpod"
->Lucy::Docs::Tutorial::BeyondSimple</a> - Rebuild the app using core classes
like <a href="../../Lucy/Index/Indexer.html" class="podlinkpod"
+<li><a href="../../Lucy/Docs/Tutorial/BeyondSimpleTutorial.html"
class="podlinkpod"
+>BeyondSimpleTutorial</a> - Rebuild the app using core classes like <a
href="../../Lucy/Index/Indexer.html" class="podlinkpod"
>Indexer</a> and <a href="../../Lucy/Search/IndexSearcher.html"
>class="podlinkpod"
>IndexSearcher</a> in place of Lucy::Simple.</li>
-<li><a href="../../Lucy/Docs/Tutorial/FieldType.html" class="podlinkpod"
->Lucy::Docs::Tutorial::FieldType</a> - Experiment with different field
characteristics using subclasses of <a href="../../Lucy/Plan/FieldType.html"
class="podlinkpod"
->Lucy::Plan::FieldType</a>.</li>
-
-<li><a href="../../Lucy/Docs/Tutorial/Analysis.html" class="podlinkpod"
->Lucy::Docs::Tutorial::Analysis</a> - Examine how the choice of <a
href="../../Lucy/Analysis/Analyzer.html" class="podlinkpod"
->Lucy::Analysis::Analyzer</a> subclass affects search results.</li>
+<li><a href="../../Lucy/Docs/Tutorial/FieldTypeTutorial.html"
class="podlinkpod"
+>FieldTypeTutorial</a> - Experiment with different field characteristics using
subclasses of <a href="../../Lucy/Plan/FieldType.html" class="podlinkpod"
+>FieldType</a>.</li>
+
+<li><a href="../../Lucy/Docs/Tutorial/AnalysisTutorial.html" class="podlinkpod"
+>AnalysisTutorial</a> - Examine how the choice of <a
href="../../Lucy/Analysis/Analyzer.html" class="podlinkpod"
+>Analyzer</a> subclass affects search results.</li>
-<li><a href="../../Lucy/Docs/Tutorial/Highlighter.html" class="podlinkpod"
->Lucy::Docs::Tutorial::Highlighter</a> - Augment search results with
highlighted excerpts.</li>
+<li><a href="../../Lucy/Docs/Tutorial/HighlighterTutorial.html"
class="podlinkpod"
+>HighlighterTutorial</a> - Augment search results with highlighted
excerpts.</li>
-<li><a href="../../Lucy/Docs/Tutorial/QueryObjects.html" class="podlinkpod"
->Lucy::Docs::Tutorial::QueryObjects</a> - Unlock advanced search features by
using Query objects instead of query strings.</li>
+<li><a href="../../Lucy/Docs/Tutorial/QueryObjectsTutorial.html"
class="podlinkpod"
+>QueryObjectsTutorial</a> - Unlock advanced search features by using Query
objects instead of query strings.</li>
</ul>
-<h2><a class='u' href='#___top' title='click to go to top of document'
+<h3><a class='u'
name="Source_materials"
->Source materials</a></h2>
+>Source materials</a></h3>
-<p>The source material used by the tutorial app -- a multi-text-file
presentation of the United States constitution -- can be found in the
<code>sample</code> directory at the root of the Lucy distribution,
+<p>The source material used by the tutorial app – a multi-text-file
presentation of the United States constitution – can be found in the
<code>sample</code> directory at the root of the Lucy distribution,
along with finished indexing and search apps.</p>
<pre> sample/indexer.pl # indexing app
sample/search.cgi # search app
sample/us_constitution # corpus</pre>
-<h2><a class='u' href='#___top' title='click to go to top of document'
+<h3><a class='u'
name="Conventions"
->Conventions</a></h2>
+>Conventions</a></h3>
<p>The user is expected to be familiar with OO Perl and basic CGI
programming.</p>
<p>The code in this tutorial assumes a Unix-flavored operating system and the
Apache webserver, but will work with minor modifications on other setups.</p>
-<h1><a class='u' href='#___top' title='click to go to top of document'
-name="SEE_ALSO"
->SEE ALSO</a></h1>
+<h3><a class='u'
+name="See_also"
+>See also</a></h3>
<p>More advanced and esoteric subjects are covered in <a
href="../../Lucy/Docs/Cookbook.html" class="podlinkpod"
->Lucy::Docs::Cookbook</a>.</p>
+>Cookbook</a>.</p>
</div>
Added:
websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/Tutorial/AnalysisTutorial.html
==============================================================================
---
websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/Tutorial/AnalysisTutorial.html
(added)
+++
websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/Tutorial/AnalysisTutorial.html
Fri Feb 26 12:52:42 2016
@@ -0,0 +1,168 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
+<html lang="en">
+ <head>
+ <meta http-equiv="Content-Type" content="text/html;charset=UTF-8">
+ <title>Lucy::Docs::Tutorial::AnalysisTutorial - Apache Lucy
Documentation</title>
+ <link rel="stylesheet" type="text/css" media="screen"
href="http://lucy.apache.org/css/lucy.css";>
+ </head>
+
+ <body>
+
+ <div id="lucy-rigid_wrapper">
+
+ <div id="lucy-top" class="container_16 lucy-white_box_3d">
+
+ <div id="lucy-logo_box" class="grid_8">
+ <a href="/"><img
src="http://lucy.apache.org/images/lucy_logo_150x100.png"; alt="Apache
Lucyâ¢"></a>
+ </div> <!-- lucy-logo_box -->
+
+ <div #id="lucy-top_nav_box" class="grid_8">
+ <div id="lucy-top_nav_bar" class="container_8">
+ <ul>
+ <li><a href="http://www.apache.org/"; title="Apache Software
Foundation">Apache Software Foundation</a></li>
+ <li><a href="http://www.apache.org/licenses/";
title="License">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html";
title="Sponsorship">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html";
title="Thanks">Thanks</a></li>
+ <li><a href="http://www.apache.org/security/ "
title="Security">Security</a></li>
+ </ul>
+ </div> <!-- lucy-top_nav_bar -->
+ <p><a href="http://www.apache.org/";>Apache</a> » <a
href="/">Lucy</a> » <a
href="/docs/">Docs</a> » <a
href="/docs/test/">Test</a> » <a
href="/docs/test/Lucy/">Lucy</a> » <a
href="/docs/test/Lucy/Docs/">Docs</a> » <a
href="/docs/test/Lucy/Docs/Tutorial/">Tutorial</a></p>
+ <form name="lucy-top_search_box" id="lucy-top_search_box"
action="http://www.google.com/search"; method="get">
+ <input value="*.apache.org" name="sitesearch" type="hidden"/>
+ <input type="text" name="q" id="query" style="width:85%">
+ <input type="submit" id="submit" value="Search">
+ </form>
+ </div> <!-- lucy-top_nav_box -->
+
+ <div class="clear"></div>
+
+ </div> <!-- lucy-top -->
+
+ <div id="lucy-main_content" class="container_16 lucy-white_box_3d">
+
+ <div class="grid_4" id="lucy-left_nav_box">
+ <h6>About</h6>
+ <ul>
+ <li><a href="/">Welcome</a></li>
+ <li><a href="/clownfish.html">Clownfish</a></li>
+ <li><a href="/faq.html">FAQ</a></li>
+ <li><a href="/people.html">People</a></li>
+ </ul>
+ <h6>Resources</h6>
+ <ul>
+ <li><a href="/download.html">Download</a></li>
+ <li><a href="/mailing_lists.html">Mailing Lists</a></li>
+ <li><a href="/docs/perl/">Documentation</a></li>
+ <li><a href="http://wiki.apache.org/lucy/";>Wiki</a></li>
+ <li><a href="https://issues.apache.org/jira/browse/LUCY";>Issue
Tracker</a></li>
+ <li><a href="/version_control.html">Version Control</a></li>
+ </ul>
+ <h6>Related Projects</h6>
+ <ul>
+ <li><a href="http://lucene.apache.org/java/";>Lucene</a></li>
+ <li><a href="http://lucene.apache.org/solr/";>Solr</a></li>
+ <li><a
href="http://incubator.apache.org/lucene.net/";>Lucene.NET</a></li>
+ <li><a
href="http://lucene.apache.org/pylucene/";>PyLucene</a></li>
+ <li><a href="http://lucene.apache.org/openrelevance/";>Open
Relevance</a></li>
+ </ul>
+ </div> <!-- lucy-left_nav_box -->
+
+ <div id="lucy-main_content_box" class="grid_9">
+ <div>
+<a name='___top' class='dummyTopAnchor' ></a>
+
+<h2><a class='u'
+name="NAME"
+>NAME</a></h2>
+
+<p>Lucy::Docs::Tutorial::AnalysisTutorial - How to choose and use
Analyzers.</p>
+
+<h2><a class='u'
+name="DESCRIPTION"
+>DESCRIPTION</a></h2>
+
+<p>Try swapping out the EasyAnalyzer in our Schema for a <a
href="../../../Lucy/Analysis/StandardTokenizer.html" class="podlinkpod"
+>StandardTokenizer</a>:</p>
+
+<pre> my $tokenizer = Lucy::Analysis::StandardTokenizer->new;
+ my $type = Lucy::Plan::FullTextType->new(
+ analyzer => $tokenizer,
+ );</pre>
+
+<p>Search for <code>senate</code>, <code>Senate</code>, and
<code>Senator</code> before and after making the change and re-indexing.</p>
+
+<p>Under EasyAnalyzer, the results are identical for all three searches, but
under StandardTokenizer, searches are case-sensitive, and the result sets for
<code>Senate</code> and <code>Senator</code> are distinct.</p>
+
+<h3><a class='u'
+name="EasyAnalyzer"
+>EasyAnalyzer</a></h3>
+
+<p>What’s happening is that <a
href="../../../Lucy/Analysis/EasyAnalyzer.html" class="podlinkpod"
+>EasyAnalyzer</a> is performing more aggressive processing than
StandardTokenizer. In addition to tokenizing, it’s also converting all
text to lower case so that searches are case-insensitive, and using a
“stemming” algorithm to reduce related words to a common stem
(<code>senat</code>, in this case).</p>
+
+<p>EasyAnalyzer is actually multiple Analyzers wrapped up in a single package.
In this case, it’s three-in-one, since specifying a EasyAnalyzer with
<code>language => 'en'</code> is equivalent to this snippet
creating a <a href="../../../Lucy/Analysis/PolyAnalyzer.html" class="podlinkpod"
+>PolyAnalyzer</a>:</p>
+
+<pre> my $tokenizer = Lucy::Analysis::StandardTokenizer->new;
+ my $normalizer = Lucy::Analysis::Normalizer->new;
+ my $stemmer = Lucy::Analysis::SnowballStemmer->new( language
=> 'en' );
+ my $polyanalyzer = Lucy::Analysis::PolyAnalyzer->new(
+ analyzers => [ $tokenizer, $normalizer, $stemmer ],
+ );</pre>
+
+<p>You can add or subtract Analyzers from there if you like. Try adding a
fourth Analyzer, a SnowballStopFilter for suppressing “stopwords”
like “the”, “if”, and “maybe”.</p>
+
+<pre> my $stopfilter = Lucy::Analysis::SnowballStopFilter->new(
+ language => 'en',
+ );
+ my $polyanalyzer = Lucy::Analysis::PolyAnalyzer->new(
+ analyzers => [ $tokenizer, $normalizer, $stopfilter, $stemmer ],
+ );</pre>
+
+<p>Also, try removing the SnowballStemmer.</p>
+
+<pre> my $polyanalyzer = Lucy::Analysis::PolyAnalyzer->new(
+ analyzers => [ $tokenizer, $normalizer ],
+ );</pre>
+
+<p>The original choice of a stock English EasyAnalyzer probably still yields
the best results for this document collection, but you get the idea: sometimes
you want a different Analyzer.</p>
+
+<h3><a class='u'
+name="When_the_best_Analyzer_is_no_Analyzer"
+>When the best Analyzer is no Analyzer</a></h3>
+
+<p>Sometimes you don’t want an Analyzer at all. That was true for our
“url” field because we didn’t need it to be searchable, but
it’s also true for certain types of searchable fields. For instance,
“category” fields are often set up to match exactly or not at all,
as are fields like “last_name” (because you may not want to
conflate results for “Humphrey” and “Humphries”).</p>
+
+<p>To specify that there should be no analysis performed at all, use
StringType:</p>
+
+<pre> my $type = Lucy::Plan::StringType->new;
+ $schema->spec_field( name => 'category', type => $type
);</pre>
+
+<h3><a class='u'
+name="Highlighting_up_next"
+>Highlighting up next</a></h3>
+
+<p>In our next tutorial chapter, <a
href="../../../Lucy/Docs/Tutorial/HighlighterTutorial.html" class="podlinkpod"
+>HighlighterTutorial</a>, we’ll add highlighted excerpts from the
“content” field to our search results.</p>
+
+</div>
+
+ </div> <!-- lucy-main_content_box -->
+ <div class="clear"></div>
+
+ </div> <!-- lucy-main_content -->
+
+ <div id="lucy-copyright" class="container_16">
+ <p>Copyright © 2010-2015 The Apache Software Foundation, Licensed
under the
+ <a href="http://www.apache.org/licenses/LICENSE-2.0";>Apache
License, Version 2.0</a>.
+ <br/>
+ Apache Lucy, Lucy, Apache, the Apache feather logo, and the Apache
Lucy project logo are trademarks of The
+ Apache Software Foundation. All other marks mentioned may be
trademarks or registered trademarks of their
+ respective owners.
+ </p>
+ </div> <!-- lucy-copyright -->
+
+ </div> <!-- lucy-rigid_wrapper -->
+
+ </body>
+</html>
Added:
websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/Tutorial/BeyondSimpleTutorial.html
==============================================================================
---
websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/Tutorial/BeyondSimpleTutorial.html
(added)
+++
websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/Tutorial/BeyondSimpleTutorial.html
Fri Feb 26 12:52:42 2016
@@ -0,0 +1,227 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
+<html lang="en">
+ <head>
+ <meta http-equiv="Content-Type" content="text/html;charset=UTF-8">
+ <title>Lucy::Docs::Tutorial::BeyondSimpleTutorial - Apache Lucy
Documentation</title>
+ <link rel="stylesheet" type="text/css" media="screen"
href="http://lucy.apache.org/css/lucy.css";>
+ </head>
+
+ <body>
+
+ <div id="lucy-rigid_wrapper">
+
+ <div id="lucy-top" class="container_16 lucy-white_box_3d">
+
+ <div id="lucy-logo_box" class="grid_8">
+ <a href="/"><img
src="http://lucy.apache.org/images/lucy_logo_150x100.png"; alt="Apache
Lucyâ¢"></a>
+ </div> <!-- lucy-logo_box -->
+
+ <div #id="lucy-top_nav_box" class="grid_8">
+ <div id="lucy-top_nav_bar" class="container_8">
+ <ul>
+ <li><a href="http://www.apache.org/"; title="Apache Software
Foundation">Apache Software Foundation</a></li>
+ <li><a href="http://www.apache.org/licenses/";
title="License">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html";
title="Sponsorship">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html";
title="Thanks">Thanks</a></li>
+ <li><a href="http://www.apache.org/security/ "
title="Security">Security</a></li>
+ </ul>
+ </div> <!-- lucy-top_nav_bar -->
+ <p><a href="http://www.apache.org/";>Apache</a> » <a
href="/">Lucy</a> » <a
href="/docs/">Docs</a> » <a
href="/docs/test/">Test</a> » <a
href="/docs/test/Lucy/">Lucy</a> » <a
href="/docs/test/Lucy/Docs/">Docs</a> » <a
href="/docs/test/Lucy/Docs/Tutorial/">Tutorial</a></p>
+ <form name="lucy-top_search_box" id="lucy-top_search_box"
action="http://www.google.com/search"; method="get">
+ <input value="*.apache.org" name="sitesearch" type="hidden"/>
+ <input type="text" name="q" id="query" style="width:85%">
+ <input type="submit" id="submit" value="Search">
+ </form>
+ </div> <!-- lucy-top_nav_box -->
+
+ <div class="clear"></div>
+
+ </div> <!-- lucy-top -->
+
+ <div id="lucy-main_content" class="container_16 lucy-white_box_3d">
+
+ <div class="grid_4" id="lucy-left_nav_box">
+ <h6>About</h6>
+ <ul>
+ <li><a href="/">Welcome</a></li>
+ <li><a href="/clownfish.html">Clownfish</a></li>
+ <li><a href="/faq.html">FAQ</a></li>
+ <li><a href="/people.html">People</a></li>
+ </ul>
+ <h6>Resources</h6>
+ <ul>
+ <li><a href="/download.html">Download</a></li>
+ <li><a href="/mailing_lists.html">Mailing Lists</a></li>
+ <li><a href="/docs/perl/">Documentation</a></li>
+ <li><a href="http://wiki.apache.org/lucy/";>Wiki</a></li>
+ <li><a href="https://issues.apache.org/jira/browse/LUCY";>Issue
Tracker</a></li>
+ <li><a href="/version_control.html">Version Control</a></li>
+ </ul>
+ <h6>Related Projects</h6>
+ <ul>
+ <li><a href="http://lucene.apache.org/java/";>Lucene</a></li>
+ <li><a href="http://lucene.apache.org/solr/";>Solr</a></li>
+ <li><a
href="http://incubator.apache.org/lucene.net/";>Lucene.NET</a></li>
+ <li><a
href="http://lucene.apache.org/pylucene/";>PyLucene</a></li>
+ <li><a href="http://lucene.apache.org/openrelevance/";>Open
Relevance</a></li>
+ </ul>
+ </div> <!-- lucy-left_nav_box -->
+
+ <div id="lucy-main_content_box" class="grid_9">
+ <div>
+<a name='___top' class='dummyTopAnchor' ></a>
+
+<h2><a class='u'
+name="NAME"
+>NAME</a></h2>
+
+<p>Lucy::Docs::Tutorial::BeyondSimpleTutorial - A more flexible app
structure.</p>
+
+<h2><a class='u'
+name="DESCRIPTION"
+>DESCRIPTION</a></h2>
+
+<h3><a class='u'
+name="Goal"
+>Goal</a></h3>
+
+<p>In this tutorial chapter,
+we’ll refactor the apps we built in <a
href="../../../Lucy/Docs/Tutorial/SimpleTutorial.html" class="podlinkpod"
+>SimpleTutorial</a> so that they look exactly the same from the end
user’s point of view,
+but offer the developer greater possibilites for expansion.</p>
+
+<p>To achieve this,
+we’ll ditch Lucy::Simple and replace it with the classes that it uses
internally:</p>
+
+<ul>
+<li><a href="../../../Lucy/Plan/Schema.html" class="podlinkpod"
+>Schema</a> - Plan out your index.</li>
+
+<li><a href="../../../Lucy/Plan/FullTextType.html" class="podlinkpod"
+>FullTextType</a> - Field type for full text search.</li>
+
+<li><a href="../../../Lucy/Analysis/EasyAnalyzer.html" class="podlinkpod"
+>EasyAnalyzer</a> - A one-size-fits-all parser/tokenizer.</li>
+
+<li><a href="../../../Lucy/Index/Indexer.html" class="podlinkpod"
+>Indexer</a> - Manipulate index content.</li>
+
+<li><a href="../../../Lucy/Search/IndexSearcher.html" class="podlinkpod"
+>IndexSearcher</a> - Search an index.</li>
+
+<li><a href="../../../Lucy/Search/Hits.html" class="podlinkpod"
+>Hits</a> - Iterate over hits returned by a Searcher.</li>
+</ul>
+
+<h3><a class='u'
+name="Adaptations_to_indexer.pl"
+>Adaptations to indexer.pl</a></h3>
+
+<p>After we load our modules…</p>
+
+<pre> use Lucy::Plan::Schema;
+ use Lucy::Plan::FullTextType;
+ use Lucy::Analysis::EasyAnalyzer;
+ use Lucy::Index::Indexer;</pre>
+
+<p>… the first item we’re going need is a <a
href="../../../Lucy/Plan/Schema.html" class="podlinkpod"
+>Schema</a>.</p>
+
+<p>The primary job of a Schema is to specify what fields are available and how
they’re defined. We’ll start off with three fields: title, content
and url.</p>
+
+<pre> # Create Schema.
+ my $schema = Lucy::Plan::Schema->new;
+ my $easyanalyzer = Lucy::Analysis::EasyAnalyzer->new(
+ language => 'en',
+ );
+ my $type = Lucy::Plan::FullTextType->new(
+ analyzer => $easyanalyzer,
+ );
+ $schema->spec_field( name => 'title', type => $type
);
+ $schema->spec_field( name => 'content', type => $type
);
+ $schema->spec_field( name => 'url', type => $type
);</pre>
+
+<p>All of the fields are spec’d out using the <a
href="../../../Lucy/Plan/FullTextType.html" class="podlinkpod"
+>FullTextType</a> FieldType, indicating that they will be searchable as
“full text” – which means that they can be searched for
individual words. The “analyzer”, which is unique to FullTextType
fields, is what breaks up the text into searchable tokens.</p>
+
+<p>Next, we’ll swap our Lucy::Simple object out for an <a
href="../../../Lucy/Index/Indexer.html" class="podlinkpod"
+>Indexer</a>. The substitution will be straightforward because Simple has
merely been serving as a thin wrapper around an inner Indexer, and we’ll
just be peeling away the wrapper.</p>
+
+<p>First, replace the constructor:</p>
+
+<pre> # Create Indexer.
+ my $indexer = Lucy::Index::Indexer->new(
+ index => $path_to_index,
+ schema => $schema,
+ create => 1,
+ truncate => 1,
+ );</pre>
+
+<p>Next, have the <code>indexer</code> object <a
href="../../../Lucy/Index/Indexer.html#add_doc" class="podlinkpod"
+>add_doc()</a> where we were having the <code>lucy</code> object adding the
document before:</p>
+
+<pre> foreach my $filename (@filenames) {
+ my $doc = parse_file($filename);
+ $indexer->add_doc($doc);
+ }</pre>
+
+<p>There’s only one extra step required: at the end of the app, you must
call commit() explicitly to close the indexing session and commit your changes.
(Lucy::Simple hides this detail, calling commit() implicitly when it needs
to).</p>
+
+<pre> $indexer->commit;</pre>
+
+<h3><a class='u'
+name="Adaptations_to_search.cgi"
+>Adaptations to search.cgi</a></h3>
+
+<p>In our search app as in our indexing app, Lucy::Simple has served as a thin
wrapper – this time around <a
href="../../../Lucy/Search/IndexSearcher.html" class="podlinkpod"
+>IndexSearcher</a> and <a href="../../../Lucy/Search/Hits.html"
class="podlinkpod"
+>Hits</a>. Swapping out Simple for these two classes is also
straightforward:</p>
+
+<pre> use Lucy::Search::IndexSearcher;
+
+ my $searcher = Lucy::Search::IndexSearcher->new(
+ index => $path_to_index,
+ );
+ my $hits = $searcher->hits( # returns a Hits object, not a hit count
+ query => $q,
+ offset => $offset,
+ num_wanted => $page_size,
+ );
+ my $hit_count = $hits->total_hits; # get the hit count here
+
+ ...
+
+ while ( my $hit = $hits->next ) {
+ ...
+ }</pre>
+
+<h3><a class='u'
+name="Hooray!"
+>Hooray!</a></h3>
+
+<p>Congratulations! Your apps do the same thing as before… but now
they’ll be easier to customize.</p>
+
+<p>In our next chapter, <a
href="../../../Lucy/Docs/Tutorial/FieldTypeTutorial.html" class="podlinkpod"
+>FieldTypeTutorial</a>, we’ll explore how to assign different behaviors
to different fields.</p>
+
+</div>
+
+ </div> <!-- lucy-main_content_box -->
+ <div class="clear"></div>
+
+ </div> <!-- lucy-main_content -->
+
+ <div id="lucy-copyright" class="container_16">
+ <p>Copyright © 2010-2015 The Apache Software Foundation, Licensed
under the
+ <a href="http://www.apache.org/licenses/LICENSE-2.0";>Apache
License, Version 2.0</a>.
+ <br/>
+ Apache Lucy, Lucy, Apache, the Apache feather logo, and the Apache
Lucy project logo are trademarks of The
+ Apache Software Foundation. All other marks mentioned may be
trademarks or registered trademarks of their
+ respective owners.
+ </p>
+ </div> <!-- lucy-copyright -->
+
+ </div> <!-- lucy-rigid_wrapper -->
+
+ </body>
+</html>
Added:
websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/Tutorial/FieldTypeTutorial.html
==============================================================================
---
websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/Tutorial/FieldTypeTutorial.html
(added)
+++
websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/Tutorial/FieldTypeTutorial.html
Fri Feb 26 12:52:42 2016
@@ -0,0 +1,151 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
+<html lang="en">
+ <head>
+ <meta http-equiv="Content-Type" content="text/html;charset=UTF-8">
+ <title>Lucy::Docs::Tutorial::FieldTypeTutorial - Apache Lucy
Documentation</title>
+ <link rel="stylesheet" type="text/css" media="screen"
href="http://lucy.apache.org/css/lucy.css";>
+ </head>
+
+ <body>
+
+ <div id="lucy-rigid_wrapper">
+
+ <div id="lucy-top" class="container_16 lucy-white_box_3d">
+
+ <div id="lucy-logo_box" class="grid_8">
+ <a href="/"><img
src="http://lucy.apache.org/images/lucy_logo_150x100.png"; alt="Apache
Lucyâ¢"></a>
+ </div> <!-- lucy-logo_box -->
+
+ <div #id="lucy-top_nav_box" class="grid_8">
+ <div id="lucy-top_nav_bar" class="container_8">
+ <ul>
+ <li><a href="http://www.apache.org/"; title="Apache Software
Foundation">Apache Software Foundation</a></li>
+ <li><a href="http://www.apache.org/licenses/";
title="License">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html";
title="Sponsorship">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html";
title="Thanks">Thanks</a></li>
+ <li><a href="http://www.apache.org/security/ "
title="Security">Security</a></li>
+ </ul>
+ </div> <!-- lucy-top_nav_bar -->
+ <p><a href="http://www.apache.org/";>Apache</a> » <a
href="/">Lucy</a> » <a
href="/docs/">Docs</a> » <a
href="/docs/test/">Test</a> » <a
href="/docs/test/Lucy/">Lucy</a> » <a
href="/docs/test/Lucy/Docs/">Docs</a> » <a
href="/docs/test/Lucy/Docs/Tutorial/">Tutorial</a></p>
+ <form name="lucy-top_search_box" id="lucy-top_search_box"
action="http://www.google.com/search"; method="get">
+ <input value="*.apache.org" name="sitesearch" type="hidden"/>
+ <input type="text" name="q" id="query" style="width:85%">
+ <input type="submit" id="submit" value="Search">
+ </form>
+ </div> <!-- lucy-top_nav_box -->
+
+ <div class="clear"></div>
+
+ </div> <!-- lucy-top -->
+
+ <div id="lucy-main_content" class="container_16 lucy-white_box_3d">
+
+ <div class="grid_4" id="lucy-left_nav_box">
+ <h6>About</h6>
+ <ul>
+ <li><a href="/">Welcome</a></li>
+ <li><a href="/clownfish.html">Clownfish</a></li>
+ <li><a href="/faq.html">FAQ</a></li>
+ <li><a href="/people.html">People</a></li>
+ </ul>
+ <h6>Resources</h6>
+ <ul>
+ <li><a href="/download.html">Download</a></li>
+ <li><a href="/mailing_lists.html">Mailing Lists</a></li>
+ <li><a href="/docs/perl/">Documentation</a></li>
+ <li><a href="http://wiki.apache.org/lucy/";>Wiki</a></li>
+ <li><a href="https://issues.apache.org/jira/browse/LUCY";>Issue
Tracker</a></li>
+ <li><a href="/version_control.html">Version Control</a></li>
+ </ul>
+ <h6>Related Projects</h6>
+ <ul>
+ <li><a href="http://lucene.apache.org/java/";>Lucene</a></li>
+ <li><a href="http://lucene.apache.org/solr/";>Solr</a></li>
+ <li><a
href="http://incubator.apache.org/lucene.net/";>Lucene.NET</a></li>
+ <li><a
href="http://lucene.apache.org/pylucene/";>PyLucene</a></li>
+ <li><a href="http://lucene.apache.org/openrelevance/";>Open
Relevance</a></li>
+ </ul>
+ </div> <!-- lucy-left_nav_box -->
+
+ <div id="lucy-main_content_box" class="grid_9">
+ <div>
+<a name='___top' class='dummyTopAnchor' ></a>
+
+<h2><a class='u'
+name="NAME"
+>NAME</a></h2>
+
+<p>Lucy::Docs::Tutorial::FieldTypeTutorial - Specify per-field properties and
behaviors.</p>
+
+<h2><a class='u'
+name="DESCRIPTION"
+>DESCRIPTION</a></h2>
+
+<p>The Schema we used in the last chapter specifies three fields:</p>
+
+<pre> my $type = Lucy::Plan::FullTextType->new(
+ analyzer => $easyanalyzer,
+ );
+ $schema->spec_field( name => 'title', type => $type
);
+ $schema->spec_field( name => 'content', type => $type
);
+ $schema->spec_field( name => 'url', type => $type
);</pre>
+
+<p>Since they are all defined as “full text” fields, they are all
searchable – including the <code>url</code> field, a dubious choice. Some
URLs contain meaningful information, but these don’t, really:</p>
+
+<pre> http://example.com/us_constitution/amend1.txt</pre>
+
+<p>We may as well not bother indexing the URL content. To achieve that we need
to assign the <code>url</code> field to a different FieldType.</p>
+
+<h3><a class='u'
+name="StringType"
+>StringType</a></h3>
+
+<p>Instead of FullTextType, we’ll use a <a
href="../../../Lucy/Plan/StringType.html" class="podlinkpod"
+>StringType</a>, which doesn’t use an Analyzer to break up text into
individual fields. Furthermore, we’ll mark this StringType as unindexed,
so that its content won’t be searchable at all.</p>
+
+<pre> my $url_type = Lucy::Plan::StringType->new( indexed => 0 );
+ $schema->spec_field( name => 'url', type => $url_type
);</pre>
+
+<p>To observe the change in behavior, try searching for
<code>us_constitution</code> both before and after changing the Schema and
re-indexing.</p>
+
+<h3><a class='u'
+name="Toggling_(8216)stored(8217)"
+>Toggling ‘stored’</a></h3>
+
+<p>For a taste of other FieldType possibilities, try turning off
<code>stored</code> for one or more fields.</p>
+
+<pre> my $content_type = Lucy::Plan::FullTextType->new(
+ analyzer => $easyanalyzer,
+ stored => 0,
+ );</pre>
+
+<p>Turning off <code>stored</code> for either <code>title</code> or
<code>url</code> mangles our results page, but since we’re not displaying
<code>content</code>, turning it off for <code>content</code> has no effect
– except on index size.</p>
+
+<h3><a class='u'
+name="Analyzers_up_next"
+>Analyzers up next</a></h3>
+
+<p>Analyzers play a crucial role in the behavior of FullTextType fields. In
our next tutorial chapter, <a
href="../../../Lucy/Docs/Tutorial/AnalysisTutorial.html" class="podlinkpod"
+>AnalysisTutorial</a>, we’ll see how changing up the Analyzer changes
search results.</p>
+
+</div>
+
+ </div> <!-- lucy-main_content_box -->
+ <div class="clear"></div>
+
+ </div> <!-- lucy-main_content -->
+
+ <div id="lucy-copyright" class="container_16">
+ <p>Copyright © 2010-2015 The Apache Software Foundation, Licensed
under the
+ <a href="http://www.apache.org/licenses/LICENSE-2.0";>Apache
License, Version 2.0</a>.
+ <br/>
+ Apache Lucy, Lucy, Apache, the Apache feather logo, and the Apache
Lucy project logo are trademarks of The
+ Apache Software Foundation. All other marks mentioned may be
trademarks or registered trademarks of their
+ respective owners.
+ </p>
+ </div> <!-- lucy-copyright -->
+
+ </div> <!-- lucy-rigid_wrapper -->
+
+ </body>
+</html>
Added:
websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/Tutorial/HighlighterTutorial.html
==============================================================================
---
websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/Tutorial/HighlighterTutorial.html
(added)
+++
websites/staging/lucy/trunk/content/docs/test/Lucy/Docs/Tutorial/HighlighterTutorial.html
Fri Feb 26 12:52:42 2016
@@ -0,0 +1,161 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
+<html lang="en">
+ <head>
+ <meta http-equiv="Content-Type" content="text/html;charset=UTF-8">
+ <title>Lucy::Docs::Tutorial::HighlighterTutorial - Apache Lucy
Documentation</title>
+ <link rel="stylesheet" type="text/css" media="screen"
href="http://lucy.apache.org/css/lucy.css";>
+ </head>
+
+ <body>
+
+ <div id="lucy-rigid_wrapper">
+
+ <div id="lucy-top" class="container_16 lucy-white_box_3d">
+
+ <div id="lucy-logo_box" class="grid_8">
+ <a href="/"><img
src="http://lucy.apache.org/images/lucy_logo_150x100.png"; alt="Apache
Lucyâ¢"></a>
+ </div> <!-- lucy-logo_box -->
+
+ <div #id="lucy-top_nav_box" class="grid_8">
+ <div id="lucy-top_nav_bar" class="container_8">
+ <ul>
+ <li><a href="http://www.apache.org/"; title="Apache Software
Foundation">Apache Software Foundation</a></li>
+ <li><a href="http://www.apache.org/licenses/";
title="License">License</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html";
title="Sponsorship">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html";
title="Thanks">Thanks</a></li>
+ <li><a href="http://www.apache.org/security/ "
title="Security">Security</a></li>
+ </ul>
+ </div> <!-- lucy-top_nav_bar -->
+ <p><a href="http://www.apache.org/";>Apache</a> » <a
href="/">Lucy</a> » <a
href="/docs/">Docs</a> » <a
href="/docs/test/">Test</a> » <a
href="/docs/test/Lucy/">Lucy</a> » <a
href="/docs/test/Lucy/Docs/">Docs</a> » <a
href="/docs/test/Lucy/Docs/Tutorial/">Tutorial</a></p>
+ <form name="lucy-top_search_box" id="lucy-top_search_box"
action="http://www.google.com/search"; method="get">
+ <input value="*.apache.org" name="sitesearch" type="hidden"/>
+ <input type="text" name="q" id="query" style="width:85%">
+ <input type="submit" id="submit" value="Search">
+ </form>
+ </div> <!-- lucy-top_nav_box -->
+
+ <div class="clear"></div>
+
+ </div> <!-- lucy-top -->
+
+ <div id="lucy-main_content" class="container_16 lucy-white_box_3d">
+
+ <div class="grid_4" id="lucy-left_nav_box">
+ <h6>About</h6>
+ <ul>
+ <li><a href="/">Welcome</a></li>
+ <li><a href="/clownfish.html">Clownfish</a></li>
+ <li><a href="/faq.html">FAQ</a></li>
+ <li><a href="/people.html">People</a></li>
+ </ul>
+ <h6>Resources</h6>
+ <ul>
+ <li><a href="/download.html">Download</a></li>
+ <li><a href="/mailing_lists.html">Mailing Lists</a></li>
+ <li><a href="/docs/perl/">Documentation</a></li>
+ <li><a href="http://wiki.apache.org/lucy/";>Wiki</a></li>
+ <li><a href="https://issues.apache.org/jira/browse/LUCY";>Issue
Tracker</a></li>
+ <li><a href="/version_control.html">Version Control</a></li>
+ </ul>
+ <h6>Related Projects</h6>
+ <ul>
+ <li><a href="http://lucene.apache.org/java/";>Lucene</a></li>
+ <li><a href="http://lucene.apache.org/solr/";>Solr</a></li>
+ <li><a
href="http://incubator.apache.org/lucene.net/";>Lucene.NET</a></li>
+ <li><a
href="http://lucene.apache.org/pylucene/";>PyLucene</a></li>
+ <li><a href="http://lucene.apache.org/openrelevance/";>Open
Relevance</a></li>
+ </ul>
+ </div> <!-- lucy-left_nav_box -->
+
+ <div id="lucy-main_content_box" class="grid_9">
+ <div>
+<a name='___top' class='dummyTopAnchor' ></a>
+
+<h2><a class='u'
+name="NAME"
+>NAME</a></h2>
+
+<p>Lucy::Docs::Tutorial::HighlighterTutorial - Augment search results with
highlighted excerpts.</p>
+
+<h2><a class='u'
+name="DESCRIPTION"
+>DESCRIPTION</a></h2>
+
+<p>Adding relevant excerpts with highlighted search terms to your search
results display makes it much easier for end users to scan the page and assess
which hits look promising,
+dramatically improving their search experience.</p>
+
+<h3><a class='u'
+name="Adaptations_to_indexer.pl"
+>Adaptations to indexer.pl</a></h3>
+
+<p><a href="../../../Lucy/Highlight/Highlighter.html" class="podlinkpod"
+>Highlighter</a> uses information generated at index time.
+To save resources,
+highlighting is disabled by default and must be turned on for individual
fields.</p>
+
+<pre> my $highlightable = Lucy::Plan::FullTextType->new(
+ analyzer => $easyanalyzer,
+ highlightable => 1,
+ );
+ $schema->spec_field( name => 'content', type =>
$highlightable );</pre>
+
+<h3><a class='u'
+name="Adaptations_to_search.cgi"
+>Adaptations to search.cgi</a></h3>
+
+<p>To add highlighting and excerpting to the search.cgi sample app, create a
<code>$highlighter</code> object outside the hits iterating loop…</p>
+
+<pre> my $highlighter = Lucy::Highlight::Highlighter->new(
+ searcher => $searcher,
+ query => $q,
+ field => 'content'
+ );</pre>
+
+<p>… then modify the loop and the per-hit display to generate and
include the excerpt.</p>
+
+<pre> # Create result list.
+ my $report = '';
+ while ( my $hit = $hits->next ) {
+ my $score = sprintf( "%0.3f", $hit->get_score );
+ my $excerpt = $highlighter->create_excerpt($hit);
+ $report .= qq|
+ <p>
+ <a
href="$hit->{url}"><strong>$hit->{title}</strong></a>
+ <em>$score</em>
+ <br />
+ $excerpt
+ <br />
+ <span
class="excerptURL">$hit->{url}</span>
+ </p>
+ |;
+ }</pre>
+
+<h3><a class='u'
+name="Next_chapter:_Query_objects"
+>Next chapter: Query objects</a></h3>
+
+<p>Our next tutorial chapter, <a
href="../../../Lucy/Docs/Tutorial/QueryObjectsTutorial.html" class="podlinkpod"
+>QueryObjectsTutorial</a>, illustrates how to build an “advanced
search” interface using <a href="../../../Lucy/Search/Query.html"
class="podlinkpod"
+>Query</a> objects instead of query strings.</p>
+
+</div>
+
+ </div> <!-- lucy-main_content_box -->
+ <div class="clear"></div>
+
+ </div> <!-- lucy-main_content -->
+
+ <div id="lucy-copyright" class="container_16">
+ <p>Copyright © 2010-2015 The Apache Software Foundation, Licensed
under the
+ <a href="http://www.apache.org/licenses/LICENSE-2.0";>Apache
License, Version 2.0</a>.
+ <br/>
+ Apache Lucy, Lucy, Apache, the Apache feather logo, and the Apache
Lucy project logo are trademarks of The
+ Apache Software Foundation. All other marks mentioned may be
trademarks or registered trademarks of their
+ respective owners.
+ </p>
+ </div> <!-- lucy-copyright -->
+
+ </div> <!-- lucy-rigid_wrapper -->
+
+ </body>
+</html>
![http://lucy.apache.org/images/lucy_logo_150x100.png"](http://lucy.apache.org/images/lucy_logo_150x100.png"){kind=link}