[
https://issues.apache.org/jira/browse/LUCENE-1812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12868719#action_12868719
]
Robert Muir commented on LUCENE-1812:
-------------------------------------
Hi Andrzej, thanks for updating the patch.
I am curious about package organization here, do you anticipate adding some
additional pruning functionality in the future that would be different than an
index modification tool?
I only ask, because looking at reorganizing our contrib area (LUCENE-2323),
I've often thought that perhaps we need a "contrib/index" for all the
index-related tools, instead of having various ones in "miscellaneous", and I
wonder what your opinions are on that.
In any event we could always reorganize this after this issue is resolved if
thats the best thing to do, and it could temporarily be contrib/pruning, its
just svn moves.
> Static index pruning by in-document term frequency (Carmel pruning)
> -------------------------------------------------------------------
>
> Key: LUCENE-1812
> URL: https://issues.apache.org/jira/browse/LUCENE-1812
> Project: Lucene - Java
> Issue Type: New Feature
> Components: contrib/*
> Affects Versions: 2.9, 3.1
> Reporter: Andrzej Bialecki
> Attachments: pruning.patch, pruning.patch, pruning.patch
>
>
> This module provides tools to produce a subset of input indexes by removing
> postings data for those terms where their in-document frequency is below a
> specified threshold. The net effect of this processing is a much smaller
> index that for common types of queries returns nearly identical top-N results
> as compared with the original index, but with increased performance.
> Optionally, stored values and term vectors can also be removed. This
> functionality is largely independent, so it can be used without term pruning
> (when term freq. threshold is set to 1).
> As the threshold value increases, the total size of the index decreases,
> search performance increases, and recall decreases (i.e. search quality
> deteriorates). NOTE: especially phrase recall deteriorates significantly at
> higher threshold values.
> Primary purpose of this class is to produce small first-tier indexes that fit
> completely in RAM, and store these indexes using
> IndexWriter.addIndexes(IndexReader[]). Usually the performance of this class
> will not be sufficient to use the resulting index view for on-the-fly pruning
> and searching.
> NOTE: If the input index is optimized (i.e. doesn't contain deletions) then
> the index produced via IndexWriter.addIndexes(IndexReader[]) will preserve
> internal document id-s so that they are in sync with the original index. This
> means that all other auxiliary information not necessary for first-tier
> processing, such as some stored fields, can also be removed, to be quickly
> retrieved on-demand from the original index using the same internal document
> id.
> Threshold values can be specified globally (for terms in all fields) using
> defaultThreshold parameter, and can be overriden using per-field or per-term
> values supplied in a thresholds map. Keys in this map are either field names,
> or terms in field:text format. The precedence of these values is the
> following: first a per-term threshold is used if present, then per-field
> threshold if present, and finally the default threshold.
> A command-line tool (PruningTool) is provided for convenience. At this moment
> it doesn't support all functionality available through API.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
