[
https://issues.apache.org/jira/browse/LUCENE-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615357#action_12615357
]
Eks Dev commented on LUCENE-1340:
---------------------------------
Great, it is already more than I expected, even indexing is going to be
somewhat faster.
I have tried your patch on smallish index with 8Mio documents and it worked on
our regression test without problems.
it worked fine with and without omitTf(true), no performance drop or bad
surprises when we do not use it. Tomorrow is scheduled real test with
production data, around 80Mio very small documents, with some very extensive
tests.... I will report back.
"The one place I know of that will still waste bytes is the term dict
(TermInfo): it stores a long proxPointer on disk (in .tii,.tis) and
also in memory because we load *.tii into RAM.... "
About this one, it would be nice not to store this as well, but I think the
pointers are already reduced to one byte, as they are 0 for these cases (are
they,?) So we have this benefit without expecting it :)
And yes, more "column stride" is great, if you followed my comments on
LUCENE-1278, that would mean we could easily "inline" very short postings into
term dict (here I expect huge performance benefit, as skip() on another large
file is going to be saved independent from omitTf(true)), without increase in
size (or minimal) of tii (no locality penalty) If we follow Zipfian
distribution, there is *a lot* of terms with postings shorter than e.g. 16 ...
Thanks again for your support, without you this patch would be just another
nice idea :)
> Make it posible not to include TF information in index
> ------------------------------------------------------
>
> Key: LUCENE-1340
> URL: https://issues.apache.org/jira/browse/LUCENE-1340
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Index
> Reporter: Eks Dev
> Priority: Minor
> Attachments: LUCENE-1340.patch, LUCENE-1340.patch, LUCENE-1340.patch,
> LUCENE-1340.patch, LUCENE-1340.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Term Frequency is typically not needed for all fields, some CPU (reading one
> VInt less and one X>>>1...) and IO can be spared by making pure boolen fields
> possible in Lucene. This topic has already been discussed and accepted as a
> part of Flexible Indexing... This issue tries to push things a bit faster
> forward as I have some concrete customer demands.
> benefits can be expected for fields that are typical candidates for Filters,
> enumerations, user rights, IDs or very short "texts", phone numbers, zip
> codes, names...
> Status: just passed standard test (compatibility), commited for early review,
> I have not tried new feature, missing some asserts and one two unit tests
> Complexity: simpler than expected
> can be used via omitTf() (who used omitNorms() will know where to find it :)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
