[
https://issues.apache.org/jira/browse/LUCENE-4600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless updated LUCENE-4600:
---------------------------------------
Attachment: LUCENE-4600.patch
New patch, adding a hacked up CachedCountingFacetsCollector.
All it does is first pre-load all payloads into a PackedBytes (just like
DocValues), and then during aggregation, instead of pulling the byte[] from
payloads it pulls it from this RAM cache.
This results in an unexpectedly big speedup:
{noformat}
Task QPS base StdDev QPS comp StdDev
Pct diff
HighTerm 0.53 (0.9%) 1.00 (2.5%)
87.3% ( 83% - 91%)
LowTerm 7.59 (0.6%) 26.75 (12.9%)
252.6% ( 237% - 267%)
MedTerm 3.35 (0.7%) 12.71 (9.0%)
279.8% ( 268% - 291%)
{noformat}
The only "real" difference is that I'm pulling the byte[] from RAM instead of
from payloads, ie I still pay the vInt+dgap decode cost per hit ... so it's
surprising payloads add THAT MUCH overhead? (The test was "hot" so payloads
were coming from OS's IO cache via MMapDir).
I think the reason why HighTerm sees the least gains is because .advance is
much less costly for it, since often the target is in the already-loaded block.
I had separately previously tested the existing int[][][] cache
(CategoryListCache) but it had smaller gains than this (73% for MedTerm), and
it required more RAM (1.9 GB vs 377 RAM for this patch).
Net/net I think we should offer an easy-to-use DV-backed facets impl...
> Facets should aggregate during collection, not at the end
> ---------------------------------------------------------
>
> Key: LUCENE-4600
> URL: https://issues.apache.org/jira/browse/LUCENE-4600
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Michael McCandless
> Attachments: LUCENE-4600.patch, LUCENE-4600.patch
>
>
> Today the facet module simply gathers all hits (as a bitset, optionally with
> a float[] to hold scores as well, if you will aggregate them) during
> collection, and then at the end when you call getFacetsResults(), it makes a
> 2nd pass over all those hits doing the actual aggregation.
> We should investigate just aggregating as we collect instead, so we don't
> have to tie up transient RAM (fairly small for the bit set but possibly big
> for the float[]).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
