[
https://issues.apache.org/jira/browse/LUCENE-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15304081#comment-15304081
]
Michael McCandless commented on LUCENE-7304:
--------------------------------------------
This is a neat idea!
Does this approach work out to less than one bit per doc? I guess it must be
more than that (our doc values compression isn't THAT good yet), but by
switching to doc values, even though we need more RAM, it moves off-heap right,
so the OS is managing keeping those bytes hot instead.
> Doc values based block join implementation
> ------------------------------------------
>
> Key: LUCENE-7304
> URL: https://issues.apache.org/jira/browse/LUCENE-7304
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Martijn van Groningen
> Priority: Minor
> Attachments: LUCENE_7304.patch
>
>
> At query time the block join relies on a bitset for finding the previous
> parent doc during advancing the doc id iterator. On large indices these
> bitsets can consume large amounts of jvm heap space. Also typically due the
> nature how these bitsets are set, the 'FixedBitSet' implementation is used.
> The idea I had was to replace the bitset usage by a numeric doc values field
> that stores offsets. Each child doc stores how many docids it is from its
> parent doc and each parent stores how many docids it is apart from its first
> child. At query time this information can be used to perform the block join.
> I think another benefit of this approach is that external tools can now
> easily determine if a doc is part of a block of documents and perhaps this
> also helps index time sorting?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
