[
https://issues.apache.org/jira/browse/LUCENE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17314123#comment-17314123
]
Julie Tibshirani commented on LUCENE-9648:
------------------------------------------
Thanks [~gf2121] for the interesting ticket and sorry for the slow reply!
In general it seems hard to effectively bound the range of query values. The
benchmark you posted randomly generates a set of query values from the range
[0, 1_000_000_000]. As an experiment I drew 1000 random values and looked at
their range: in a typical example we get min: 128_082, max: 999_833_281. So the
range covers 0.999% of the possible values. Even with only 2 random query
values I think the range would cover 1/3 of the possible values in expectation.
So it seems the speed-up in the benchmark must be from something else -- maybe
we just check against the query values more efficiently? It'd be interesting to
dig into this to understand what makes it faster.
I'm also curious if you have a specific use case in mind. My assumption was
that the query values don't tend to cluster together, but maybe they do in your
case?
> Add a numbers query in sandbox that takes advantage of index sorting.
> ---------------------------------------------------------------------
>
> Key: LUCENE-9648
> URL: https://issues.apache.org/jira/browse/LUCENE-9648
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/sandbox
> Reporter: Feng Guo
> Priority: Minor
> Time Spent: 40m
> Remaining Estimate: 0h
>
> LUCENE-6539 introduced DocValuesNumbersQuery to benefit some special cases
> (many terms/numbers and fewish matching hits). Inspired by LUCENE-7714, maybe
> we can speed it up when meeting index sort since we can do a binary search to
> bound the docvalues before iter it.
> Here are some simple benchmarks:
> *Based on 50,000,000 doc, each long in query can hit at least one doc:*
> |LongsCount|PointInSetQuery(s/op)|IndexSortDocValuesNumbersQuery(s/op)|DocValuesNumbersQuery(s/op)|
> |1000|0.144 ± 0.001|0.428 ± 0.008|1.159 ± 0.031|
> |5000|0.404 ± 0.003|0.424 ± 0.008|1.343 ± 0.022|
> |10000|0.661 ± 0.012|0.436 ± 0.008|1.372 ± 0.011|
> |50000|1.319 ± 0.011|0.450 ± 0.005|1.173 ± 0.017|
> |100000|1.328 ± 0.007|0.459 ± 0.003|1.160 ± 0.098|
> *Based on 100,000,000 doc, each long in query can hit at least one doc:*
> |LongsCount|PointInSetQuery(s/op)|IndexSortDocValuesNumbersQuery(s/op)|DocValuesNumbersQuery(s/op)|
> |1000|0.189 ± 0.003|0.860 ± 0.028|2.408 ± 0.087|
> |5000|0.715 ± 0.010|0.879 ± 0.010|2.642 ± 0.105|
> |10000|1.075 ± 0.034|0.859 ± 0.010|2.629 ± 0.077|
> |50000|2.652 ± 0.114|0.865 ± 0.009|2.737 ± 0.099|
> |100000|2.905 ± 0.027|0.881 ± 0.005|2.820 ± 0.474|
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
