sherman opened a new issue, #12076:
URL: https://github.com/apache/lucene/issues/12076
Hi!
AFAICS, the family of multiple ranges queries (MultiRangeQuery) were added
to the sandbox module recently.
In our index we have a few indexed fields with type long (aren't ranges,
it's just long ids, dimension = 1).
We have about 16% of search queries which include a predicate like: where
field in (id1, id2, id3), and the length of argument list is over 40 elements.
Currently, our parser maps such predicates as a list of separate
PointRangeQuery. When I found MultiRangeQuery, I immediately started the tests
to see the way how to optimize such queries.
The first results were impressive, because I saw the difference in the
amount of data were read from mapped files.
The list of PointRangeQuery
```
16:39:23.587 [ProfilingSearchThread] INFO
com.ozon.grpc.BlockingRequestExecutor - Query statistics [null]. Seeks:
[2363], Bytes were read: [78198], Shorts were read: [1266], Ints were read:
[48], Longs were read: [9821], Floats were read: [0], Total bytes were read:
[159490]
16:39:27.030 [ProfilingSearchThread] INFO
com.ozon.grpc.BlockingRequestExecutor - Query statistics [null]. Seeks:
[2363], Bytes were read: [78198], Shorts were read: [1266], Ints were read:
[48], Longs were read: [9821], Floats were read: [0], Total bytes were read:
[159490]
16:39:35.890 [ProfilingSearchThread] INFO
com.ozon.grpc.BlockingRequestExecutor - Query statistics [null]. Seeks:
[2363], Bytes were read: [78198], Shorts were read: [1266], Ints were read:
[48], Longs were read: [9821], Floats were read: [0], Total bytes were read:
[159490]
```
A single MultiRangeQuery
```
16:49:02.372 [ProfilingSearchThread] INFO
com.ozon.grpc.BlockingRequestExecutor - Query statistics [null]. Seeks: [505],
Bytes were read: [61917], Shorts were read: [1266], Ints were read: [48],
Longs were read: [9053], Floats were read: [0], Total bytes were read: [137065]
16:49:03.794 [ProfilingSearchThread] INFO
com.ozon.grpc.BlockingRequestExecutor - Query statistics [null]. Seeks: [505],
Bytes were read: [61917], Shorts were read: [1266], Ints were read: [48],
Longs were read: [9053], Floats were read: [0], Total bytes were read: [137065]
16:49:05.582 [ProfilingSearchThread] INFO
com.ozon.grpc.BlockingRequestExecutor - Query statistics [null]. Seeks: [505],
Bytes were read: [61917], Shorts were read: [1266], Ints were read: [48],
Longs were read: [9053], Floats were read: [0], Total bytes were read: [137065]
```
The numbers above said in case of MultiRangeQuery we have a reduction of
seek and read long operations, as expected.
Surprisingly, MultiRangeQuery is slower in terms of wall clock time.
I compared the timings for the same queries and I see the significant
slowdown in the build score stage.
The list of PointRangeQuery
```
"timings": {
"score": "50297",
"build_scorer_count": "2",
"match_count": "29",
"create_weight": "562343",
"next_doc": "252004",
"match": "12136",
"create_weight_count": "1",
"next_doc_count": "30",
"score_count": "29",
"build_scorer": "2884027",
"advance": "0",
"advance_count": "0"
},
"statistics": {
"match_avg": "418",
"next_doc_avg": "8400",
"build_scorer_avg": "1442013",
"score_avg": "1734"
},
"total": "3760898"
}
```
A single MultiRangeQuery
```
timings": {
"score": "7793",
"build_scorer_count": "2",
"match_count": "0",
"create_weight": "7762",
"next_doc": "12713",
"match": "0",
"create_weight_count": "1",
"next_doc_count": "30",
"score_count": "29",
"build_scorer": "5283654",
"advance": "0",
"advance_count": "0"
},
"statistics": {
"next_doc_avg": "423",
"build_scorer_avg": "2641827",
"score_avg": "268"
},
"total": "5311984"
}
```
As you can see, build_scorer is pretty "fat" in MultiRangeQuery. I guess,
the most of time was spent for a building an interval tree in the method:
`MultiRangeQuery.scorerSupplier()`
Is it expected behaviour?
And what the case of usage for MultiRangeQuery (a number of ranges reduction
through the merges is probably one of, but it's not mine)?
Is any known problems in the interval tree implementation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
