rohangarg commented on code in PR #13133:
URL: https://github.com/apache/druid/pull/13133#discussion_r989861633
##########
processing/src/main/java/org/apache/druid/segment/serde/DictionaryEncodedStringIndexSupplier.java:
##########
@@ -280,15 +287,35 @@ public ImmutableBitmap next()
private void findNext()
{
- while (next < 0 && iterator.hasNext()) {
- ByteBuffer nextValue = iterator.next();
- next = dictionary.indexOf(nextValue);
-
- if (next == -dictionarySize - 1) {
- // nextValue is past the end of the dictionary.
- // Note: we can rely on indexOf returning (-(insertion
point) - 1), even though Indexed doesn't
- // guarantee it, because "dictionary" comes from
GenericIndexed singleThreaded().
- break;
+ // if the size of in-filter values is less than the threshold
percentage of dictionary size, then use binary search
+ // based lookup per value. The algorithm works well for smaller
number of values.
+ if (size < SORTED_MERGE_RATIO_THRESHOLD * dictionary.size()) {
Review Comment:
Makes sense, done. Had done it earlier like that, but changed it last moment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
