jasperjiaguo edited a comment on issue #5596:
URL:
https://github.com/apache/incubator-pinot/issues/5596#issuecomment-646910972
1. I have considered 2 extreme cases:
a. We have a long consecutive rows of hit in each column. In this case, yes
we can loose locality because we are essentially doing row by row scanning.
However, now the condition we are using to switch between the two approaches is
`(numIndexBasedDocIdIterators > 0 && numScanBasedDocIdIterators > 0) ||
numIndexBasedDocIdIterators > 1` which is essentially using the "number of
bitmaps we have in and-predicate >= 2". I think "the percentage of bits set in
bitmap" can be a better criteria and is equally easy to get.
b. Say A is a bitmapIterator and B is a MVScanDocIdIterator. If it's in
this shape (1 is hit and 0 is miss):
Then indeed by only using approach in `AndDocIdIterator.next()` is
insufficient; so I have proposed a solution for this in my branch by passing
the bitmapIterator to `SVScanDocIdIterator.advance()` and
`MVScanDocIdIterator.advance()`.
2. Agreed. I'm setting up some experiments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
