Hi John,
I hope you are doing well, i didn't have a chance to contribute to Fastbit for
a while now.
I came across an interesting bug recently where i had queries failing with:
Warning -- query[96w0-4EVxo----8F]::getRIDs -- got 2620 row IDs from partition
data_property_flat, expected 1964
After some investigation, i think i have isolated the source of the problem.
It seems that concurrent invocations of ibis::direkte::evaluate(const
ibis::qContinuousRange& expr, ibis::bitvector& lower) (used by
ibis::category::stringSearch(const char* str, ibis::bitvector& hits)) and
ibis::direkte::estimate(const ibis::qContinuousRange& expr) (used by
ibis::category::stringSearch(const char* str)) happens when you are running
concurrent queries on the same columns, and it becomes problematic. The issues
seems to be related to the fact that ibis::direkte::estimate does use the
ibis::bitvector::cnt() method of bit vector, which modifies its nset field in a
non thread-safe way and is not protected by the column lock (as it is a read
lock once the index is loaded).
uint32_t ibis::direkte::estimate(const ibis::qContinuousRange& expr) const {
uint32_t ib, ie, cnt;
locate(expr, ib, ie);
activate(ib, ie);
cnt = 0;
for (uint32_t j = ib; j < ie; ++ j)
if (bits[j])
cnt += bits[j]->cnt();
return cnt;
} // ibis::direkte::estimate
Therefore, it can happen that ibis::direkte::evaluate copies (in sumBins) a bit
vector which nset field is incorrect (because it is being computed by a
concurrent call to ibis::direkte::estimate which is in a the cnt method of the
same bitvector).
As for the fix, i don't know what would be the best. I'm tempted to think that
computing bitvector count when activating the bin is the "cleanest" solution as
it is likely than count will be used at some point. On the other hand, maybe
simply replacing bits[j]->cnt() by bits[j]->sloppyCnt() would work just fine as
well at the expense of the precision of the estimation.
What do you think ?
Thanks,
Dominique Prunier
APG Architect
[Logo-W4N-100dpi]
4388, rue Saint-Denis
Bureau 309
Montreal (Quebec) H2J 2L1
Tel. +1 514-985-6110
Fax +1 514-842-3989
[email protected]<mailto:[email protected]>
www.watch4net.com<http://www.watch4net.com/>
This message is for the designated recipient only and may contain privileged,
proprietary, or otherwise private information. If you have received it in
error, please notify the sender immediately and delete the original. Any other
use of this electronic mail by you is prohibited.
Ce message est pour le récipiendaire désigné seulement et peut contenir des
informations privilégiées, propriétaires ou autrement privées. Si vous l'avez
reçu par erreur, S.V.P. avisez l'expéditeur immédiatement et effacez
l'original. Toute autre utilisation de ce courrier électronique par vous est
prohibée.
_______________________________________________
FastBit-users mailing list
[email protected]
https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
