Hey John, Yes. For this particular bug, initializing nset in ibis::bitvector::bitvector(const array_t<ibis::bitvector::word_t>& arr) would work perfectly, as it is the constructor invoked by the index activate method.
However, keeping nset always up-to-date seems to be a bit more challenging. It affects more than the few constructors of bitvector but anywhere in the code where nset is resetted to 0 for non obvious cases (like clear for example). As an immediate fix for my issue, i think i'll call do_cnt at the end of the ibis::bitvector::bitvector(const array_t<ibis::bitvector::word_t>& arr) constructor, but for the more permanent fix, i'll look at the code you end-up with. Thanks, -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of K. John Wu Sent: Tuesday, August 05, 2014 2:35 PM To: FastBit Users Subject: Re: [FastBit-users] Concurrency issue with bitvector count calculation in direkte Hi, Dominique, Thanks for tracking this problem down. In this case, I prefer the solution of computing nset when a bitvector is initialized. Let me try to get this done in the next few days. Will likely to surround the code with some sort of conditional compilation macro for the moment. John On 8/5/14 9:40 AM, Prunier, Dominique wrote: > Hi John, > > > > I hope you are doing well, i didn’t have a chance to contribute to > Fastbit for a while now. > > > > I came across an interesting bug recently where i had queries failing > with: > > > > Warning -- query[96w0-4EVxo----8F]::getRIDs -- got 2620 row IDs from > partition data_property_flat, expected 1964 > > > > After some investigation, i think i have isolated the source of the > problem. > > > > It seems that concurrent invocations of > *ibis::direkte::evaluate*(*const*ibis::qContinuousRange& > expr,ibis::bitvector& lower)(used by > *ibis::category::stringSearch*(*const**char** str,ibis::bitvector& > hits)) and *ibis::direkte::estimate*(*const*ibis::qContinuousRange& > expr) (used by *ibis::category::stringSearch*(*const**char** str)) > happens when you are running concurrent queries on the same columns, > and it becomes problematic. The issues seems to be related to the fact > that *ibis::direkte::estimate* does use the > *ibis::bitvector::cnt*()method of bit vector, which modifies its nset > field in a non thread-safe way and is not protected by the column lock > (as it is a read lock once the index is loaded). > > > > uint32_t *ibis::direkte::estimate*(*const*ibis::qContinuousRange& > expr) *const*{ > > uint32_t ib, ie, cnt; > > locate(expr, ib, ie); > > activate(ib, ie); > > cnt = 0; > > *for*(uint32_t j = ib; j < ie; ++ j) > > *if*(bits[j]) > > cnt += bits[j]->cnt(); > > *return*cnt; > > } // ibis::direkte::estimate > > > > Therefore, it can happen that *ibis::direkte::evaluate* copies (in > sumBins) a bit vector which nset field is incorrect (because it is > being computed by a concurrent call to *ibis::direkte::estimate* which > is in a the cnt method of the same bitvector). > > > > As for the fix, i don’t know what would be the best. I’m tempted to > think that computing bitvector count when activating the bin is the > “cleanest” solution as it is likely than count will be used at some > point. On the other hand, maybe simply replacing bits[j]->cnt() by > bits[j]->sloppyCnt() would work just fine as well at the expense of > the precision of the estimation. > > > > What do you think ? > > > > Thanks, > > > > */Dominique Prunier/**//* > > APG Architect > > Logo-W4N-100dpi > > 4388, rue Saint-Denis > > Bureau 309 > > Montreal (Quebec) H2J 2L1 > > Tel. +1 514-985-6110 > > Fax +1 514-842-3989 > > [email protected] <mailto:[email protected]> > > www.watch4net.com <http://www.watch4net.com/> > > / / > > /This message is for the designated recipient only and may contain > privileged, proprietary, or otherwise private information. If you have > received it in error, please notify the sender immediately and delete > the original. Any other use of this electronic mail by you is prohibited. > > //Ce message est pour le récipiendaire désigné seulement et peut > contenir des informations privilégiées, propriétaires ou autrement > privées. Si vous l'avez reçu par erreur, S.V.P. avisez l'expéditeur > immédiatement et effacez l'original. Toute autre utilisation de ce > courrier électronique par vous est prohibée./// > > > > > > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
