Hi, Dominique, Please check out SVN Revision 742 and see if it address the problem you are experiencing.
Thanks. John On 8/5/14 1:52 PM, Prunier, Dominique wrote: > Hey John, > > Yes. For this particular bug, initializing nset in > ibis::bitvector::bitvector(const array_t<ibis::bitvector::word_t>& arr) would > work perfectly, as it is the constructor invoked by the index activate method. > > However, keeping nset always up-to-date seems to be a bit more challenging. > It affects more than the few constructors of bitvector but anywhere in the > code where nset is resetted to 0 for non obvious cases (like clear for > example). > > As an immediate fix for my issue, i think i'll call do_cnt at the end of the > ibis::bitvector::bitvector(const array_t<ibis::bitvector::word_t>& arr) > constructor, but for the more permanent fix, i'll look at the code you end-up > with. > > Thanks, > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of K. John Wu > Sent: Tuesday, August 05, 2014 2:35 PM > To: FastBit Users > Subject: Re: [FastBit-users] Concurrency issue with bitvector count > calculation in direkte > > Hi, Dominique, > > Thanks for tracking this problem down. In this case, I prefer the > solution of computing nset when a bitvector is initialized. Let me > try to get this done in the next few days. Will likely to surround > the code with some sort of conditional compilation macro for the moment. > > John > > > On 8/5/14 9:40 AM, Prunier, Dominique wrote: >> Hi John, >> >> >> >> I hope you are doing well, i didn’t have a chance to contribute to >> Fastbit for a while now. >> >> >> >> I came across an interesting bug recently where i had queries failing >> with: >> >> >> >> Warning -- query[96w0-4EVxo----8F]::getRIDs -- got 2620 row IDs from >> partition data_property_flat, expected 1964 >> >> >> >> After some investigation, i think i have isolated the source of the >> problem. >> >> >> >> It seems that concurrent invocations of >> *ibis::direkte::evaluate*(*const*ibis::qContinuousRange& >> expr,ibis::bitvector& lower)(used by >> *ibis::category::stringSearch*(*const**char** str,ibis::bitvector& >> hits)) and *ibis::direkte::estimate*(*const*ibis::qContinuousRange& >> expr) (used by *ibis::category::stringSearch*(*const**char** str)) >> happens when you are running concurrent queries on the same columns, >> and it becomes problematic. The issues seems to be related to the fact >> that *ibis::direkte::estimate* does use the >> *ibis::bitvector::cnt*()method of bit vector, which modifies its nset >> field in a non thread-safe way and is not protected by the column lock >> (as it is a read lock once the index is loaded). >> >> >> >> uint32_t *ibis::direkte::estimate*(*const*ibis::qContinuousRange& >> expr) *const*{ >> >> uint32_t ib, ie, cnt; >> >> locate(expr, ib, ie); >> >> activate(ib, ie); >> >> cnt = 0; >> >> *for*(uint32_t j = ib; j < ie; ++ j) >> >> *if*(bits[j]) >> >> cnt += bits[j]->cnt(); >> >> *return*cnt; >> >> } // ibis::direkte::estimate >> >> >> >> Therefore, it can happen that *ibis::direkte::evaluate* copies (in >> sumBins) a bit vector which nset field is incorrect (because it is >> being computed by a concurrent call to *ibis::direkte::estimate* which >> is in a the cnt method of the same bitvector). >> >> >> >> As for the fix, i don’t know what would be the best. I’m tempted to >> think that computing bitvector count when activating the bin is the >> “cleanest” solution as it is likely than count will be used at some >> point. On the other hand, maybe simply replacing bits[j]->cnt() by >> bits[j]->sloppyCnt() would work just fine as well at the expense of >> the precision of the estimation. >> >> >> >> What do you think ? >> >> >> >> Thanks, >> >> >> >> */Dominique Prunier/**//* >> >> APG Architect >> >> Logo-W4N-100dpi >> >> 4388, rue Saint-Denis >> >> Bureau 309 >> >> Montreal (Quebec) H2J 2L1 >> >> Tel. +1 514-985-6110 >> >> Fax +1 514-842-3989 >> >> [email protected] <mailto:[email protected]> >> >> www.watch4net.com <http://www.watch4net.com/> >> >> / / >> >> /This message is for the designated recipient only and may contain >> privileged, proprietary, or otherwise private information. If you have >> received it in error, please notify the sender immediately and delete >> the original. Any other use of this electronic mail by you is prohibited. >> >> //Ce message est pour le récipiendaire désigné seulement et peut >> contenir des informations privilégiées, propriétaires ou autrement >> privées. Si vous l'avez reçu par erreur, S.V.P. avisez l'expéditeur >> immédiatement et effacez l'original. Toute autre utilisation de ce >> courrier électronique par vous est prohibée./// >> >> >> >> >> >> _______________________________________________ >> FastBit-users mailing list >> [email protected] >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >> > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
