Hi, Dominique,
Thanks for tracking this problem down. In this case, I prefer the
solution of computing nset when a bitvector is initialized. Let me
try to get this done in the next few days. Will likely to surround
the code with some sort of conditional compilation macro for the moment.
John
On 8/5/14 9:40 AM, Prunier, Dominique wrote:
> Hi John,
>
>
>
> I hope you are doing well, i didn’t have a chance to contribute to
> Fastbit for a while now.
>
>
>
> I came across an interesting bug recently where i had queries failing
> with:
>
>
>
> Warning -- query[96w0-4EVxo----8F]::getRIDs -- got 2620 row IDs from
> partition data_property_flat, expected 1964
>
>
>
> After some investigation, i think i have isolated the source of the
> problem.
>
>
>
> It seems that concurrent invocations of
> *ibis::direkte::evaluate*(*const*ibis::qContinuousRange&
> expr,ibis::bitvector& lower)(used by
> *ibis::category::stringSearch*(*const**char** str,ibis::bitvector&
> hits)) and *ibis::direkte::estimate*(*const*ibis::qContinuousRange&
> expr) (used by *ibis::category::stringSearch*(*const**char** str))
> happens when you are running concurrent queries on the same columns,
> and it becomes problematic. The issues seems to be related to the fact
> that *ibis::direkte::estimate* does use the
> *ibis::bitvector::cnt*()method of bit vector, which modifies its nset
> field in a non thread-safe way and is not protected by the column lock
> (as it is a read lock once the index is loaded).
>
>
>
> uint32_t *ibis::direkte::estimate*(*const*ibis::qContinuousRange&
> expr) *const*{
>
> uint32_t ib, ie, cnt;
>
> locate(expr, ib, ie);
>
> activate(ib, ie);
>
> cnt = 0;
>
> *for*(uint32_t j = ib; j < ie; ++ j)
>
> *if*(bits[j])
>
> cnt += bits[j]->cnt();
>
> *return*cnt;
>
> } // ibis::direkte::estimate
>
>
>
> Therefore, it can happen that *ibis::direkte::evaluate* copies (in
> sumBins) a bit vector which nset field is incorrect (because it is
> being computed by a concurrent call to *ibis::direkte::estimate* which
> is in a the cnt method of the same bitvector).
>
>
>
> As for the fix, i don’t know what would be the best. I’m tempted to
> think that computing bitvector count when activating the bin is the
> “cleanest” solution as it is likely than count will be used at some
> point. On the other hand, maybe simply replacing bits[j]->cnt() by
> bits[j]->sloppyCnt() would work just fine as well at the expense of
> the precision of the estimation.
>
>
>
> What do you think ?
>
>
>
> Thanks,
>
>
>
> */Dominique Prunier/**//*
>
> APG Architect
>
> Logo-W4N-100dpi
>
> 4388, rue Saint-Denis
>
> Bureau 309
>
> Montreal (Quebec) H2J 2L1
>
> Tel. +1 514-985-6110
>
> Fax +1 514-842-3989
>
> [email protected] <mailto:[email protected]>
>
> www.watch4net.com <http://www.watch4net.com/>
>
> / /
>
> /This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise private information. If you have
> received it in error, please notify the sender immediately and delete
> the original. Any other use of this electronic mail by you is prohibited.
>
> //Ce message est pour le récipiendaire désigné seulement et peut
> contenir des informations privilégiées, propriétaires ou autrement
> privées. Si vous l'avez reçu par erreur, S.V.P. avisez l'expéditeur
> immédiatement et effacez l'original. Toute autre utilisation de ce
> courrier électronique par vous est prohibée.///
>
>
>
>
>
> _______________________________________________
> FastBit-users mailing list
> [email protected]
> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
>
_______________________________________________
FastBit-users mailing list
[email protected]
https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
