Hi, Srikanth, The additional comments about the cost of setBit is just a reminder of the cost, it does not really fix anything ;-)
The fix is in bitvector64.cpp, where I have added statements to skip over 0-length fills (the words 9 and 16 I mentioned in the previous message were 0-length fills). I thought the software was not supposed to produce 0-length fills, but they obviously have appeared by some magic. I would like to figure out how they got produced and find a way to get around it. The first place to look for this problem is in setBit, however, so far I have no luck in finding anything obvious. John On 5/7/13 10:36 AM, Srikanth Sundarrajan wrote: > Thanks John, the patch seemed to have helped. May be I am asking too > much, but do you mind help me understand what the issue was and how > the fix helped? Also the patch mentioned that the setBit is going to > be expensive and is to be avoided. What do you recommend we use > instead to specifically set selective bits ? > > Thanks much > Srikanth Sundarrajan > > > On Tue, May 7, 2013 at 1:13 PM, K. John Wu <[email protected] > <mailto:[email protected]>> wrote: > > Hi, Srikanth, > > Just checked in some temporary fixes to the bitwise logical operations > as SVN Revision 628. You can download it with > > svn checkout https://codeforge.lbl.gov/anonscm/fastbit > > > I'd still like to have your test program if you are able to share it > with us. > > Thanks. > > John > > > > On 5/6/13 11:48 PM, Srikanth Sundarrajan wrote: > > Thanks much for looking into this so quickly. > > > > This is part of a unit test to verify the thread safety of the > > bitvector class in java. The unit test essentially creates a fixed > > number of bitvector 64 objects each of size 4096 bits through the > > function below. > > > > JNIEXPORT jlong JNICALL > > Java_com_inmobi_shadowfax_FastbitWrapper_IcreateBitset > > (JNIEnv *env, jobject obj, jlong numbits) { > > > > ibis::bitvector64* bitsetptr = new ibis::bitvector64(); > > bitsetptr->setBit((long)numbits, 0); > > > > return (long)bitsetptr; > > } > > > > and 3 random bits are set using the following set method > > > > JNIEXPORT void JNICALL > Java_com_inmobi_shadowfax_FastbitWrapper_IsetBit > > (JNIEnv *env, jobject obj, jlong bitsetptr, jlong pos) { > > > > ((ibis::bitvector64*)bitsetptr)->setBit(pos, 1); > > > > } > > > > on this bit map a series of bit operations are performed from > parallel > > threads in java > > > > Implementations of and/or/xor are given below > > > > JNIEXPORT void JNICALL Java_com_inmobi_shadowfax_FastbitWrapper_Iand > > (JNIEnv *env, jobject obj, jlong targetbitsetptr, jlong > sourcebitsetptr) { > > > > ((ibis::bitvector64*)targetbitsetptr)->operator&=(*(ibis::bitvector64*)sourcebitsetptr); > > } > > > > JNIEXPORT void JNICALL Java_com_inmobi_shadowfax_FastbitWrapper_Ior > > (JNIEnv *env, jobject obj, jlong targetbitsetptr, jlong > sourcebitsetptr) { > > > > ((ibis::bitvector64*)targetbitsetptr)->operator|=(*(ibis::bitvector64*)sourcebitsetptr); > > } > > > > > > JNIEXPORT void JNICALL Java_com_inmobi_shadowfax_FastbitWrapper_Ixor > > (JNIEnv *env, jobject obj, jlong targetbitsetptr, jlong > sourcebitsetptr) { > > > > ((ibis::bitvector64*)targetbitsetptr)->operator^=(*(ibis::bitvector64*)sourcebitsetptr); > > } > > > > > > Regards > > Srikanth Sundarrajan > > > > > > On Tue, May 7, 2013 at 12:13 PM, K. John Wu <[email protected] > <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>>> wrote: > > > > Hi, Srikanth, > > > > I can verify that the problem was caused by those two unexpected > > words. This appears to be a flaw in the design of the > compression > > scheme that left some special cases unhandled. Since it would > > introduce extra cost to handle this special case, it might be > > worthwhile to figure out how to prevent it being generated > in the > > first place. Would you be willing to explain how you > generate those > > bitvector64 objects? > > > > Thanks. > > > > John > > > > > > > > On 5/6/13 10:22 PM, Srikanth Sundarrajan wrote: > > > Hi, > > > I am fairly new to fasbit and have been using it for > around a > > > month. Have been running into an issue with or_c2 & > xor_c2 when run > > > at scale from parallel threads (am using fastbit lib via jni > > from java). > > > > > > The error looks something like > > > > > > ERROR bitvector64::or_c2 -- serious problem here ... > > > terminate called throwing an exception > > > Error -- bitvector64::or_c2 expects to exhaust i0 but > there are -159 > > > word(s) left > > > > > > I tried adding print statement to print the copy (deep) of the > > > original lhs & rhs value of the bitmaps and looks like the > bitmaps > > > weren't mutated while the function was running. > > > > > > lhs - copy of original > > > > > > This bitvector64 stores 4095 bits of a 4097-bit (25 set) > > sequence in a > > > 36-word array and 2 bits in the active word > > > 0 0 1 1 2 2 3 3 4 4 5 5 6 > > > > 012345678901234567890123456789012345678901234567890123456789012 > > > > --------------------------------------------------------------- > > > > > > > 00000000000000000000000000000000000000000000000000000000000000000000000000000000 > > > > > > > 14000000000000000100000000000000000000000000000000000000000000000000000000000000 > > > 28000000000000002126*0 > > > > > > > 30000000000000040000000000000000000000000000000000000000000000000000000001000000 > > > 4800000000000000163*0 > > > > > > > 50020000000000000000000000100000000000000000000000000000000000000000000000000000 > > > > > > > 60004000000000000000000000000100000000000000000000000000000000000000000000000000 > > > > > > > 70000001000000000000000000000000000000000001000000000000000000000000000000000000 > > > > > > > 80000002000000044000000000000000000000000010000000000000000000000000000001000100 > > > 980000000000000000*0 > > > > > > > 100000001000000000000000000000000000000000001000000000000000000000000000000000000 > > > 118000000000000004252*0 > > > > > > > 120000400000000000000000000000000010000000000000000000000000000000000000000000000 > > > 138000000000000002126*0 > > > > > > > 140000008000000000000000000000000000000001000000000000000000000000000000000000000 > > > > > > > 150040000000000020000000001000000000000000000000000000000000000000000000000100000 > > > 1680000000000000000*0 > > > > > > > 170000000000080000000000000000000000000000000000000000000000010000000000000000000 > > > 18800000000000000163*0 > > > > > > > 190000000000102000000000000000000000000000000000000000000000100000010000000000000 > > > > > > > 200000000000000000000000000000000000000000000000000000000000000000000000000000000 > > > > > > > 210000000000800000000000000000000000000000000000000000000100000000000000000000000 > > > 228000000000000002126*0 > > > > > > > 230000000002000000000000000000000000000000000000000000010000000000000000000000000 > > > 248000000000000004252*0 > > > > > > > 250000000100000000000000000000000000000000000000100000000000000000000000000000000 > > > 268000000000000002126*0 > > > > > > > 270000000000020000000000000000000000000000000000000000000000000100000000000000000 > > > 28800000000000000163*0 > > > > > > > 290000000000001800000000000000000000000000000000000000000000000000001100000000000 > > > > > > > 300000000004000000000000000000000000000000000000000000100000000000000000000000000 > > > 31800000000000000163*0 > > > > > > > 320000000080000000000000000000000000000000000000010000000000000000000000000000000 > > > 338000000000000006378*0 > > > > > > > 340000010000000000000000000000000000000010000000000000000000000000000000000000000 > > > 3580000000000000111071*0 > > > 000000000000000000 > > > > > > lhs - original > > > > > > This bitvector64 stores 4095 bits of a 4097-bit (25 set) > > sequence in a > > > 36-word array and 2 bits in the active word > > > 0 0 1 1 2 2 3 3 4 4 5 5 6 > > > > 012345678901234567890123456789012345678901234567890123456789012 > > > > --------------------------------------------------------------- > > > > > > > 00000000000000000000000000000000000000000000000000000000000000000000000000000000 > > > > > > > 14000000000000000100000000000000000000000000000000000000000000000000000000000000 > > > 28000000000000002126*0 > > > > > > > 30000000000000040000000000000000000000000000000000000000000000000000000001000000 > > > 4800000000000000163*0 > > > > > > > 50020000000000000000000000100000000000000000000000000000000000000000000000000000 > > > > > > > 60004000000000000000000000000100000000000000000000000000000000000000000000000000 > > > > > > > 70000001000000000000000000000000000000000001000000000000000000000000000000000000 > > > > > > > 80000002000000044000000000000000000000000010000000000000000000000000000001000100 > > > 980000000000000000*0 > > > > > > > 100000001000000000000000000000000000000000001000000000000000000000000000000000000 > > > 118000000000000004252*0 > > > > > > > 120000400000000000000000000000000010000000000000000000000000000000000000000000000 > > > 138000000000000002126*0 > > > > > > > 140000008000000000000000000000000000000001000000000000000000000000000000000000000 > > > > > > > 150040000000000020000000001000000000000000000000000000000000000000000000000100000 > > > 1680000000000000000*0 > > > > > > > 170000000000080000000000000000000000000000000000000000000000010000000000000000000 > > > 18800000000000000163*0 > > > > > > > 190000000000102000000000000000000000000000000000000000000000100000010000000000000 > > > > > > > 200000000000000000000000000000000000000000000000000000000000000000000000000000000 > > > > > > > 210000000000800000000000000000000000000000000000000000000100000000000000000000000 > > > 228000000000000002126*0 > > > > > > > 230000000002000000000000000000000000000000000000000000010000000000000000000000000 > > > 248000000000000004252*0 > > > > > > > 250000000100000000000000000000000000000000000000100000000000000000000000000000000 > > > 268000000000000002126*0 > > > > > > > 270000000000020000000000000000000000000000000000000000000000000100000000000000000 > > > 28800000000000000163*0 > > > > > > > 290000000000001800000000000000000000000000000000000000000000000000001100000000000 > > > > > > > 300000000004000000000000000000000000000000000000000000100000000000000000000000000 > > > 31800000000000000163*0 > > > > > > > 320000000080000000000000000000000000000000000000010000000000000000000000000000000 > > > 338000000000000006378*0 > > > > > > > 340000010000000000000000000000000000000010000000000000000000000000000000000000000 > > > 3580000000000000111071*0 > > > 000000000000000000 > > > > > > rhs - copy of original > > > > > > This bitvector64 stores 4095 bits of a 4097-bit (3 set) > sequence > > in a > > > 7-word array and 2 bits in the active word > > > 0 0 1 1 2 2 3 3 4 4 5 5 6 > > > > 012345678901234567890123456789012345678901234567890123456789012 > > > > --------------------------------------------------------------- > > > 0800000000000000e882*0 > > > > > > > 10000000000040000000000000000000000000000000000000000000000001000000000000000000 > > > 28000000000000005315*0 > > > > > > > 30000010000000000000000000000000000000010000000000000000000000000000000000000000 > > > 4800000000000001a1638*0 > > > > > > > 50000080000000000000000000000000000010000000000000000000000000000000000000000000 > > > 680000000000000111071*0 > > > 000000000000000000 > > > > > > rhs - original > > > > > > This bitvector64 stores 4095 bits of a 4097-bit (3 set) > sequence > > in a > > > 7-word array and 2 bits in the active word > > > 0 0 1 1 2 2 3 3 4 4 5 5 6 > > > > 012345678901234567890123456789012345678901234567890123456789012 > > > > --------------------------------------------------------------- > > > 0800000000000000e882*0 > > > > > > > 10000000000040000000000000000000000000000000000000000000000001000000000000000000 > > > 28000000000000005315*0 > > > > > > > 30000010000000000000000000000000000000010000000000000000000000000000000000000000 > > > 4800000000000001a1638*0 > > > > > > > 50000080000000000000000000000000000010000000000000000000000000000000000000000000 > > > 680000000000000111071*0 > > > 000000000000000000 > > > > > > Any guiding pointers are appreciated. > > > > > > Regards > > > Srikanth Sundarrajan > > > > > > > > > > > > _____________________________________________________________ > > > The information contained in this communication is intended > > solely for > > > the use of the individual or entity to whom it is > addressed and > > others > > > authorized to receive it. It may contain confidential or > legally > > > privileged information. If you are not the intended recipient > > you are > > > hereby notified that any disclosure, copying, distribution > or taking > > > any action in reliance on the contents of this information is > > strictly > > > prohibited and may be unlawful. If you have received this > > > communication in error, please notify us immediately by > > responding to > > > this email and then delete it from your system. The firm > is neither > > > liable for the proper and complete transmission of the > information > > > contained in this communication nor for any delay in its > receipt. > > > > > > > > > _______________________________________________ > > > FastBit-users mailing list > > > [email protected] > <mailto:[email protected]> > <mailto:[email protected] > <mailto:[email protected]>> > > > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > > > > > > > > > > > _____________________________________________________________ > > The information contained in this communication is intended > solely for > > the use of the individual or entity to whom it is addressed and > others > > authorized to receive it. It may contain confidential or legally > > privileged information. If you are not the intended recipient > you are > > hereby notified that any disclosure, copying, distribution or taking > > any action in reliance on the contents of this information is > strictly > > prohibited and may be unlawful. If you have received this > > communication in error, please notify us immediately by > responding to > > this email and then delete it from your system. The firm is neither > > liable for the proper and complete transmission of the information > > contained in this communication nor for any delay in its receipt. > > > > _____________________________________________________________ > The information contained in this communication is intended solely for > the use of the individual or entity to whom it is addressed and others > authorized to receive it. It may contain confidential or legally > privileged information. If you are not the intended recipient you are > hereby notified that any disclosure, copying, distribution or taking > any action in reliance on the contents of this information is strictly > prohibited and may be unlawful. If you have received this > communication in error, please notify us immediately by responding to > this email and then delete it from your system. The firm is neither > liable for the proper and complete transmission of the information > contained in this communication nor for any delay in its receipt. _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
