Hi, Srikanth,
Thanks for the information. I suspect that the problem is
ibis::bitvector64::setBit, but can not spot the problem with a code
review. So it looks like I will have to dig a little deeper.
Sounds you are using a test program, not involving any sensitive data.
Would you mind share that the test code with me?
John
On 5/6/13 11:48 PM, Srikanth Sundarrajan wrote:
> Thanks much for looking into this so quickly.
>
> This is part of a unit test to verify the thread safety of the
> bitvector class in java. The unit test essentially creates a fixed
> number of bitvector 64 objects each of size 4096 bits through the
> function below.
>
> JNIEXPORT jlong JNICALL
> Java_com_inmobi_shadowfax_FastbitWrapper_IcreateBitset
> (JNIEnv *env, jobject obj, jlong numbits) {
>
> ibis::bitvector64* bitsetptr = new ibis::bitvector64();
> bitsetptr->setBit((long)numbits, 0);
>
> return (long)bitsetptr;
> }
>
> and 3 random bits are set using the following set method
>
> JNIEXPORT void JNICALL Java_com_inmobi_shadowfax_FastbitWrapper_IsetBit
> (JNIEnv *env, jobject obj, jlong bitsetptr, jlong pos) {
>
> ((ibis::bitvector64*)bitsetptr)->setBit(pos, 1);
>
> }
>
> on this bit map a series of bit operations are performed from parallel
> threads in java
>
> Implementations of and/or/xor are given below
>
> JNIEXPORT void JNICALL Java_com_inmobi_shadowfax_FastbitWrapper_Iand
> (JNIEnv *env, jobject obj, jlong targetbitsetptr, jlong sourcebitsetptr) {
> ((ibis::bitvector64*)targetbitsetptr)->operator&=(*(ibis::bitvector64*)sourcebitsetptr);
> }
>
> JNIEXPORT void JNICALL Java_com_inmobi_shadowfax_FastbitWrapper_Ior
> (JNIEnv *env, jobject obj, jlong targetbitsetptr, jlong sourcebitsetptr) {
> ((ibis::bitvector64*)targetbitsetptr)->operator|=(*(ibis::bitvector64*)sourcebitsetptr);
> }
>
>
> JNIEXPORT void JNICALL Java_com_inmobi_shadowfax_FastbitWrapper_Ixor
> (JNIEnv *env, jobject obj, jlong targetbitsetptr, jlong sourcebitsetptr) {
> ((ibis::bitvector64*)targetbitsetptr)->operator^=(*(ibis::bitvector64*)sourcebitsetptr);
> }
>
>
> Regards
> Srikanth Sundarrajan
>
>
> On Tue, May 7, 2013 at 12:13 PM, K. John Wu <[email protected]
> <mailto:[email protected]>> wrote:
>
> Hi, Srikanth,
>
> I can verify that the problem was caused by those two unexpected
> words. This appears to be a flaw in the design of the compression
> scheme that left some special cases unhandled. Since it would
> introduce extra cost to handle this special case, it might be
> worthwhile to figure out how to prevent it being generated in the
> first place. Would you be willing to explain how you generate those
> bitvector64 objects?
>
> Thanks.
>
> John
>
>
>
> On 5/6/13 10:22 PM, Srikanth Sundarrajan wrote:
> > Hi,
> > I am fairly new to fasbit and have been using it for around a
> > month. Have been running into an issue with or_c2 & xor_c2 when run
> > at scale from parallel threads (am using fastbit lib via jni
> from java).
> >
> > The error looks something like
> >
> > ERROR bitvector64::or_c2 -- serious problem here ...
> > terminate called throwing an exception
> > Error -- bitvector64::or_c2 expects to exhaust i0 but there are -159
> > word(s) left
> >
> > I tried adding print statement to print the copy (deep) of the
> > original lhs & rhs value of the bitmaps and looks like the bitmaps
> > weren't mutated while the function was running.
> >
> > lhs - copy of original
> >
> > This bitvector64 stores 4095 bits of a 4097-bit (25 set)
> sequence in a
> > 36-word array and 2 bits in the active word
> > 0 0 1 1 2 2 3 3 4 4 5 5 6
> > 012345678901234567890123456789012345678901234567890123456789012
> > ---------------------------------------------------------------
> >
>
> 00000000000000000000000000000000000000000000000000000000000000000000000000000000
> >
>
> 14000000000000000100000000000000000000000000000000000000000000000000000000000000
> > 28000000000000002126*0
> >
>
> 30000000000000040000000000000000000000000000000000000000000000000000000001000000
> > 4800000000000000163*0
> >
>
> 50020000000000000000000000100000000000000000000000000000000000000000000000000000
> >
>
> 60004000000000000000000000000100000000000000000000000000000000000000000000000000
> >
>
> 70000001000000000000000000000000000000000001000000000000000000000000000000000000
> >
>
> 80000002000000044000000000000000000000000010000000000000000000000000000001000100
> > 980000000000000000*0
> >
>
> 100000001000000000000000000000000000000000001000000000000000000000000000000000000
> > 118000000000000004252*0
> >
>
> 120000400000000000000000000000000010000000000000000000000000000000000000000000000
> > 138000000000000002126*0
> >
>
> 140000008000000000000000000000000000000001000000000000000000000000000000000000000
> >
>
> 150040000000000020000000001000000000000000000000000000000000000000000000000100000
> > 1680000000000000000*0
> >
>
> 170000000000080000000000000000000000000000000000000000000000010000000000000000000
> > 18800000000000000163*0
> >
>
> 190000000000102000000000000000000000000000000000000000000000100000010000000000000
> >
>
> 200000000000000000000000000000000000000000000000000000000000000000000000000000000
> >
>
> 210000000000800000000000000000000000000000000000000000000100000000000000000000000
> > 228000000000000002126*0
> >
>
> 230000000002000000000000000000000000000000000000000000010000000000000000000000000
> > 248000000000000004252*0
> >
>
> 250000000100000000000000000000000000000000000000100000000000000000000000000000000
> > 268000000000000002126*0
> >
>
> 270000000000020000000000000000000000000000000000000000000000000100000000000000000
> > 28800000000000000163*0
> >
>
> 290000000000001800000000000000000000000000000000000000000000000000001100000000000
> >
>
> 300000000004000000000000000000000000000000000000000000100000000000000000000000000
> > 31800000000000000163*0
> >
>
> 320000000080000000000000000000000000000000000000010000000000000000000000000000000
> > 338000000000000006378*0
> >
>
> 340000010000000000000000000000000000000010000000000000000000000000000000000000000
> > 3580000000000000111071*0
> > 000000000000000000
> >
> > lhs - original
> >
> > This bitvector64 stores 4095 bits of a 4097-bit (25 set)
> sequence in a
> > 36-word array and 2 bits in the active word
> > 0 0 1 1 2 2 3 3 4 4 5 5 6
> > 012345678901234567890123456789012345678901234567890123456789012
> > ---------------------------------------------------------------
> >
>
> 00000000000000000000000000000000000000000000000000000000000000000000000000000000
> >
>
> 14000000000000000100000000000000000000000000000000000000000000000000000000000000
> > 28000000000000002126*0
> >
>
> 30000000000000040000000000000000000000000000000000000000000000000000000001000000
> > 4800000000000000163*0
> >
>
> 50020000000000000000000000100000000000000000000000000000000000000000000000000000
> >
>
> 60004000000000000000000000000100000000000000000000000000000000000000000000000000
> >
>
> 70000001000000000000000000000000000000000001000000000000000000000000000000000000
> >
>
> 80000002000000044000000000000000000000000010000000000000000000000000000001000100
> > 980000000000000000*0
> >
>
> 100000001000000000000000000000000000000000001000000000000000000000000000000000000
> > 118000000000000004252*0
> >
>
> 120000400000000000000000000000000010000000000000000000000000000000000000000000000
> > 138000000000000002126*0
> >
>
> 140000008000000000000000000000000000000001000000000000000000000000000000000000000
> >
>
> 150040000000000020000000001000000000000000000000000000000000000000000000000100000
> > 1680000000000000000*0
> >
>
> 170000000000080000000000000000000000000000000000000000000000010000000000000000000
> > 18800000000000000163*0
> >
>
> 190000000000102000000000000000000000000000000000000000000000100000010000000000000
> >
>
> 200000000000000000000000000000000000000000000000000000000000000000000000000000000
> >
>
> 210000000000800000000000000000000000000000000000000000000100000000000000000000000
> > 228000000000000002126*0
> >
>
> 230000000002000000000000000000000000000000000000000000010000000000000000000000000
> > 248000000000000004252*0
> >
>
> 250000000100000000000000000000000000000000000000100000000000000000000000000000000
> > 268000000000000002126*0
> >
>
> 270000000000020000000000000000000000000000000000000000000000000100000000000000000
> > 28800000000000000163*0
> >
>
> 290000000000001800000000000000000000000000000000000000000000000000001100000000000
> >
>
> 300000000004000000000000000000000000000000000000000000100000000000000000000000000
> > 31800000000000000163*0
> >
>
> 320000000080000000000000000000000000000000000000010000000000000000000000000000000
> > 338000000000000006378*0
> >
>
> 340000010000000000000000000000000000000010000000000000000000000000000000000000000
> > 3580000000000000111071*0
> > 000000000000000000
> >
> > rhs - copy of original
> >
> > This bitvector64 stores 4095 bits of a 4097-bit (3 set) sequence
> in a
> > 7-word array and 2 bits in the active word
> > 0 0 1 1 2 2 3 3 4 4 5 5 6
> > 012345678901234567890123456789012345678901234567890123456789012
> > ---------------------------------------------------------------
> > 0800000000000000e882*0
> >
>
> 10000000000040000000000000000000000000000000000000000000000001000000000000000000
> > 28000000000000005315*0
> >
>
> 30000010000000000000000000000000000000010000000000000000000000000000000000000000
> > 4800000000000001a1638*0
> >
>
> 50000080000000000000000000000000000010000000000000000000000000000000000000000000
> > 680000000000000111071*0
> > 000000000000000000
> >
> > rhs - original
> >
> > This bitvector64 stores 4095 bits of a 4097-bit (3 set) sequence
> in a
> > 7-word array and 2 bits in the active word
> > 0 0 1 1 2 2 3 3 4 4 5 5 6
> > 012345678901234567890123456789012345678901234567890123456789012
> > ---------------------------------------------------------------
> > 0800000000000000e882*0
> >
>
> 10000000000040000000000000000000000000000000000000000000000001000000000000000000
> > 28000000000000005315*0
> >
>
> 30000010000000000000000000000000000000010000000000000000000000000000000000000000
> > 4800000000000001a1638*0
> >
>
> 50000080000000000000000000000000000010000000000000000000000000000000000000000000
> > 680000000000000111071*0
> > 000000000000000000
> >
> > Any guiding pointers are appreciated.
> >
> > Regards
> > Srikanth Sundarrajan
> >
> >
> >
> > _____________________________________________________________
> > The information contained in this communication is intended
> solely for
> > the use of the individual or entity to whom it is addressed and
> others
> > authorized to receive it. It may contain confidential or legally
> > privileged information. If you are not the intended recipient
> you are
> > hereby notified that any disclosure, copying, distribution or taking
> > any action in reliance on the contents of this information is
> strictly
> > prohibited and may be unlawful. If you have received this
> > communication in error, please notify us immediately by
> responding to
> > this email and then delete it from your system. The firm is neither
> > liable for the proper and complete transmission of the information
> > contained in this communication nor for any delay in its receipt.
> >
> >
> > _______________________________________________
> > FastBit-users mailing list
> > [email protected] <mailto:[email protected]>
> > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
> >
>
>
>
> _____________________________________________________________
> The information contained in this communication is intended solely for
> the use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally
> privileged information. If you are not the intended recipient you are
> hereby notified that any disclosure, copying, distribution or taking
> any action in reliance on the contents of this information is strictly
> prohibited and may be unlawful. If you have received this
> communication in error, please notify us immediately by responding to
> this email and then delete it from your system. The firm is neither
> liable for the proper and complete transmission of the information
> contained in this communication nor for any delay in its receipt.
_______________________________________________
FastBit-users mailing list
[email protected]
https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
