Hi, Srikanth,
The code you have here only sets three bits to 1 for each bitmap,
however the two bitmaps you showed in earlier message has 25 ones and
7 ones respectively. What are the operations you performed on these
bitmaps with only 3 ones each to generate the ones with 25 ones?
Which java wrapper are you using?
John
On 5/7/13 12:44 AM, Srikanth Sundarrajan wrote:
> Test it is a test code. Test is in Java, so sending you the relevant
> pieces that will allow you to stitch them together. Should you need
> further information, please do let me know.
>
> Test Code:
>
> @Test
> public void testBitmapThreadSafety() throws Exception {
> PrintStream debug;
> if (new File("target").exists()) {
> debug = new PrintStream("target/debug.txt");
> } else {
> debug = new PrintStream("store/target/debug.txt");
> }
> try {
> FastbitWrapper.debug(debug);
> final int rows = 1000;
> final int threads = 6;
> final int bitMapMaxSize = 4096;
> final ConcurrentMap<Integer, Pair<AtomicInteger, Bitmap>>
> map =
> new ConcurrentHashMap<Integer, Pair<AtomicInteger,
> Bitmap>>();
> ScheduledThreadPoolExecutor executor = new
> ScheduledThreadPoolExecutor(threads);
> for (int index = 0; index < rows; index++) {
> Random random = new Random();
> Bitmap bitmap = getBitmap(random, bitMapMaxSize);
> map.put(index, Pair.of(new AtomicInteger(0), bitmap));
> }
>
> Runnable bitmapManipulator = new Runnable() {
> @Override
> public void run() {
> for (int counter = 0; counter < 5000; counter++) {
> Random random = new Random();
> int key = random.nextInt(rows);
> Bitmap bitmap = map.get(key).second;
> map.get(key).first.incrementAndGet();
> Bitmap tmp = getBitmap(random, bitMapMaxSize);
> int op = random.nextInt(8);
> switch (op) {
> case 0:
> bitmap.and(tmp);
> break;
> case 1:
> bitmap.xor(tmp);
> break;
> case 2:
> bitmap.or(tmp);
> break;
> case 3:
> bitmap.flip();
> break;
> case 4:
> long pos = random.nextInt(bitMapMaxSize);
> bitmap.setBit(pos);
> break;
> case 5:
> long unsetpos =
> random.nextInt(bitMapMaxSize);
> bitmap.unsetBit(unsetpos);
> break;
> case 6:
> long getPos =
> random.nextInt(bitMapMaxSize);
> bitmap.get(getPos);
> break;
> case 7:
> bitmap.cardinality();
> break;
> default:
> }
> BitmapUtils.closeBitmap(tmp);
> }
> }
> };
>
> for (int counter = 0; counter < threads; counter++) {
> executor.schedule(bitmapManipulator, 0,
> TimeUnit.MILLISECONDS);
> }
> while (executor.getCompletedTaskCount() < threads) {
> Thread.sleep(500);
> }
>
> for (Pair<AtomicInteger, Bitmap> pair : map.values()) {
> BitmapUtils.closeBitmap(pair.second);
> }
> } finally {
> debug.close();
> }
> }
>
> private Bitmap getBitmap(Random random, long bitMapMaxSize) {
> Bitmap bitmap =
> BitmapFactory.getInstance().newBitmap(bitMapMaxSize);
> int quarter = (int) bitMapMaxSize / 4;
> int bits[] = new int[] {random.nextInt(quarter), quarter +
> random.nextInt(quarter),
> quarter * 2 + random.nextInt(quarter)};
> bitmap.setBit(bits[0]);
> bitmap.setBit(bits[1]);
> bitmap.setBit(bits[2]);
> return bitmap;
> }
>
> public final class BitmapFactory {
>
> private static final BitmapFactory INSTANCE = new BitmapFactory();
>
> private BitmapFactory() {
> }
>
> public static BitmapFactory getInstance() {
> return INSTANCE;
> }
> public Bitmap newBitmap(long bits) {
> return new FastbitWrapper(bits);
> }
> }
>
>
>
>
> On Tue, May 7, 2013 at 12:45 PM, K. John Wu <[email protected]
> <mailto:[email protected]>> wrote:
>
> Hi, Srikanth,
>
> Thanks for the information. I suspect that the problem is
> ibis::bitvector64::setBit, but can not spot the problem with a code
> review. So it looks like I will have to dig a little deeper.
>
> Sounds you are using a test program, not involving any sensitive data.
> Would you mind share that the test code with me?
>
> John
>
>
> On 5/6/13 11:48 PM, Srikanth Sundarrajan wrote:
> > Thanks much for looking into this so quickly.
> >
> > This is part of a unit test to verify the thread safety of the
> > bitvector class in java. The unit test essentially creates a fixed
> > number of bitvector 64 objects each of size 4096 bits through the
> > function below.
> >
> > JNIEXPORT jlong JNICALL
> > Java_com_inmobi_shadowfax_FastbitWrapper_IcreateBitset
> > (JNIEnv *env, jobject obj, jlong numbits) {
> >
> > ibis::bitvector64* bitsetptr = new ibis::bitvector64();
> > bitsetptr->setBit((long)numbits, 0);
> >
> > return (long)bitsetptr;
> > }
> >
> > and 3 random bits are set using the following set method
> >
> > JNIEXPORT void JNICALL
> Java_com_inmobi_shadowfax_FastbitWrapper_IsetBit
> > (JNIEnv *env, jobject obj, jlong bitsetptr, jlong pos) {
> >
> > ((ibis::bitvector64*)bitsetptr)->setBit(pos, 1);
> >
> > }
> >
> > on this bit map a series of bit operations are performed from
> parallel
> > threads in java
> >
> > Implementations of and/or/xor are given below
> >
> > JNIEXPORT void JNICALL Java_com_inmobi_shadowfax_FastbitWrapper_Iand
> > (JNIEnv *env, jobject obj, jlong targetbitsetptr, jlong
> sourcebitsetptr) {
> >
>
> ((ibis::bitvector64*)targetbitsetptr)->operator&=(*(ibis::bitvector64*)sourcebitsetptr);
> > }
> >
> > JNIEXPORT void JNICALL Java_com_inmobi_shadowfax_FastbitWrapper_Ior
> > (JNIEnv *env, jobject obj, jlong targetbitsetptr, jlong
> sourcebitsetptr) {
> >
>
> ((ibis::bitvector64*)targetbitsetptr)->operator|=(*(ibis::bitvector64*)sourcebitsetptr);
> > }
> >
> >
> > JNIEXPORT void JNICALL Java_com_inmobi_shadowfax_FastbitWrapper_Ixor
> > (JNIEnv *env, jobject obj, jlong targetbitsetptr, jlong
> sourcebitsetptr) {
> >
>
> ((ibis::bitvector64*)targetbitsetptr)->operator^=(*(ibis::bitvector64*)sourcebitsetptr);
> > }
> >
> >
> > Regards
> > Srikanth Sundarrajan
> >
> >
> > On Tue, May 7, 2013 at 12:13 PM, K. John Wu <[email protected]
> <mailto:[email protected]>
> > <mailto:[email protected] <mailto:[email protected]>>> wrote:
> >
> > Hi, Srikanth,
> >
> > I can verify that the problem was caused by those two unexpected
> > words. This appears to be a flaw in the design of the
> compression
> > scheme that left some special cases unhandled. Since it would
> > introduce extra cost to handle this special case, it might be
> > worthwhile to figure out how to prevent it being generated
> in the
> > first place. Would you be willing to explain how you
> generate those
> > bitvector64 objects?
> >
> > Thanks.
> >
> > John
> >
> >
> >
> > On 5/6/13 10:22 PM, Srikanth Sundarrajan wrote:
> > > Hi,
> > > I am fairly new to fasbit and have been using it for
> around a
> > > month. Have been running into an issue with or_c2 &
> xor_c2 when run
> > > at scale from parallel threads (am using fastbit lib via jni
> > from java).
> > >
> > > The error looks something like
> > >
> > > ERROR bitvector64::or_c2 -- serious problem here ...
> > > terminate called throwing an exception
> > > Error -- bitvector64::or_c2 expects to exhaust i0 but
> there are -159
> > > word(s) left
> > >
> > > I tried adding print statement to print the copy (deep) of the
> > > original lhs & rhs value of the bitmaps and looks like the
> bitmaps
> > > weren't mutated while the function was running.
> > >
> > > lhs - copy of original
> > >
> > > This bitvector64 stores 4095 bits of a 4097-bit (25 set)
> > sequence in a
> > > 36-word array and 2 bits in the active word
> > > 0 0 1 1 2 2 3 3 4 4 5 5 6
> > >
> 012345678901234567890123456789012345678901234567890123456789012
> > >
> ---------------------------------------------------------------
> > >
> >
>
> 00000000000000000000000000000000000000000000000000000000000000000000000000000000
> > >
> >
>
> 14000000000000000100000000000000000000000000000000000000000000000000000000000000
> > > 28000000000000002126*0
> > >
> >
>
> 30000000000000040000000000000000000000000000000000000000000000000000000001000000
> > > 4800000000000000163*0
> > >
> >
>
> 50020000000000000000000000100000000000000000000000000000000000000000000000000000
> > >
> >
>
> 60004000000000000000000000000100000000000000000000000000000000000000000000000000
> > >
> >
>
> 70000001000000000000000000000000000000000001000000000000000000000000000000000000
> > >
> >
>
> 80000002000000044000000000000000000000000010000000000000000000000000000001000100
> > > 980000000000000000*0
> > >
> >
>
> 100000001000000000000000000000000000000000001000000000000000000000000000000000000
> > > 118000000000000004252*0
> > >
> >
>
> 120000400000000000000000000000000010000000000000000000000000000000000000000000000
> > > 138000000000000002126*0
> > >
> >
>
> 140000008000000000000000000000000000000001000000000000000000000000000000000000000
> > >
> >
>
> 150040000000000020000000001000000000000000000000000000000000000000000000000100000
> > > 1680000000000000000*0
> > >
> >
>
> 170000000000080000000000000000000000000000000000000000000000010000000000000000000
> > > 18800000000000000163*0
> > >
> >
>
> 190000000000102000000000000000000000000000000000000000000000100000010000000000000
> > >
> >
>
> 200000000000000000000000000000000000000000000000000000000000000000000000000000000
> > >
> >
>
> 210000000000800000000000000000000000000000000000000000000100000000000000000000000
> > > 228000000000000002126*0
> > >
> >
>
> 230000000002000000000000000000000000000000000000000000010000000000000000000000000
> > > 248000000000000004252*0
> > >
> >
>
> 250000000100000000000000000000000000000000000000100000000000000000000000000000000
> > > 268000000000000002126*0
> > >
> >
>
> 270000000000020000000000000000000000000000000000000000000000000100000000000000000
> > > 28800000000000000163*0
> > >
> >
>
> 290000000000001800000000000000000000000000000000000000000000000000001100000000000
> > >
> >
>
> 300000000004000000000000000000000000000000000000000000100000000000000000000000000
> > > 31800000000000000163*0
> > >
> >
>
> 320000000080000000000000000000000000000000000000010000000000000000000000000000000
> > > 338000000000000006378*0
> > >
> >
>
> 340000010000000000000000000000000000000010000000000000000000000000000000000000000
> > > 3580000000000000111071*0
> > > 000000000000000000
> > >
> > > lhs - original
> > >
> > > This bitvector64 stores 4095 bits of a 4097-bit (25 set)
> > sequence in a
> > > 36-word array and 2 bits in the active word
> > > 0 0 1 1 2 2 3 3 4 4 5 5 6
> > >
> 012345678901234567890123456789012345678901234567890123456789012
> > >
> ---------------------------------------------------------------
> > >
> >
>
> 00000000000000000000000000000000000000000000000000000000000000000000000000000000
> > >
> >
>
> 14000000000000000100000000000000000000000000000000000000000000000000000000000000
> > > 28000000000000002126*0
> > >
> >
>
> 30000000000000040000000000000000000000000000000000000000000000000000000001000000
> > > 4800000000000000163*0
> > >
> >
>
> 50020000000000000000000000100000000000000000000000000000000000000000000000000000
> > >
> >
>
> 60004000000000000000000000000100000000000000000000000000000000000000000000000000
> > >
> >
>
> 70000001000000000000000000000000000000000001000000000000000000000000000000000000
> > >
> >
>
> 80000002000000044000000000000000000000000010000000000000000000000000000001000100
> > > 980000000000000000*0
> > >
> >
>
> 100000001000000000000000000000000000000000001000000000000000000000000000000000000
> > > 118000000000000004252*0
> > >
> >
>
> 120000400000000000000000000000000010000000000000000000000000000000000000000000000
> > > 138000000000000002126*0
> > >
> >
>
> 140000008000000000000000000000000000000001000000000000000000000000000000000000000
> > >
> >
>
> 150040000000000020000000001000000000000000000000000000000000000000000000000100000
> > > 1680000000000000000*0
> > >
> >
>
> 170000000000080000000000000000000000000000000000000000000000010000000000000000000
> > > 18800000000000000163*0
> > >
> >
>
> 190000000000102000000000000000000000000000000000000000000000100000010000000000000
> > >
> >
>
> 200000000000000000000000000000000000000000000000000000000000000000000000000000000
> > >
> >
>
> 210000000000800000000000000000000000000000000000000000000100000000000000000000000
> > > 228000000000000002126*0
> > >
> >
>
> 230000000002000000000000000000000000000000000000000000010000000000000000000000000
> > > 248000000000000004252*0
> > >
> >
>
> 250000000100000000000000000000000000000000000000100000000000000000000000000000000
> > > 268000000000000002126*0
> > >
> >
>
> 270000000000020000000000000000000000000000000000000000000000000100000000000000000
> > > 28800000000000000163*0
> > >
> >
>
> 290000000000001800000000000000000000000000000000000000000000000000001100000000000
> > >
> >
>
> 300000000004000000000000000000000000000000000000000000100000000000000000000000000
> > > 31800000000000000163*0
> > >
> >
>
> 320000000080000000000000000000000000000000000000010000000000000000000000000000000
> > > 338000000000000006378*0
> > >
> >
>
> 340000010000000000000000000000000000000010000000000000000000000000000000000000000
> > > 3580000000000000111071*0
> > > 000000000000000000
> > >
> > > rhs - copy of original
> > >
> > > This bitvector64 stores 4095 bits of a 4097-bit (3 set)
> sequence
> > in a
> > > 7-word array and 2 bits in the active word
> > > 0 0 1 1 2 2 3 3 4 4 5 5 6
> > >
> 012345678901234567890123456789012345678901234567890123456789012
> > >
> ---------------------------------------------------------------
> > > 0800000000000000e882*0
> > >
> >
>
> 10000000000040000000000000000000000000000000000000000000000001000000000000000000
> > > 28000000000000005315*0
> > >
> >
>
> 30000010000000000000000000000000000000010000000000000000000000000000000000000000
> > > 4800000000000001a1638*0
> > >
> >
>
> 50000080000000000000000000000000000010000000000000000000000000000000000000000000
> > > 680000000000000111071*0
> > > 000000000000000000
> > >
> > > rhs - original
> > >
> > > This bitvector64 stores 4095 bits of a 4097-bit (3 set)
> sequence
> > in a
> > > 7-word array and 2 bits in the active word
> > > 0 0 1 1 2 2 3 3 4 4 5 5 6
> > >
> 012345678901234567890123456789012345678901234567890123456789012
> > >
> ---------------------------------------------------------------
> > > 0800000000000000e882*0
> > >
> >
>
> 10000000000040000000000000000000000000000000000000000000000001000000000000000000
> > > 28000000000000005315*0
> > >
> >
>
> 30000010000000000000000000000000000000010000000000000000000000000000000000000000
> > > 4800000000000001a1638*0
> > >
> >
>
> 50000080000000000000000000000000000010000000000000000000000000000000000000000000
> > > 680000000000000111071*0
> > > 000000000000000000
> > >
> > > Any guiding pointers are appreciated.
> > >
> > > Regards
> > > Srikanth Sundarrajan
> > >
> > >
> > >
> > > _____________________________________________________________
> > > The information contained in this communication is intended
> > solely for
> > > the use of the individual or entity to whom it is
> addressed and
> > others
> > > authorized to receive it. It may contain confidential or
> legally
> > > privileged information. If you are not the intended recipient
> > you are
> > > hereby notified that any disclosure, copying, distribution
> or taking
> > > any action in reliance on the contents of this information is
> > strictly
> > > prohibited and may be unlawful. If you have received this
> > > communication in error, please notify us immediately by
> > responding to
> > > this email and then delete it from your system. The firm
> is neither
> > > liable for the proper and complete transmission of the
> information
> > > contained in this communication nor for any delay in its
> receipt.
> > >
> > >
> > > _______________________________________________
> > > FastBit-users mailing list
> > > [email protected]
> <mailto:[email protected]>
> <mailto:[email protected]
> <mailto:[email protected]>>
> > > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
> > >
> >
> >
> >
> > _____________________________________________________________
> > The information contained in this communication is intended
> solely for
> > the use of the individual or entity to whom it is addressed and
> others
> > authorized to receive it. It may contain confidential or legally
> > privileged information. If you are not the intended recipient
> you are
> > hereby notified that any disclosure, copying, distribution or taking
> > any action in reliance on the contents of this information is
> strictly
> > prohibited and may be unlawful. If you have received this
> > communication in error, please notify us immediately by
> responding to
> > this email and then delete it from your system. The firm is neither
> > liable for the proper and complete transmission of the information
> > contained in this communication nor for any delay in its receipt.
>
>
>
> _____________________________________________________________
> The information contained in this communication is intended solely for
> the use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally
> privileged information. If you are not the intended recipient you are
> hereby notified that any disclosure, copying, distribution or taking
> any action in reliance on the contents of this information is strictly
> prohibited and may be unlawful. If you have received this
> communication in error, please notify us immediately by responding to
> this email and then delete it from your system. The firm is neither
> liable for the proper and complete transmission of the information
> contained in this communication nor for any delay in its receipt.
_______________________________________________
FastBit-users mailing list
[email protected]
https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
