Test it is a test code. Test is in Java, so sending you the relevant pieces
that will allow you to stitch them together. Should you need further
information, please do let me know.
Test Code:
@Test
public void testBitmapThreadSafety() throws Exception {
PrintStream debug;
if (new File("target").exists()) {
debug = new PrintStream("target/debug.txt");
} else {
debug = new PrintStream("store/target/debug.txt");
}
try {
FastbitWrapper.debug(debug);
final int rows = 1000;
final int threads = 6;
final int bitMapMaxSize = 4096;
final ConcurrentMap<Integer, Pair<AtomicInteger, Bitmap>> map =
new ConcurrentHashMap<Integer, Pair<AtomicInteger,
Bitmap>>();
ScheduledThreadPoolExecutor executor = new
ScheduledThreadPoolExecutor(threads);
for (int index = 0; index < rows; index++) {
Random random = new Random();
Bitmap bitmap = getBitmap(random, bitMapMaxSize);
map.put(index, Pair.of(new AtomicInteger(0), bitmap));
}
Runnable bitmapManipulator = new Runnable() {
@Override
public void run() {
for (int counter = 0; counter < 5000; counter++) {
Random random = new Random();
int key = random.nextInt(rows);
Bitmap bitmap = map.get(key).second;
map.get(key).first.incrementAndGet();
Bitmap tmp = getBitmap(random, bitMapMaxSize);
int op = random.nextInt(8);
switch (op) {
case 0:
bitmap.and(tmp);
break;
case 1:
bitmap.xor(tmp);
break;
case 2:
bitmap.or(tmp);
break;
case 3:
bitmap.flip();
break;
case 4:
long pos = random.nextInt(bitMapMaxSize);
bitmap.setBit(pos);
break;
case 5:
long unsetpos =
random.nextInt(bitMapMaxSize);
bitmap.unsetBit(unsetpos);
break;
case 6:
long getPos = random.nextInt(bitMapMaxSize);
bitmap.get(getPos);
break;
case 7:
bitmap.cardinality();
break;
default:
}
BitmapUtils.closeBitmap(tmp);
}
}
};
for (int counter = 0; counter < threads; counter++) {
executor.schedule(bitmapManipulator, 0,
TimeUnit.MILLISECONDS);
}
while (executor.getCompletedTaskCount() < threads) {
Thread.sleep(500);
}
for (Pair<AtomicInteger, Bitmap> pair : map.values()) {
BitmapUtils.closeBitmap(pair.second);
}
} finally {
debug.close();
}
}
private Bitmap getBitmap(Random random, long bitMapMaxSize) {
Bitmap bitmap =
BitmapFactory.getInstance().newBitmap(bitMapMaxSize);
int quarter = (int) bitMapMaxSize / 4;
int bits[] = new int[] {random.nextInt(quarter), quarter +
random.nextInt(quarter),
quarter * 2 + random.nextInt(quarter)};
bitmap.setBit(bits[0]);
bitmap.setBit(bits[1]);
bitmap.setBit(bits[2]);
return bitmap;
}
public final class BitmapFactory {
private static final BitmapFactory INSTANCE = new BitmapFactory();
private BitmapFactory() {
}
public static BitmapFactory getInstance() {
return INSTANCE;
}
public Bitmap newBitmap(long bits) {
return new FastbitWrapper(bits);
}
}
On Tue, May 7, 2013 at 12:45 PM, K. John Wu <[email protected]> wrote:
> Hi, Srikanth,
>
> Thanks for the information. I suspect that the problem is
> ibis::bitvector64::setBit, but can not spot the problem with a code
> review. So it looks like I will have to dig a little deeper.
>
> Sounds you are using a test program, not involving any sensitive data.
> Would you mind share that the test code with me?
>
> John
>
>
> On 5/6/13 11:48 PM, Srikanth Sundarrajan wrote:
> > Thanks much for looking into this so quickly.
> >
> > This is part of a unit test to verify the thread safety of the
> > bitvector class in java. The unit test essentially creates a fixed
> > number of bitvector 64 objects each of size 4096 bits through the
> > function below.
> >
> > JNIEXPORT jlong JNICALL
> > Java_com_inmobi_shadowfax_FastbitWrapper_IcreateBitset
> > (JNIEnv *env, jobject obj, jlong numbits) {
> >
> > ibis::bitvector64* bitsetptr = new ibis::bitvector64();
> > bitsetptr->setBit((long)numbits, 0);
> >
> > return (long)bitsetptr;
> > }
> >
> > and 3 random bits are set using the following set method
> >
> > JNIEXPORT void JNICALL Java_com_inmobi_shadowfax_FastbitWrapper_IsetBit
> > (JNIEnv *env, jobject obj, jlong bitsetptr, jlong pos) {
> >
> > ((ibis::bitvector64*)bitsetptr)->setBit(pos, 1);
> >
> > }
> >
> > on this bit map a series of bit operations are performed from parallel
> > threads in java
> >
> > Implementations of and/or/xor are given below
> >
> > JNIEXPORT void JNICALL Java_com_inmobi_shadowfax_FastbitWrapper_Iand
> > (JNIEnv *env, jobject obj, jlong targetbitsetptr, jlong sourcebitsetptr)
> {
> >
> ((ibis::bitvector64*)targetbitsetptr)->operator&=(*(ibis::bitvector64*)sourcebitsetptr);
> > }
> >
> > JNIEXPORT void JNICALL Java_com_inmobi_shadowfax_FastbitWrapper_Ior
> > (JNIEnv *env, jobject obj, jlong targetbitsetptr, jlong sourcebitsetptr)
> {
> >
> ((ibis::bitvector64*)targetbitsetptr)->operator|=(*(ibis::bitvector64*)sourcebitsetptr);
> > }
> >
> >
> > JNIEXPORT void JNICALL Java_com_inmobi_shadowfax_FastbitWrapper_Ixor
> > (JNIEnv *env, jobject obj, jlong targetbitsetptr, jlong sourcebitsetptr)
> {
> >
> ((ibis::bitvector64*)targetbitsetptr)->operator^=(*(ibis::bitvector64*)sourcebitsetptr);
> > }
> >
> >
> > Regards
> > Srikanth Sundarrajan
> >
> >
> > On Tue, May 7, 2013 at 12:13 PM, K. John Wu <[email protected]
> > <mailto:[email protected]>> wrote:
> >
> > Hi, Srikanth,
> >
> > I can verify that the problem was caused by those two unexpected
> > words. This appears to be a flaw in the design of the compression
> > scheme that left some special cases unhandled. Since it would
> > introduce extra cost to handle this special case, it might be
> > worthwhile to figure out how to prevent it being generated in the
> > first place. Would you be willing to explain how you generate those
> > bitvector64 objects?
> >
> > Thanks.
> >
> > John
> >
> >
> >
> > On 5/6/13 10:22 PM, Srikanth Sundarrajan wrote:
> > > Hi,
> > > I am fairly new to fasbit and have been using it for around a
> > > month. Have been running into an issue with or_c2 & xor_c2 when
> run
> > > at scale from parallel threads (am using fastbit lib via jni
> > from java).
> > >
> > > The error looks something like
> > >
> > > ERROR bitvector64::or_c2 -- serious problem here ...
> > > terminate called throwing an exception
> > > Error -- bitvector64::or_c2 expects to exhaust i0 but there are
> -159
> > > word(s) left
> > >
> > > I tried adding print statement to print the copy (deep) of the
> > > original lhs & rhs value of the bitmaps and looks like the bitmaps
> > > weren't mutated while the function was running.
> > >
> > > lhs - copy of original
> > >
> > > This bitvector64 stores 4095 bits of a 4097-bit (25 set)
> > sequence in a
> > > 36-word array and 2 bits in the active word
> > > 0 0 1 1 2 2 3 3 4 4 5 5 6
> > > 012345678901234567890123456789012345678901234567890123456789012
> > > ---------------------------------------------------------------
> > >
> >
> 00000000000000000000000000000000000000000000000000000000000000000000000000000000
> > >
> >
> 14000000000000000100000000000000000000000000000000000000000000000000000000000000
> > > 28000000000000002126*0
> > >
> >
> 30000000000000040000000000000000000000000000000000000000000000000000000001000000
> > > 4800000000000000163*0
> > >
> >
> 50020000000000000000000000100000000000000000000000000000000000000000000000000000
> > >
> >
> 60004000000000000000000000000100000000000000000000000000000000000000000000000000
> > >
> >
> 70000001000000000000000000000000000000000001000000000000000000000000000000000000
> > >
> >
> 80000002000000044000000000000000000000000010000000000000000000000000000001000100
> > > 980000000000000000*0
> > >
> >
> 100000001000000000000000000000000000000000001000000000000000000000000000000000000
> > > 118000000000000004252*0
> > >
> >
> 120000400000000000000000000000000010000000000000000000000000000000000000000000000
> > > 138000000000000002126*0
> > >
> >
> 140000008000000000000000000000000000000001000000000000000000000000000000000000000
> > >
> >
> 150040000000000020000000001000000000000000000000000000000000000000000000000100000
> > > 1680000000000000000*0
> > >
> >
> 170000000000080000000000000000000000000000000000000000000000010000000000000000000
> > > 18800000000000000163*0
> > >
> >
> 190000000000102000000000000000000000000000000000000000000000100000010000000000000
> > >
> >
> 200000000000000000000000000000000000000000000000000000000000000000000000000000000
> > >
> >
> 210000000000800000000000000000000000000000000000000000000100000000000000000000000
> > > 228000000000000002126*0
> > >
> >
> 230000000002000000000000000000000000000000000000000000010000000000000000000000000
> > > 248000000000000004252*0
> > >
> >
> 250000000100000000000000000000000000000000000000100000000000000000000000000000000
> > > 268000000000000002126*0
> > >
> >
> 270000000000020000000000000000000000000000000000000000000000000100000000000000000
> > > 28800000000000000163*0
> > >
> >
> 290000000000001800000000000000000000000000000000000000000000000000001100000000000
> > >
> >
> 300000000004000000000000000000000000000000000000000000100000000000000000000000000
> > > 31800000000000000163*0
> > >
> >
> 320000000080000000000000000000000000000000000000010000000000000000000000000000000
> > > 338000000000000006378*0
> > >
> >
> 340000010000000000000000000000000000000010000000000000000000000000000000000000000
> > > 3580000000000000111071*0
> > > 000000000000000000
> > >
> > > lhs - original
> > >
> > > This bitvector64 stores 4095 bits of a 4097-bit (25 set)
> > sequence in a
> > > 36-word array and 2 bits in the active word
> > > 0 0 1 1 2 2 3 3 4 4 5 5 6
> > > 012345678901234567890123456789012345678901234567890123456789012
> > > ---------------------------------------------------------------
> > >
> >
> 00000000000000000000000000000000000000000000000000000000000000000000000000000000
> > >
> >
> 14000000000000000100000000000000000000000000000000000000000000000000000000000000
> > > 28000000000000002126*0
> > >
> >
> 30000000000000040000000000000000000000000000000000000000000000000000000001000000
> > > 4800000000000000163*0
> > >
> >
> 50020000000000000000000000100000000000000000000000000000000000000000000000000000
> > >
> >
> 60004000000000000000000000000100000000000000000000000000000000000000000000000000
> > >
> >
> 70000001000000000000000000000000000000000001000000000000000000000000000000000000
> > >
> >
> 80000002000000044000000000000000000000000010000000000000000000000000000001000100
> > > 980000000000000000*0
> > >
> >
> 100000001000000000000000000000000000000000001000000000000000000000000000000000000
> > > 118000000000000004252*0
> > >
> >
> 120000400000000000000000000000000010000000000000000000000000000000000000000000000
> > > 138000000000000002126*0
> > >
> >
> 140000008000000000000000000000000000000001000000000000000000000000000000000000000
> > >
> >
> 150040000000000020000000001000000000000000000000000000000000000000000000000100000
> > > 1680000000000000000*0
> > >
> >
> 170000000000080000000000000000000000000000000000000000000000010000000000000000000
> > > 18800000000000000163*0
> > >
> >
> 190000000000102000000000000000000000000000000000000000000000100000010000000000000
> > >
> >
> 200000000000000000000000000000000000000000000000000000000000000000000000000000000
> > >
> >
> 210000000000800000000000000000000000000000000000000000000100000000000000000000000
> > > 228000000000000002126*0
> > >
> >
> 230000000002000000000000000000000000000000000000000000010000000000000000000000000
> > > 248000000000000004252*0
> > >
> >
> 250000000100000000000000000000000000000000000000100000000000000000000000000000000
> > > 268000000000000002126*0
> > >
> >
> 270000000000020000000000000000000000000000000000000000000000000100000000000000000
> > > 28800000000000000163*0
> > >
> >
> 290000000000001800000000000000000000000000000000000000000000000000001100000000000
> > >
> >
> 300000000004000000000000000000000000000000000000000000100000000000000000000000000
> > > 31800000000000000163*0
> > >
> >
> 320000000080000000000000000000000000000000000000010000000000000000000000000000000
> > > 338000000000000006378*0
> > >
> >
> 340000010000000000000000000000000000000010000000000000000000000000000000000000000
> > > 3580000000000000111071*0
> > > 000000000000000000
> > >
> > > rhs - copy of original
> > >
> > > This bitvector64 stores 4095 bits of a 4097-bit (3 set) sequence
> > in a
> > > 7-word array and 2 bits in the active word
> > > 0 0 1 1 2 2 3 3 4 4 5 5 6
> > > 012345678901234567890123456789012345678901234567890123456789012
> > > ---------------------------------------------------------------
> > > 0800000000000000e882*0
> > >
> >
> 10000000000040000000000000000000000000000000000000000000000001000000000000000000
> > > 28000000000000005315*0
> > >
> >
> 30000010000000000000000000000000000000010000000000000000000000000000000000000000
> > > 4800000000000001a1638*0
> > >
> >
> 50000080000000000000000000000000000010000000000000000000000000000000000000000000
> > > 680000000000000111071*0
> > > 000000000000000000
> > >
> > > rhs - original
> > >
> > > This bitvector64 stores 4095 bits of a 4097-bit (3 set) sequence
> > in a
> > > 7-word array and 2 bits in the active word
> > > 0 0 1 1 2 2 3 3 4 4 5 5 6
> > > 012345678901234567890123456789012345678901234567890123456789012
> > > ---------------------------------------------------------------
> > > 0800000000000000e882*0
> > >
> >
> 10000000000040000000000000000000000000000000000000000000000001000000000000000000
> > > 28000000000000005315*0
> > >
> >
> 30000010000000000000000000000000000000010000000000000000000000000000000000000000
> > > 4800000000000001a1638*0
> > >
> >
> 50000080000000000000000000000000000010000000000000000000000000000000000000000000
> > > 680000000000000111071*0
> > > 000000000000000000
> > >
> > > Any guiding pointers are appreciated.
> > >
> > > Regards
> > > Srikanth Sundarrajan
> > >
> > >
> > >
> > > _____________________________________________________________
> > > The information contained in this communication is intended
> > solely for
> > > the use of the individual or entity to whom it is addressed and
> > others
> > > authorized to receive it. It may contain confidential or legally
> > > privileged information. If you are not the intended recipient
> > you are
> > > hereby notified that any disclosure, copying, distribution or
> taking
> > > any action in reliance on the contents of this information is
> > strictly
> > > prohibited and may be unlawful. If you have received this
> > > communication in error, please notify us immediately by
> > responding to
> > > this email and then delete it from your system. The firm is neither
> > > liable for the proper and complete transmission of the information
> > > contained in this communication nor for any delay in its receipt.
> > >
> > >
> > > _______________________________________________
> > > FastBit-users mailing list
> > > [email protected] <mailto:[email protected]>
> > > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
> > >
> >
> >
> >
> > _____________________________________________________________
> > The information contained in this communication is intended solely for
> > the use of the individual or entity to whom it is addressed and others
> > authorized to receive it. It may contain confidential or legally
> > privileged information. If you are not the intended recipient you are
> > hereby notified that any disclosure, copying, distribution or taking
> > any action in reliance on the contents of this information is strictly
> > prohibited and may be unlawful. If you have received this
> > communication in error, please notify us immediately by responding to
> > this email and then delete it from your system. The firm is neither
> > liable for the proper and complete transmission of the information
> > contained in this communication nor for any delay in its receipt.
>
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.
_______________________________________________
FastBit-users mailing list
[email protected]
https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
