Hi, Jilong,
Thanks for the test code. It is very helpful for me to figure out the
source of the problem. A fix has been checked in as SVN revision 658.
Attached is a slightly expanded test case along with its expected
output. Please give this fix a try and let me know if you encounter
any problems.
Great detective work.
John
On 10/15/13 7:36 AM, wang jilong wrote:
> Dear John,
>
> Thanks for your reply. The following test program should expose the issue:
>
> ////////////////////////////////////////////////////////////////////////
> #include <stdio.h>
>
> #include "ibis.h"
> #include "array_t.h"
> #include "bitvector.h"
>
> using namespace std;
>
> typedef ibis::array_t<ibis::bitvector::word_t> ARRAY_WORD;
> typedef ibis::bitvector::word_t WORD_T;
>
> int main(int argc, char** argv)
> {
> WORD_T v_arr_word[3];
> v_arr_word[0] = 0xC014B9BF;
> v_arr_word[1] = 0x00007FFF;
> v_arr_word[2] = 0x0000000F;
>
> ARRAY_WORD v_array_word(v_arr_word, sizeof(v_arr_word)/sizeof(WORD_T));
>
> // the v-bitvect represents an bitmaps (size=42106416, cnt=42106416,
> bytes=44, isCompressed=0)
> ibis::bitvector v_bitvect(v_array_word);
>
> ibis::bitvector *v_ptr_bitvect_ret = v_bitvect & v_bitvect;
> // the bitmap pointed by v_ptr_bitvect_ret should be same as v_bitvect
> // the actual result is: size=42106416, cnt=42106416, bytes=40,
> isCompressed=1
> // if iterating it using "for (ibis::bitvector::const_iterator v_iter =
> p_ptr_bitvector->begin(); v_iter != p_ptr_bitvector->end(); ++v_iter)",
> // only to see what is inside. It is 111111111111111, up to 15 bits
>
> // if printed word-wise, it is: xFF\x7F\x00\x00\x0F\x00\x00\x00
>
> delete v_ptr_bitvect_ret;
>
> return 0;
> }
>
> ////////////////////////////////////////////////////////////////////////
>
> Thanks
>
> Jilong Wang
>
> -----Original Message-----
> From: [email protected]
> [mailto:[email protected]] On Behalf Of K. John Wu
> Sent: Tuesday, October 08, 2013 10:13 AM
> To: FastBit Users
> Subject: Re: [FastBit-users] fastbit bitvector AND operation result does not
> seem correct
>
> Dear Jilong,
>
> Thanks for your interest in FastBit. It will take me some time to
> reconstruct a test program from your description. Would you mind share you
> test program with me?
>
> By the way, would you be willing to describe your use case? It is usually
> better to avoid iterate through all bit.
>
> John
>
>
> On 10/7/13 2:23 PM, wang jilong wrote:
>> Hi,
>>
>>
>>
>> I am new to the mailing list. It is about fastbit 1.3.6 and 1.3.7 on a
>> linux machine.
>>
>>
>>
>> I am "AND-ing" two bitvectors together.
>>
>>
>>
>> The 1^st vector: v_vect1, original bitvector, size=42106416,
>> cnt=42106416, bytes=44, isCompressed=0
>>
>> Its bytes: \xBF\xB9\x14\xC0\xFF\x7F\x00\x00\x0F\x00\x00\x00
>>
>> So the words are: C014B9BF 00007FFF 0000000F
>>
>>
>>
>> The 2^nd vector: size=42106416, cnt=2922440, bytes=3512672,
>> isCompressed=0
>>
>>
>>
>> The result vector: size=42106416, cnt=2922440, bytes=40,
>> isCompressed=1
>>
>>
>>
>> Question 1:
>>
>> when I iterate through the result vector using
>>
>> "for (ibis::bitvector::const_iterator v_iter =
>> p_ptr_bitvector->begin(); v_iter != p_ptr_bitvector->end();
>> ++v_iter)",
>>
>> It only runs up to 15 bits.
>>
>>
>>
>> Is this a expected behavior for iteration?
>>
>>
>>
>> Question 2:
>>
>> The result vector as it is written to words and printed as bytes.
>>
>> They (bytes sequence) are: \x00\x00\x00\x00\x0F\x00\x00\x00
>>
>>
>>
>> When the result vector is decompressed, it has long byte sequence.
>>
>>
>>
>> I read your thesis about WAH, had difficult to interpret the bitmap above.
>>
>> Can you please help?
>>
>>
>>
>> Question 3:
>>
>>
>>
>> If the result vector is stored as bytes sequence and uses it later in
>> fastbit again,
>>
>> Say engaged in BIT-AND operation, the bytes-sequence is treated as
>> un-compressed.
>>
>>
>>
>> How can I make it treated as compressed, like when right after the
>> result-vector is generated?
>>
>>
>>
>> Thanks
>>
>>
>>
>> Jilong Wang
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> FastBit-users mailing list
>> [email protected]
>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
>>
> _______________________________________________
> FastBit-users mailing list
> [email protected]
> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
> _______________________________________________
> FastBit-users mailing list
> [email protected]
> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
>
/// A test case reported by Wang JiLong <wang.jilong at huawei.com>,
/// on 10/7/2013, original email message at
/// <https://hpcrdm.lbl.gov/pipermail/fastbit-users/2013-October/001810.html>
///
#include "ibis.h"
#include "array_t.h"
#include "bitvector.h"
typedef ibis::array_t<ibis::bitvector::word_t> ARRAY_WORD;
typedef ibis::bitvector::word_t WORD_T;
int main(int argc, char** argv)
{
WORD_T v_arr_word[3];
v_arr_word[0] = 0xC014B9BF;
v_arr_word[1] = 0x00007FFF;
v_arr_word[2] = 0x0000000F;
ARRAY_WORD v_array_word(v_arr_word, sizeof(v_arr_word)/sizeof(WORD_T));
ibis::bitvector v_bitvect(v_array_word);
std::cout << "input bitvector (v_bitvect)\n" << v_bitvect << std::endl;
ibis::bitvector *v_ret = v_bitvect & v_bitvect;
std::cout << "\noutput of v_bitvect & v_bitvect\n" << *v_ret
<< std::endl;
*v_ret -= v_bitvect;
if (v_ret->cnt() == 0 && v_ret->size() == v_bitvect.size())
std::cout << "Operator& produced the expected result" << std::endl;
else
std::cout << "Warning -- Operator& did not produce the expected result"
<< std::endl;
delete v_ret;
v_ret = v_bitvect | v_bitvect;
std::cout << "\n\noutput of v_bitvect | v_bitvect\n" << *v_ret
<< std::endl;
*v_ret -= v_bitvect;
if (v_ret->cnt() == 0 && v_ret->size() == v_bitvect.size())
std::cout << "Operator| produced the expected result" << std::endl;
else
std::cout << "Warning -- Operator| did not produce the expected result"
<< std::endl;
delete v_ret;
v_ret = v_bitvect ^ v_bitvect;
std::cout << "\n\noutput of v_bitvect ^ v_bitvect\n" << *v_ret
<< std::endl;
if (v_ret->cnt() == 0 && v_ret->size() == v_bitvect.size())
std::cout << "Operator^ produced the expected result" << std::endl;
else
std::cout << "Warning -- Operator^ did not produce the expected result"
<< std::endl;
delete v_ret;
return 0;
}
input bitvector (v_bitvect)
This bitvector stores 42106401 bits of a 42106416-bit (42106416 set) sequence
in a 1-word array and 15 bits in the active word
0 1 1 2 2 3
0123456789012345678901234567890
-------------------------------
0 c014b9bf 42106401*1
7fff0000 111111111111111
output of v_bitvect & v_bitvect
This bitvector stores 42106401 bits of a 42106416-bit (42106416 set) sequence
in a 1-word array and 15 bits in the active word
0 1 1 2 2 3
0123456789012345678901234567890
-------------------------------
0 c014b9bf 42106401*1
7fff0000 111111111111111
Operator& produced the expected result
output of v_bitvect | v_bitvect
This bitvector stores 42106401 bits of a 42106416-bit (42106416 set) sequence
in a 1-word array and 15 bits in the active word
0 1 1 2 2 3
0123456789012345678901234567890
-------------------------------
0 c014b9bf 42106401*1
7fff0000 111111111111111
Operator| produced the expected result
output of v_bitvect ^ v_bitvect
This bitvector stores 42106401 bits of a 42106416-bit (0 set) sequence in a
1-word array and 15 bits in the active word
0 1 1 2 2 3
0123456789012345678901234567890
-------------------------------
0 8014b9bf 42106401*0
00000000 000000000000000
Operator^ produced the expected result
_______________________________________________
FastBit-users mailing list
[email protected]
https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
