Hi John: I have performed the same evaluation multiple times, but it always produced similar warnings. I observed that the actual memory consumed is about half (4G, I have total 8G memory). Moreover, I wonder what happened if the necessary bit vectors cannot fit into the memory since I have found that in sumBits these functions, you need to activate all necessary bit vectors. Can I also adjust the memory size assigned to Fastbit? e.g., I want Fastbit to use as much as possible memory for large data sets.
Here are some information about my machine. I only use configure and make commands to build the Fastbit without any modifications: OS: CentOS 5.2, the kernel Linux version is 2.6.18-92.e15. Memory size: 8 GB size_t: 8 bytes long int: 8 bytes int: 4 bytes Regards, Mian > Hi, Mian, > > Thanks for the information. The warning messages appear odd to me. > For example, the 1st warning message says that the maximum allowed > memory (maxBytes) is about 4GB, the amount of memory used currently > (totalBytes) is only 33 MB, but it can not find enough space for 628 > MB. From what I can see there are plenty of free memory available -- > there is probably something computed in 32-bit integer arithmetic that > is causing this problem. It would be helpful for us to get some > information about your machine and compiler in order to help us locate > the actual cause of these erroneous warnings, more specifically, your > OS version, memory size, sizes of size_t, long int, and int. > > The reason that FastBit can still produce the expected answer because > there are fall-back options in many cases. > > John > > > On 10/25/2009 9:27 AM, Mian Lu wrote: >> Actually when I perform an evaluation for about 15 attributes (each >> attribute size is totally 1GB, and the selectivity for each attribute is >> 40%), I will receive some warnings like this: >> >> Warning -- fileManager::unload unable to free enough space for 628021720 >> bytes (totalBytes=33032688, maxBytes=4182255616) >> Warning -- fileManager::storage is unable to find 628021720 bytes of >> space >> to read file descriptor 3 >> Sat Oct 24 19:38:29 2009 >> Warning -- column[test2.attr12](INT)::estimateRange -- received a >> std::exception -- storage::ctor(read file) failed >> Warning -- fileManager::unload unable to free enough space for 628059044 >> bytes (totalBytes=66064972, maxBytes=4182255616) >> Warning -- fileManager::storage is unable to find 628059044 bytes of >> space >> to read file descriptor 4 >> Sat Oct 24 19:40:06 2009 >> Warning -- column[test2.attr0](INT)::estimateRange -- received a >> std::exception -- storage::ctor(read file) failed >> ...... >> >> But the final result is correct. I guess Fastbit does has some memory >> limitations for efficiency reasons. Moreover, I also want to know the >> exact answers. Thanks. >> >> Mian >> >> >>> Hi guys, >>> >>> I am a newbie of fastbit. I do not know whether bitmap index is loaded >>> by >>> fastbit into memory. if so, then the size of the bitmap index will be >>> limited by the memory size of that machine runs fastbit. Can fastbit >>> overcome that problem? If it can't, how many records at most can >>> fastbit >>> build bitmap indices on them? >>> >>> >>> Thanks, >>> Min >>> -- >>> My research interests are distributed systems, parallel computing and >>> bytecode based virtual machine. >>> >>> My profile: >>> http://www.linkedin.com/in/coderplay >>> My blog: >>> http://coderplay.javaeye.com >>> _______________________________________________ >>> FastBit-users mailing list >>> [email protected] >>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>> >> >> >> _______________________________________________ >> FastBit-users mailing list >> [email protected] >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
