Hi, Mian, By default, FastBit's memory manager (actually called ibis::fileManager) uses half of the physical memory available. The way to change this default is to provide a run-time parameter file. Including the following line
fileManager::maxBytes = 6GB will increase the maximum amount of memory to 6GB instead of the default 0.5*8GB. The name of the parameter file can be passed to ibis::init. Note that you need to call ibis::init before calling any other FastBit functions in order to guarantee the parameter maxBytes will have any effects. (Of course, the really requirement is to make sure the parameter file is read before the constructor of ibis::fileManager is called if you want to avoid calling ibis::init). Another note that might be worth mentioning is that those static functions addBits and sumBits are somewhat suboptimal because the optimal decision that minimizes the cost of bitwise logical operations should be made based on the volume of the compressed bitmaps as oppose to the number of bitmaps (see our TODS paper for analysis http://lbl.gov/%7Ekwu/ps/LBNL-49626.html). This optimality is not only true for I/O cost, but also for CPU cost. However, those static function can not access the bitmap size information because the size information (represented by member variable ibis::index::offsets) is not accessible to the static functions. This may not make a big difference, but is something to be aware of especially since you seem to be on a track to do a serious comparison study. Now that I have mentioned ibis::index::offsets, I need to tell you that in the current version of code, it is an array of 32-bit signed integers. Therefore, the index files can not be more than 2GB in size! You have to break the data into smaller partitions if the index size is going to be more than 2GB. John On 10/25/2009 8:28 PM, Mian Lu wrote: > Hi John: > > I have performed the same evaluation multiple times, but it always > produced similar warnings. I observed that the actual memory consumed is > about half (4G, I have total 8G memory). Moreover, I wonder what happened > if the necessary bit vectors cannot fit into the memory since I have found > that in sumBits these functions, you need to activate all necessary bit > vectors. Can I also adjust the memory size assigned to Fastbit? e.g., I > want Fastbit to use as much as possible memory for large data sets. > > Here are some information about my machine. I only use configure and make > commands to build the Fastbit without any modifications: > > OS: CentOS 5.2, the kernel Linux version is 2.6.18-92.e15. > Memory size: 8 GB > size_t: 8 bytes > long int: 8 bytes > int: 4 bytes > > Regards, > Mian > >> Hi, Mian, >> >> Thanks for the information. The warning messages appear odd to me. >> For example, the 1st warning message says that the maximum allowed >> memory (maxBytes) is about 4GB, the amount of memory used currently >> (totalBytes) is only 33 MB, but it can not find enough space for 628 >> MB. From what I can see there are plenty of free memory available -- >> there is probably something computed in 32-bit integer arithmetic that >> is causing this problem. It would be helpful for us to get some >> information about your machine and compiler in order to help us locate >> the actual cause of these erroneous warnings, more specifically, your >> OS version, memory size, sizes of size_t, long int, and int. >> >> The reason that FastBit can still produce the expected answer because >> there are fall-back options in many cases. >> >> John >> >> >> On 10/25/2009 9:27 AM, Mian Lu wrote: >>> Actually when I perform an evaluation for about 15 attributes (each >>> attribute size is totally 1GB, and the selectivity for each attribute is >>> 40%), I will receive some warnings like this: >>> >>> Warning -- fileManager::unload unable to free enough space for 628021720 >>> bytes (totalBytes=33032688, maxBytes=4182255616) >>> Warning -- fileManager::storage is unable to find 628021720 bytes of >>> space >>> to read file descriptor 3 >>> Sat Oct 24 19:38:29 2009 >>> Warning -- column[test2.attr12](INT)::estimateRange -- received a >>> std::exception -- storage::ctor(read file) failed >>> Warning -- fileManager::unload unable to free enough space for 628059044 >>> bytes (totalBytes=66064972, maxBytes=4182255616) >>> Warning -- fileManager::storage is unable to find 628059044 bytes of >>> space >>> to read file descriptor 4 >>> Sat Oct 24 19:40:06 2009 >>> Warning -- column[test2.attr0](INT)::estimateRange -- received a >>> std::exception -- storage::ctor(read file) failed >>> ...... >>> >>> But the final result is correct. I guess Fastbit does has some memory >>> limitations for efficiency reasons. Moreover, I also want to know the >>> exact answers. Thanks. >>> >>> Mian >>> >>> >>>> Hi guys, >>>> >>>> I am a newbie of fastbit. I do not know whether bitmap index is loaded >>>> by >>>> fastbit into memory. if so, then the size of the bitmap index will be >>>> limited by the memory size of that machine runs fastbit. Can fastbit >>>> overcome that problem? If it can't, how many records at most can >>>> fastbit >>>> build bitmap indices on them? >>>> >>>> >>>> Thanks, >>>> Min >>>> -- >>>> My research interests are distributed systems, parallel computing and >>>> bytecode based virtual machine. >>>> >>>> My profile: >>>> http://www.linkedin.com/in/coderplay >>>> My blog: >>>> http://coderplay.javaeye.com >>>> _______________________________________________ >>>> FastBit-users mailing list >>>> [email protected] >>>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>>> >>> >>> _______________________________________________ >>> FastBit-users mailing list >>> [email protected] >>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >> _______________________________________________ >> FastBit-users mailing list >> [email protected] >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >> > > > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
