Hi, Mian, We are able to confirm that the function ibis::fileManager::unload (which printed the warning message) was indeed performing a number of computations in 32-bit arithmetic. They will be changed to use size_t in the coming releases. This should address the problem you've seen.
We are in the process of preparing the next release. Due to the extensive changes involved, it might take some time before we are done with all the testing.. John On 10/25/2009 8:28 PM, Mian Lu wrote: > Hi John: > > I have performed the same evaluation multiple times, but it always > produced similar warnings. I observed that the actual memory consumed is > about half (4G, I have total 8G memory). Moreover, I wonder what happened > if the necessary bit vectors cannot fit into the memory since I have found > that in sumBits these functions, you need to activate all necessary bit > vectors. Can I also adjust the memory size assigned to Fastbit? e.g., I > want Fastbit to use as much as possible memory for large data sets. > > Here are some information about my machine. I only use configure and make > commands to build the Fastbit without any modifications: > > OS: CentOS 5.2, the kernel Linux version is 2.6.18-92.e15. > Memory size: 8 GB > size_t: 8 bytes > long int: 8 bytes > int: 4 bytes > > Regards, > Mian > >> Hi, Mian, >> >> Thanks for the information. The warning messages appear odd to me. >> For example, the 1st warning message says that the maximum allowed >> memory (maxBytes) is about 4GB, the amount of memory used currently >> (totalBytes) is only 33 MB, but it can not find enough space for 628 >> MB. From what I can see there are plenty of free memory available -- >> there is probably something computed in 32-bit integer arithmetic that >> is causing this problem. It would be helpful for us to get some >> information about your machine and compiler in order to help us locate >> the actual cause of these erroneous warnings, more specifically, your >> OS version, memory size, sizes of size_t, long int, and int. >> >> The reason that FastBit can still produce the expected answer because >> there are fall-back options in many cases. >> >> John >> >> >> On 10/25/2009 9:27 AM, Mian Lu wrote: >>> Actually when I perform an evaluation for about 15 attributes (each >>> attribute size is totally 1GB, and the selectivity for each attribute is >>> 40%), I will receive some warnings like this: >>> >>> Warning -- fileManager::unload unable to free enough space for 628021720 >>> bytes (totalBytes=33032688, maxBytes=4182255616) >>> Warning -- fileManager::storage is unable to find 628021720 bytes of >>> space >>> to read file descriptor 3 >>> Sat Oct 24 19:38:29 2009 >>> Warning -- column[test2.attr12](INT)::estimateRange -- received a >>> std::exception -- storage::ctor(read file) failed >>> Warning -- fileManager::unload unable to free enough space for 628059044 >>> bytes (totalBytes=66064972, maxBytes=4182255616) >>> Warning -- fileManager::storage is unable to find 628059044 bytes of >>> space >>> to read file descriptor 4 >>> Sat Oct 24 19:40:06 2009 >>> Warning -- column[test2.attr0](INT)::estimateRange -- received a >>> std::exception -- storage::ctor(read file) failed >>> ...... >>> >>> But the final result is correct. I guess Fastbit does has some memory >>> limitations for efficiency reasons. Moreover, I also want to know the >>> exact answers. Thanks. >>> >>> Mian >>> >>> >>>> Hi guys, >>>> >>>> I am a newbie of fastbit. I do not know whether bitmap index is loaded >>>> by >>>> fastbit into memory. if so, then the size of the bitmap index will be >>>> limited by the memory size of that machine runs fastbit. Can fastbit >>>> overcome that problem? If it can't, how many records at most can >>>> fastbit >>>> build bitmap indices on them? >>>> >>>> >>>> Thanks, >>>> Min >>>> -- >>>> My research interests are distributed systems, parallel computing and >>>> bytecode based virtual machine. >>>> >>>> My profile: >>>> http://www.linkedin.com/in/coderplay >>>> My blog: >>>> http://coderplay.javaeye.com >>>> _______________________________________________ >>>> FastBit-users mailing list >>>> [email protected] >>>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>>> >>> >>> _______________________________________________ >>> FastBit-users mailing list >>> [email protected] >>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >> _______________________________________________ >> FastBit-users mailing list >> [email protected] >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >> > > > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
