Hi John: Thanks for your information!
Regards, Mian > Hi, Mian, > > We are able to confirm that the function ibis::fileManager::unload > (which printed the warning message) was indeed performing a number of > computations in 32-bit arithmetic. They will be changed to use size_t > in the coming releases. This should address the problem you've seen. > > We are in the process of preparing the next release. Due to the > extensive changes involved, it might take some time before we are done > with all the testing.. > > John > > > On 10/25/2009 8:28 PM, Mian Lu wrote: >> Hi John: >> >> I have performed the same evaluation multiple times, but it always >> produced similar warnings. I observed that the actual memory consumed is >> about half (4G, I have total 8G memory). Moreover, I wonder what >> happened >> if the necessary bit vectors cannot fit into the memory since I have >> found >> that in sumBits these functions, you need to activate all necessary bit >> vectors. Can I also adjust the memory size assigned to Fastbit? e.g., I >> want Fastbit to use as much as possible memory for large data sets. >> >> Here are some information about my machine. I only use configure and >> make >> commands to build the Fastbit without any modifications: >> >> OS: CentOS 5.2, the kernel Linux version is 2.6.18-92.e15. >> Memory size: 8 GB >> size_t: 8 bytes >> long int: 8 bytes >> int: 4 bytes >> >> Regards, >> Mian >> >>> Hi, Mian, >>> >>> Thanks for the information. The warning messages appear odd to me. >>> For example, the 1st warning message says that the maximum allowed >>> memory (maxBytes) is about 4GB, the amount of memory used currently >>> (totalBytes) is only 33 MB, but it can not find enough space for 628 >>> MB. From what I can see there are plenty of free memory available -- >>> there is probably something computed in 32-bit integer arithmetic that >>> is causing this problem. It would be helpful for us to get some >>> information about your machine and compiler in order to help us locate >>> the actual cause of these erroneous warnings, more specifically, your >>> OS version, memory size, sizes of size_t, long int, and int. >>> >>> The reason that FastBit can still produce the expected answer because >>> there are fall-back options in many cases. >>> >>> John >>> >>> >>> On 10/25/2009 9:27 AM, Mian Lu wrote: >>>> Actually when I perform an evaluation for about 15 attributes (each >>>> attribute size is totally 1GB, and the selectivity for each attribute >>>> is >>>> 40%), I will receive some warnings like this: >>>> >>>> Warning -- fileManager::unload unable to free enough space for >>>> 628021720 >>>> bytes (totalBytes=33032688, maxBytes=4182255616) >>>> Warning -- fileManager::storage is unable to find 628021720 bytes of >>>> space >>>> to read file descriptor 3 >>>> Sat Oct 24 19:38:29 2009 >>>> Warning -- column[test2.attr12](INT)::estimateRange -- received a >>>> std::exception -- storage::ctor(read file) failed >>>> Warning -- fileManager::unload unable to free enough space for >>>> 628059044 >>>> bytes (totalBytes=66064972, maxBytes=4182255616) >>>> Warning -- fileManager::storage is unable to find 628059044 bytes of >>>> space >>>> to read file descriptor 4 >>>> Sat Oct 24 19:40:06 2009 >>>> Warning -- column[test2.attr0](INT)::estimateRange -- received a >>>> std::exception -- storage::ctor(read file) failed >>>> ...... >>>> >>>> But the final result is correct. I guess Fastbit does has some memory >>>> limitations for efficiency reasons. Moreover, I also want to know the >>>> exact answers. Thanks. >>>> >>>> Mian >>>> >>>> >>>>> Hi guys, >>>>> >>>>> I am a newbie of fastbit. I do not know whether bitmap index is >>>>> loaded >>>>> by >>>>> fastbit into memory. if so, then the size of the bitmap index will be >>>>> limited by the memory size of that machine runs fastbit. Can fastbit >>>>> overcome that problem? If it can't, how many records at most can >>>>> fastbit >>>>> build bitmap indices on them? >>>>> >>>>> >>>>> Thanks, >>>>> Min >>>>> -- >>>>> My research interests are distributed systems, parallel computing and >>>>> bytecode based virtual machine. >>>>> >>>>> My profile: >>>>> http://www.linkedin.com/in/coderplay >>>>> My blog: >>>>> http://coderplay.javaeye.com >>>>> _______________________________________________ >>>>> FastBit-users mailing list >>>>> [email protected] >>>>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>>>> >>>> >>>> _______________________________________________ >>>> FastBit-users mailing list >>>> [email protected] >>>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>> _______________________________________________ >>> FastBit-users mailing list >>> [email protected] >>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>> >> >> >> _______________________________________________ >> FastBit-users mailing list >> [email protected] >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
