Hi, Matt, Thanks for your interest in FastBit software.
The messages from write64 both indicated that 2147479552 bytes are to be written, the number of bytes corresponds to 374,471,226 doubles (8-bytes per entry). However, the I/O function write somehow did not want to write that many bytes. In both cases, only 2147479552 bytes are written - because this number is less than 2**31, I guess the I/O function actually wrote that many bytes and the return value is not a negative number indicating errors. You are likely using an I/O library without large file support. To fix this problem, please link with the 64-bit version of your I/O library. You should be able to search on "large file support" and your OS name to find such information. The function write64 is from ibis::relic class (in file src/irelic.cpp, you can find the exact location of this warning message by looking for "return -8"). The particular statement was involving writing the keyvalues of the index class. It indicates that value1 and value2 each has 374 million distinct values. Corresponding to each distinct value, there will be a bitmap. I don't know exactly what you plan to do with the indexes, but having to deal with 374 million bitmaps in order to answer a query might be time-consuming. I suggest that you look into alternative indexing options such as "<binning precision=2/>". Of course, you could also break up your data into more partitions as was suggested earlier. Good luck. John On 8/1/12 1:39 PM, Matt Bomhoff wrote: > Hello, > > I am getting an I/O error when trying to index an 11GB database. > > The ibis command prints warnings: > ibis -d db_rice_1_1_0_0 -v -b "<binning none/><encoding equality/>" > > Warning -- relic[db_rice_1_1_0_0.value1]::write64 expected to write > 2995769808 bytes to file descriptor 8, but actually wrote 2147479552 > Warning -- index[db_rice_1_1_0_0.value1]::create failed to write the > index (basic) to db_rice_1_1_0_0/value1.idx, ierr = -8 > Warning -- relic[db_rice_1_1_0_0.value2]::write64 expected to write > 2995769808 bytes to file descriptor 9, but actually wrote 2147479552 > Warning -- index[db_rice_1_1_0_0.value2]::create failed to write the > index (basic) to db_rice_1_1_0_0/value2.idx, ierr = -8 > > It appears that write64() call is not able to write >2GB. Has anyone > seen this before? I am guessing the problem is not in ibis code, but > more likely a compilation or system config issue. > > Thanks! > Matt > > > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
