Hi, Yifeng, Thanks for your interest in our software. The most likely reason for this problem is that the interval-equality index is very large. To estimate how large the index is, you pretty much have to assume all the bitmaps are incompressible for an interval-equality index. In this case, you might need to have 100GB of memory.
I would suggest that you break you 1-billion-row dataset into 100 pieces of 10-million-row each. Depending on your queries, another binning option or another encoding option might be more appropriate. John On 12/4/13, 11:39 PM, 耿益锋 wrote: > Hi, I want to build an index with <binning none/><encoding > interval-equality/> on an 8G double column named v1 but it fails. It > can work with <binning nbins=2000/><encoding interval-equality/>. So > what may be the problem? > > > The log is as follows. > > ibis -v -d fr2048 -b "<binning none/><encoding interval-equality/>" > > Constructed a part named fr2048 > /home/yifeng/lib/bin/ibis: start building indexes (nthreads=1, > indexingOption= <binning none/><encoding interval-equality/>) ... > index::create create an empty index for d1 > Warning -- index[fr2048.d1]::create failed to create an index of type > <binning none/><encoding interval-equality/> > index::create create an empty index for d1 > Warning -- index[fr2048.d1]::create failed to create an index of type > <binning none/><encoding interval-equality/> > index::create create an empty index for d2 > Warning -- index[fr2048.d2]::create failed to create an index of type > <binning none/><encoding interval-equality/> > index::create create an empty index for d2 > Warning -- index[fr2048.d2]::create failed to create an index of type > <binning none/><encoding interval-equality/> > index::create create an empty index for d3 > Warning -- index[fr2048.d3]::create failed to create an index of type > <binning none/><encoding interval-equality/> > index::create create an empty index for d3 > Warning -- index[fr2048.d3]::create failed to create an index of type > <binning none/><encoding interval-equality/> > index::create create an empty index for v1 > Warning -- index[fr2048.v1]::create failed to create an index of type > <binning none/><encoding interval-equality/> > index::create create an empty index for v1 > Warning -- index[fr2048.v1]::create failed to create an index of type > <binning none/><encoding interval-equality/> > part[fr2048]::buildIndexes processed 4 columns using 1 thread took 0 > CPU seconds and 0.00066901 elapsed seconds > /home/yifeng/lib/bin/ibis: building indexes for 1 data partition took > 0.001 CPU seconds, 0.0100073 elapsed seconds > > /home/yifeng/lib/bin/ibis -- total CPU time 0.001 s, total elapsed > time 0.0340256 s > fileManager::clear -- completed with 56 bytes of storage remain in > memory after removing all managed objects > fileManager::clear -- completed with 56 bytes of storage remain in > memory after removing all managed objects > > Thanks, > Yifeng. > > -- > 耿益锋 > Yifeng Geng > > 清华大学计算机科学与技术系 > 高性能计算研究所 > HPC Research Center, > Department of Computer Science & Technology, > Tsinghua University, Beijing, P.R.China. > > > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
