Thanks for your explanation. So I'll use <binning nbins=2000/><encoding interval-equality/> as the index specification for large datasets.
On Fri, Dec 6, 2013 at 12:10 AM, K. John Wu <[email protected]> wrote: > Hi, Yifeng, > > Thanks for your interest in our software. The most likely reason for > this problem is that the interval-equality index is very large. To > estimate how large the index is, you pretty much have to assume all > the bitmaps are incompressible for an interval-equality index. In > this case, you might need to have 100GB of memory. > > I would suggest that you break you 1-billion-row dataset into 100 > pieces of 10-million-row each. > > Depending on your queries, another binning option or another encoding > option might be more appropriate. > > John > > > > On 12/4/13, 11:39 PM, 耿益锋 wrote: > > Hi, I want to build an index with <binning none/><encoding > > interval-equality/> on an 8G double column named v1 but it fails. It > > can work with <binning nbins=2000/><encoding interval-equality/>. So > > what may be the problem? > > > > > > The log is as follows. > > > > ibis -v -d fr2048 -b "<binning none/><encoding interval-equality/>" > > > > Constructed a part named fr2048 > > /home/yifeng/lib/bin/ibis: start building indexes (nthreads=1, > > indexingOption= <binning none/><encoding interval-equality/>) ... > > index::create create an empty index for d1 > > Warning -- index[fr2048.d1]::create failed to create an index of type > > <binning none/><encoding interval-equality/> > > index::create create an empty index for d1 > > Warning -- index[fr2048.d1]::create failed to create an index of type > > <binning none/><encoding interval-equality/> > > index::create create an empty index for d2 > > Warning -- index[fr2048.d2]::create failed to create an index of type > > <binning none/><encoding interval-equality/> > > index::create create an empty index for d2 > > Warning -- index[fr2048.d2]::create failed to create an index of type > > <binning none/><encoding interval-equality/> > > index::create create an empty index for d3 > > Warning -- index[fr2048.d3]::create failed to create an index of type > > <binning none/><encoding interval-equality/> > > index::create create an empty index for d3 > > Warning -- index[fr2048.d3]::create failed to create an index of type > > <binning none/><encoding interval-equality/> > > index::create create an empty index for v1 > > Warning -- index[fr2048.v1]::create failed to create an index of type > > <binning none/><encoding interval-equality/> > > index::create create an empty index for v1 > > Warning -- index[fr2048.v1]::create failed to create an index of type > > <binning none/><encoding interval-equality/> > > part[fr2048]::buildIndexes processed 4 columns using 1 thread took 0 > > CPU seconds and 0.00066901 elapsed seconds > > /home/yifeng/lib/bin/ibis: building indexes for 1 data partition took > > 0.001 CPU seconds, 0.0100073 elapsed seconds > > > > /home/yifeng/lib/bin/ibis -- total CPU time 0.001 s, total elapsed > > time 0.0340256 s > > fileManager::clear -- completed with 56 bytes of storage remain in > > memory after removing all managed objects > > fileManager::clear -- completed with 56 bytes of storage remain in > > memory after removing all managed objects > > > > Thanks, > > Yifeng. > > > > -- > > 耿益锋 > > Yifeng Geng > > > > 清华大学计算机科学与技术系 > > 高性能计算研究所 > > HPC Research Center, > > Department of Computer Science & Technology, > > Tsinghua University, Beijing, P.R.China. > > > > > > _______________________________________________ > > FastBit-users mailing list > > [email protected] > > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > > > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > -- 耿益锋 Yifeng Geng 清华大学计算机科学与技术系 高性能计算研究所 HPC Research Center, Department of Computer Science & Technology, Tsinghua University, Beijing, P.R.China.
_______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
