Hi, Anderson, The core limitation of FastBit is that when building indexes at least one column and its corresponding index must fit into memory. Since you have about 44 million rows, to hold a double-precision column in memory table abut 350 MB. The size of the corresponding index is like about the same size -- however, because the memory is allocated in relatively small chunks (especially if there are many distinct values in the data), there is likely a lot of waste. The more distinct values there are, the more waste there will be. For double precision values, especially those computed from simulations, there are many different distinct values.
With that explanation, here are two suggestions for dealing with the problem. One suggestion is to break the data into smaller partitions. For example convert each CSV file into a data partition. Since the total volume is relatively small, another possibility is to tell FastBit to use more memory. By default, FastBit will use half of the physical memory. You can tell it to use more memory by using a configuration parameter called fileManager.maxBtyes. The easiest way to get ibis to read this parameter is to put the following line in a file named ibis.rc in the current working directory. fileManager.maxBytes = 1.5GB Hope these help. John On 5/26/11 8:51 AM, Anderson C. Carniel wrote: > Hi John! > > I'm using fastbit 1.2.3. I have 5 CSV files, each csv file has > 6,438,450 rows and about 460 MB. These data are organized into eight > columns on which I build the data partition without problems, as follows: > > /opt/fastbit-ibis1.2.3/examples/ardea -d /test/agg/index0 -m > "col5:key,col4:key,col7:key,col6:key,col1:double,col0:double,col3:key,col2:int" > -t /test/agg/csv0.csv -t /test/agg/csv1.csv -t /test/agg/csv2.csv > /opt/fastbit-ibis1.2.3/examples/ardea -d /test/agg/index1 -m > "col5:key,col4:key,col7:key,col6:key,col1:double,col0:double,col3:key,col2:int" > -t /test/agg/csv3.csv -t /test/agg/csv4.csv > > But when I build the index: > > /opt/fastbit-ibis1.2.3/examples/ibis-d / test/agg/index0-b "<bining > none/> <encoding equality/>" > > The ibis consumes all available memory, and do much swap and not > complete the construction, this operation has been running for about > 15 hours. > > My machine has 2 GB of RAM, where the accounts should support up to > 44,564,480 lines to build the index. But even using only about 19 > million lines for the first partition, the ibis was unable to build > the index. > > What could be the problem? > > Thanks for the help. > Ouvir > Ler foneticamente > > Best regards > > []s > > > > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
