Hi, Teryl, Here is a web page with some information about data organization <http://crd.lbl.gov/~kewu/fastbit/doc/dataLoading.html>.
After you have put your data in the format used by FastBit, say in a directory called data_directory, then you can use ibis command line tool to generate indexes explicitly ibis -d data_directory -b -v or cause indexes to be generated by issuing queries. There are more information about the command line tool at this web page <http://crd.lbl.gov/~kewu/fastbit/doc/ibisCommandLine.html>. If you have data_dir1 and data_dir2, then you can use ibis command to append data_dir2 to data_dir1 by the following command ibis -append data_dir2 to data_dir1 This command leaves data_dir2 as is, but append all values in data_dir2 to the corresponding files in data_dir1. It also modifies the relevant metadata files. However, it does not update the indexes unless both data_dir1 and data_dir2 contain the same type of indexes for each column. Hope this helps. John On 2/25/2010 11:57 AM, Teryl Taylor wrote: > Hi John, > > Thanks for the quick response. If I was to put the data in the proper > column oriented format. How would I get FastBit to take the > column/columns and build the index file? As well as the meta data > file? Or is that done on the call to append? And when you say > "append" puts two data partitions together, do you mean the columns for > two partitions are merged physically (on the hard disk) into a single > set of columns or just in memory? > > Regards, > > Teryl > > > On Thu, Feb 25, 2010 at 3:46 PM, K. John Wu <[email protected] > <mailto:[email protected]>> wrote: > > Hi, Teryl, > > Thanks for your interest in FastBit. > > If your data are already in the format usable by FastBit, then there > is function called ibis::part::append that can put two data > partitions together. > > Of course, you can always write your own function to extend an > existing data partition if your input data is not exactly in the > format FastBit likes.. > > John > > > > On 2/25/2010 11:41 AM, Teryl Taylor wrote: > > Hi everyone, > > I'm new to FastBit and I just had a quick question. I have a > bunch of > binary data files with integers, floats, and chars in them and I > want to > be able to bulk load into fastbit at interval times. What > would be > the best way to do this? I don't really want to convert to CSV > files. Can I create column data through my loader program and then > just have FastBit create the indexes and metafiles for it? > Which API > would be the best to use? > > Thanks for any information you can provide, > > Cheers, > > Teryl > > > > > _______________________________________________ > FastBit-users mailing list > [email protected] <mailto:[email protected]> > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > > _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
