hey, all, I currently have a 64GB binary data file ( one variable/one column data), and I want to build indexes for this file. The problem I had is how to do partitioning, so that fastbit can handle the index building. On the fastbit website, it has the reason why fastbit can not handle one large partition data, but it seems it does not have an example to show how to do the partitioning. Or probably it is very straightforward way which does not need the example. But, I just dont know how to deal with it. I did a way that I feel right, but it turns out fastbit does not like it. What I did was manually chopping the file into multiple 2GB binary data files, and created separate folder for each data. Then, I generated -part.txt file under every folder. The -part.txt file has same content (same variable, row number etc ). The reason I did this that, probably, fastbit will treat this as one single column/variable, but with multiple partitions. However, it turns out fastibt treated those files are same, so, it only kept the last partition. My question is how can I do it correctly? Another way I was imaging is only keeping one -part.txt for all the partitions, and specifying the partition size in the file. Something like: BEGIN HEADERName = tv-table$Description = "Created on Thu Apr 10 23:23:29 EDT 2014 with 4294967296 rows and 1 columns."Number_of_columns = 1Number_of_rows = 4294967296END HEADER BEGIN COLUMNname="tv"data_type="DOUBLE"partition_size=134217728 rowsEND COLUMN But I guess the -part.txt does not support this. Anyway, probably fastbit has the approach to solve it, but I did not find it. Would be possible you can point me to there? Thanks nan
_______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
