Hi, Jon, Thanks for your interest in FastBit. Here is a web page with a little bit more information about -part.txt: <https://sdm.lbl.gov/fastbit/doc/dataLoading.html>
If you use FastBit software directly, each "column" of a "table" is stored in its own file in raw binary format (with no padding, no decoration). Guess that is what you describe as 0 offset, dense columnar binary data. It is possible to write a wrapper to allow FastBit indexing capability to work with data in different organizations. I imagine that you will need to write this wrapper layer in order for FastBit to work with your data that packs multiple columns into a single file. The In-memory API might be something helpful for you in this case. Here is a pointer to the documentation generated by doxygen <https://sdm.lbl.gov/fastbit/doc/html/group__FastBitIAPI.html>. Please feel free to let us know if you have more quesitons. John On 11/3/16 4:32 PM, Woodring, Jonathan Lee wrote: > I was looking to see if there was more documentation on -part.txt for the > Ibis implementation. > > In particular, is it possible to use an binary data that is a appended arrays > of columnar data, by providing the offset per array? As a related question, > could you specify a struct offset if the data happen to be an array of > structs (this particular use case isn't, but other data might be). > > Or does FastBit fundamentally require that the data are written as 0 offset, > dense columnar binary data? > > If not, how easy/hard would it be to use the library to be able to generate > indices on that sort of data? Essentially, the use case will not/cannot > reformat/copy the data because it is too big. Though, it is essentially > columnar, but not separate files per column (columns are offsets into a > single file). > > Thanks, > -Jon > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
