Am Dienstag, den 26.03.2013, 15:44 +0100 schrieb Andrea Mauri: > one more thing. my data is more similar to a huge spreadsheet than a > relational DB, anyway I am looking for the best option already available > in lazarus/fpc to store and query the dataset > > Il 26/03/2013 15:38, Andrea Mauri ha scritto: > > Dear all, > > I am looking for the best option in order to store big datasets with > > thousands of columns. > > The dataset can contains from tens to hundred thousands lines and > > thousand of columns (some columns are string some numbers). > > Which is the best option to store and retrieve information from a > > dataset like this? > > Actually I am using sqlite, tens of tables including maximum 200 > > columns. But I am not sure this is the best option. > > Within my application I have to query this dataset: > > - retrieve a particular line (all or some columns); > > - retrieve a particular column (all or some lines); > > - order the dataset with respect to a particular column; > > - delete/add line(s); > > - delete/add column(s); > > ... > > > > SQLIte is easy to use when I need to query the dataset but I am not sure > > that is the most suitable. > > > > Any hint?
Scientific data formats maybe best for this task. Something like HDF or NetCDF or the like. I have wrapped the header of HDF5 in version 1.6.4 (or was it .5?) a long while ago with the help of MvC and that is lingering on my harddisk somewhere in an unknown state - but has been working quite good in the cases tested. That might help somehow. I'd have to clean up a bit for sending you a copy, but be aware that curently HDF5 is coming in to flavors: the "old" style that is sort of abandoned but still usable had version number 1.6.9 last time I had looked after it. The new "product line" is starting with version 1.8.x and I have no idea what changes would be necessary. Those two flavors are not installable in parallel (at least on FreeBSD I'm using). There are some problems in declaring hfd's "compound types" in source code coming from the original source using C-macro-voodoo that is not very handily portable, but you should not be forced to use that data types, simple "array"-type is yours, I think. Look there to get an impression: http://www.hdfgroup.org/HDF5/ If you're intrested and have a little time to wait for results from my side, I'll send you a copy. Have fun, Marc -- Marc Santhoff <[email protected]> -- _______________________________________________ Lazarus mailing list [email protected] http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
