A Wednesday 16 September 2009 20:51:25 escriguéreu: > When I initially migrated my sqlite code to pytables, the pytables code was > a lot smaller. But then I realized if I stored the keys in the data table. > It will make my data very large (I also found out that having one large > table is the way to go against having a large number of smaller tables). > Imagine having to repeat the same keys over and over again on 10+ Million > Rows (I'm guessing that compression would help, however, don't know what > the downside to that is). > > Then I replaced the keys (as you suggested with unsigned 16 ints) and have > a separate table which stores the mapping from keys to indexes.
Won't PyTables' enumerated types would serve you better in this case? > I have > another table which stores metrics on the table, so as I update the data > table the metrics are kept up to date. The code now has gotten larger than > sqlite. I'm hoping that the performance improvement will more than make up > for the additional complexity. I'm also looking forward to compression > which I would not have gotten with sqlite. Compression should help a lot when dealing with columns whose entries are repeated with frequency. And much more for keeping the indexes, that will be very much reduced (compressed). > Things which would be nice to have on indexed columns (without having to > read all the data): > > 1. Min and Max > 2. Distinct Yeah. I think these are feasible to obtain by using the indexes in PyTables (I mean, OPSI indexes). I'll *try* to implement that for 2.2 release. > And be able to limit number of rows to get back from the db. Will look into that too. -- Francesc Alted ------------------------------------------------------------------------------ Come build with us! The BlackBerry® Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9-12, 2009. Register now! http://p.sf.net/sfu/devconf _______________________________________________ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
