So I played around a little bit futher. I tried to use my new solution (using table.append instead of row.append) with following settings in table.openFile() METADATA_CACHE_SIZE=2*1024*1024 NODE_CACHE_SLOTS=1024
I saw the same performance problems like in the previous code. So it slows down after a while to 1 table/sec instead of 10 tables/sec When i change it to NODE_CACHE_SLOTS=0 I don't have any performance problems. On Thu, Jan 19, 2012 at 7:43 AM, Francesc Alted <fal...@pytables.org> wrote: > 2012/1/18 Ümit Seren <uemit.se...@gmail.com> >> >> Hi Francesc, >> I will try to get some numbers as soon as I have some time at hand. >> However I am not sure if I can come with an absolute number. >> It seems that at the beginning (first 1000 tables) I see no >> performance penalty, however after that the performance quickly >> degrades. Does traversing/accessing a huge number of groups/dataset >> have an effect on row.append? > > > I'd say not row.append() directly, but just node management speed. It is > always best to keep your number of datasets low and your datasets size > large. > >> >> Just as a sidenote: when I did my test I didn't change any of the >> default parameters like METADATA_CACHE_SIZE or NODE_CACHE_SLOTS=1024. >> BTW I am using pytables 2.3 and HDF 1.8.7 > > > Hmm, you could disable the internal node caching completely by setting > NODE_CACHE_SLOTS to 0. That could help performance too. > > -- > Francesc Alted > > ------------------------------------------------------------------------------ > Keep Your Developer Skills Current with LearnDevNow! > The most comprehensive online learning library for Microsoft developers > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, > Metro Style Apps, more. Free future releases when you subscribe now! > http://p.sf.net/sfu/learndevnow-d2d > _______________________________________________ > Pytables-users mailing list > Pytables-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/pytables-users > ------------------------------------------------------------------------------ Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d _______________________________________________ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
