Davide Alberani wrote: >> I disabled it and did executed "COMMIT" every 1000 inserts. It >> makes the thing slightly faster (around 10--20%) but not much. > > I hoped for better performances, on DB2 - but I fear we're not > using it properly. :-)
I'm not sure whether there is a way to improve significantly here. I found a document at IBM's website giving some hints on how to speed up INSERTs in DB2: <http://www.ibm.com/developerworks/db2/library/tips/dm-0403wilkins/> I would be great if you could have a quick look at it. Maybe you get an idea how an easy-to-implement solution could look like. :-) To get at least the minor performance increase I reported, the following could be done (maybe you can integrate it easily somehow): (1) Switch off autocommit after the cursor has been created: connection.set_autocommit(False) (2) Execute the SQL command "COMMIT" after a bunch of INSERT commands has been executed A fix that definitely will improve insert performance on all databases is exporting all data into CSV files (one for each database table). These files then can be bulkloaded by special tools provided by each database (e.g. the LOAD command in DB2 or LOAD DATA INFILE in MySQL). I'm quite sure that for most database systems this is the recommended way to insert large data sets. Do you think IMDbPY could be extended by such a feature? :-) > I hope to have fixed this one, too: have a look at the TableAdapter > code: I hope the ibm_db driver does the right thing, when the CLOB > type is used. > Oh, actually CLOB is only used for ibm db2, and only for the > person_info.info column; if this is the right choice for other > cols, we can change the code. Works great now, thanks! :-) I took DB2 just amazing 48 minutes to import all biographies ... :-( Thanks, Joachim -- M. Sc. Joachim Selke Technische Universität Braunschweig, Institut für Informationssysteme Mühlenpfordtstraße 23, 38106 Braunschweig, Germany <http://www.l3s.uni-hannover.de/~selke> ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Imdbpy-devel mailing list Imdbpy-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/imdbpy-devel
