On 10/25/06, Mike Bernson <[EMAIL PROTECTED]> wrote: > > I Known there are faster ways of loading the data. I am looking for the > fastest way under SQLAlchemy. If the loads become far too slow I will > look into using methods outside of SQLAlchemy. At this stage there are > about 10,000 records being loaded and things are slow workable. I am > just checking to make sure that I am not overlooking something.
You don't necessarily have to go outside SQLAlchemy, since you can pass whatever literal SQL command you want as a string in the select() call. You *do* want to bypass SQLAlchemy's "insert()" mechanism here -- but that's just bypassing the INSERT statement. It's probably not worth adding some kind of bulkload() command to SQLAlchemy, since every database that has the feature has *very* different syntax and the feature's not used all that often. But it's designed for exactly the kind of situation you're in. Michael's right -- check a bulk load statement vs. a bunch of INSERT statements and see how much faster it goes. I'd say, do it at the SQL level, even lower than the DBAPI level, so you can see just how much better LOAD DATA INFILE is even when the INSERT statements are in their "optimum" condition. Practicality beats purity -- you'll get a massive speed boost here, and you're not really deriving any benefit from doing it the hard way anyway, unless that simplified example was *really* simplified. Use SQLAlchemy where it makes sense -- but when your really need a LOAD DATA INFILE command, don't let the fact that SQLAlchemy only offers insert() be a straightjacket on your programming style. Use the right tool for the job, and your program will be better and faster. -- Robin Munn [EMAIL PROTECTED] GPG key 0xD6497014 --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy -~----------~----~----~----~------~----~------~--~---
