not sure i've read all the posts on this, and i don't fully understand the problem, but someone's mentioned sqlalchemy, so here's my experience with that and large updates using mapped objects.
1 - don't commit each object as you modify it. instead, process a whole pile in memory and then (perhaps every 10,000 - when your memory is about to run out and start paging) flush the session. you'll find that the memory use is very predictable - it ramps up as objects are cached and then drops back down to the original level once they're sent to the database. 2 - various sql related commands in sqlalchemy will take lists rather than single values, and process all elements in the list in one "chunk". this is much more efficient. using sql directly is faster and uses lest memory than using mapped objects (the nice thing about sqlalchemy is that you can use sql directly when it's the best solution, and mapped objects when they are more useful). these are both kind-of obvious, but that's all i needed to handle fairly large data volumes with sqlalchemy. andrew Carbon Man wrote: > I have a program that is generated from a generic process. It's job is to > check to see whether records (replicated from another system) exist in a > local table, and if it doesn't, to add them. I have 1 of these programs > for > every table in the database. Everything works well until I do the postcode > table. The generated code is 5MB for a system with no current data. > Normally > the file would not be this big as only the changes are copied over. Python > just quits, I have tried stepping through the code in the debugger but it > doesn't even start. > I am thinking that dynamically generating the programs to run might not be > such a good idea. It would be a shame to drop it because the system needs > to > be generic and it runs from an XML file so the resulting code could be > pretty complex, and I am new to Python. The program did generate a pyc so > it > was able to compile. > Thoughts anyone? > > > -- > http://mail.python.org/mailman/listinfo/python-list > > -- http://mail.python.org/mailman/listinfo/python-list
