Michael Bayer wrote:
So youre running 189K records through the mapper system ? or just the sql construction system ? if the former, that WeakValueDictionary change should also help.
Yep, the former - I know it's overkill, but originally this particular area only dealt with very small subsets (100 or so) - a requirements change asked for the whole data set, so I just reused the existing mapper-based code and it all worked fine - a testament to SQLAlchemy!
In this case, I don't think the WeakValueDictionary change will have a lot of impact as it is a one-shot command-line thing, but it will in later processing (e.g. I'm generating a lot [> 40k] of graphs).
As I've mentioned previously, this system processes millions of records each night, with the first stage - processing external data - done almost entirely using the lower level, sql construction side of SQLAlchemy for performance and the latter stages (graphs, etc) done using the mapper side of things for convenience. The web app is now using Pylons/SQLAlchemy and the combination is working a treat! It's still in stealth mode, but the target is early Feb for a limited preview release. Again, the lack of problems, even in these early releases of both SQLAlchemy and Pylons speaks volumes to the quality of the code and the people behind it!!!!!!!!
Robert ------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click _______________________________________________ Sqlalchemy-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/sqlalchemy-users
